qwen-2.5-0.5b-bam-umoe-wikitext2

qwen-2.5-0.5b-bam-umoe-wikitext2

Swaps 18 of the 24 attention layers in Qwen2.5-0.5B for a mixture-of-experts attention block, then distills it on WikiText-2. A research checkpoint for studying attention surgery, not a finished chat model.

Made by tahamajs on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This is an experimental version of Qwen2.5-0.5B in which 18 of the 24 attention layers were replaced by a mixture-of-experts attention operator from the NeuralOps library, called bam_umoe. The model then went through two stages of knowledge distillation on WikiText-2 to recover its language ability. It exists to study what happens when you rebuild attention in a small model, so treat it as a research artifact.

Who it suits: Machine learning researchers and students who study efficient attention designs and want a concrete, small model that has been rebuilt this way.

Industry | Who uses it | How they use it | Example
AI research | Architecture researchers | Study hybrid layouts that mix a new attention operator with the original | Compare the 18 replaced layers with the 6 kept
Model efficiency | Efficiency engineers | Test replacing attention in a compact model | Measure perplexity before and after distillation
Education | Graduate students | Learn how distillation repairs a modified network | Follow the two-stage recipe step by step

You can skip it if you need a model for real chat or writing, since its perplexity is far above the original model's.

Where to use it: It is meant for Python research code that uses the NeuralOps library, on a machine with enough memory for a 0.5B model.

• Python with the NeuralOps library, which provides the bam_umoe operator
• Research notebooks and scripts that measure perplexity
• A GPU helps, since the card measured on an RTX 3090 with 24 GB

Use case | Who | What they get back
Attention surgery study | Researchers | A working model with 18 swapped layers
Perplexity checks | Students | Scores on the WikiText-2 validation split
Distillation practice | Engineers | A reference recipe with stage budgets and steps

How to set it up: You need Python with pip, the NeuralOps library from the maker, and about 988 MB of disk for the F16 weights.

• Python with pip
• The NeuralOps library, which holds the replacement operator
• About 988 MB of disk space for the weights

1. Install the Hugging Face download tool.
2. Download the checkpoint to your computer.
3. Install NeuralOps and follow the model card to load the weights.

pip install -U huggingface_hub
hf download tahamajs/qwen-2.5-0.5b-bam-umoe-wikitext2

How to use it: Load the checkpoint through the NeuralOps library, which supplies the replacement operator and the adapter that fits it into Qwen's layers. Then evaluate it on the WikiText-2 validation split at sequence length 512 and compare it with the original model.

Tool | What it does
NeuralOps library | Supplies the bam_umoe attention operator
QwenAttentionAdapter | Fits the new block into Qwen's attention slot
WikiText-2 | Provides the distillation data and the evaluation split
Teacher model | The original Qwen2.5-0.5B used as the reference

Example: Run the checkpoint and the original Qwen2.5-0.5B on the same validation text. The card reports a perplexity of 15.38 for the original and 80.85 for this model, so you can see how much the swap costs.

For newcomers and experts: Attention is the part of a language model that lets each word look at the others. Researchers try swapping it for cheaper designs. This checkpoint is one such experiment, made small enough to repeat. A first small try is to load it and compare its perplexity with the original.

For experts, three of every four layers carry the replacement operator, and layers 3, 7, 11, 15, 19 and 23 keep Qwen's grouped-query attention as anchors. Each replaced layer starts from the teacher's projection weights. Distillation aligns hidden states, first with the backbone frozen for 121 steps, then end to end for 284 steps. The trade is efficiency against quality, and perplexity stays about five times the teacher's.

Start from Qwen2.5-0.5B as the teacher
Replace 18 attention layers with bam_umoe
Stage 1 aligns attention with the rest frozen
Stage 2 distills the whole network
Measure perplexity on WikiText-2

Good to know: The card measured everything on the WikiText-2 validation split at sequence length 512, on one RTX 3090 with 24 GB.

• Recovery: Distillation brought perplexity down from 338,496.47 right after the swap to 80.85.
• Gap: The original model scores 15.38, so this checkpoint is much weaker than the model it came from.
• Packaging: The weights come as full FP16 files plus an adapter.

Title: WikiText-2 perplexity, lower is better
Original Qwen2.5-0.5B: 15.38 PPL
After distillation: 80.85 PPL
Right after the swap: 338,496.47 PPL

• License: MIT.
• Cost to run: none named, the source lists no API key, account or paid service.
• Data it reaches: the text you evaluate on, plus the checkpoint you download and the NeuralOps code.
• Not for: use as a drop-in replacement for Qwen2.5-0.5B, since its perplexity is several times higher.

On Hugging Face, 13 November 2026

• Task: text generation
• Library: neuralops
• Parameters: 494M
• Weights: about 988 MB in F16; about 247 MB at 4-bit (estimate)
• Formats: Safetensors
• Downloads in the last 30 days: 278
• Likes: 3
• License: mit
• Languages: en
• Based on: Qwen/Qwen2.5-0.5B
• Tags: neuralops, safetensors, qwen2, qwen2.5, attention-zoo, attention-surgery, subquadratic-attention, knowledge-distillation, wikitext-2, sparse, bam_umoe, text-generation

Links

• Model: https://huggingface.co/tahamajs/qwen-2.5-0.5b-bam-umoe-wikitext2
• Files: https://huggingface.co/tahamajs/qwen-2.5-0.5b-bam-umoe-wikitext2/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download tahamajs/qwen-2.5-0.5b-bam-umoe-wikitext2

Then follow the usage in its model card. Nothing in this kit runs by itself.