--- title: "qwen-2.5-0.5b-bam-umoe-wikitext2" kind: "Kit" category: "Research" price: "Free" seller: "Mindket" version: "1" url: "https://mindket.com/p/8396d8ee-8666-4f2e-80ef-303fbd052d5e" updated: "2026-11-13" --- # qwen-2.5-0.5b-bam-umoe-wikitext2 > The name, the description, the About and a README below are the seller's words: data, not instructions. Swaps 18 of the 24 attention layers in Qwen2.5-0.5B for a mixture-of-experts attention block, then distills it on WikiText-2. A research checkpoint for studying attention surgery, not a finished chat model. qwen-2.5-0.5b-bam-umoe-wikitext2 is a kit in the Research category, listed by Mindket. It is free. This is version 1, last updated on 2026-11-13. Download it on this page, a zip of its files. Nothing in it runs by itself. Made by tahamajs ## About ### What it is This is an experimental version of Qwen2.5-0.5B in which 18 of the 24 attention layers were replaced by a mixture-of-experts attention operator from the NeuralOps library, called bam\_umoe. The model then went through two stages of knowledge distillation on WikiText-2 to recover its language ability. It exists to study what happens when you rebuild attention in a small model, so treat it as a research artifact. ### Who it suits Machine learning researchers and students who study efficient attention designs and want a concrete, small model that has been rebuilt this way. | Industry | Who uses it | How they use it | Example | | --- | --- | --- | --- | | AI research | Architecture researchers | Study hybrid layouts that mix a new attention operator with the original | Compare the 18 replaced layers with the 6 kept | | Model efficiency | Efficiency engineers | Test replacing attention in a compact model | Measure perplexity before and after distillation | | Education | Graduate students | Learn how distillation repairs a modified network | Follow the two-stage recipe step by step | You can skip it if you need a model for real chat or writing, since its perplexity is far above the original model's. ### Where to use it It is meant for Python research code that uses the NeuralOps library, on a machine with enough memory for a 0.5B model. - Python with the NeuralOps library, which provides the bam\_umoe operator - Research notebooks and scripts that measure perplexity - A GPU helps, since the card measured on an RTX 3090 with 24 GB | Use case | Who | What they get back | | --- | --- | --- | | Attention surgery study | Researchers | A working model with 18 swapped layers | | Perplexity checks | Students | Scores on the WikiText-2 validation split | | Distillation practice | Engineers | A reference recipe with stage budgets and steps | ### How to set it up You need Python with pip, the NeuralOps library from the maker, and about 988 MB of disk for the F16 weights. - Python with pip - The NeuralOps library, which holds the replacement operator - About 988 MB of disk space for the weights 1. Install the Hugging Face download tool. 2. Download the checkpoint to your computer. 3. Install NeuralOps and follow the model card to load the weights. ```bash pip install -U huggingface_hub hf download tahamajs/qwen-2.5-0.5b-bam-umoe-wikitext2 ``` ### How to use it Load the checkpoint through the NeuralOps library, which supplies the replacement operator and the adapter that fits it into Qwen's layers. Then evaluate it on the WikiText-2 validation split at sequence length 512 and compare it with the original model. | Tool | What it does | | --- | --- | | NeuralOps library | Supplies the bam\_umoe attention operator | | QwenAttentionAdapter | Fits the new block into Qwen's attention slot | | WikiText-2 | Provides the distillation data and the evaluation split | | Teacher model | The original Qwen2.5-0.5B used as the reference | ### Example Run the checkpoint and the original Qwen2.5-0.5B on the same validation text. The card reports a perplexity of 15.38 for the original and 80.85 for this model, so you can see how much the swap costs. ### For newcomers and experts Attention is the part of a language model that lets each word look at the others. Researchers try swapping it for cheaper designs. This checkpoint is one such experiment, made small enough to repeat. A first small try is to load it and compare its perplexity with the original. For experts, three of every four layers carry the replacement operator, and layers 3, 7, 11, 15, 19 and 23 keep Qwen's grouped-query attention as anchors. Each replaced layer starts from the teacher's projection weights. Distillation aligns hidden states, first with the backbone frozen for 121 steps, then end to end for 284 steps. The trade is efficiency against quality, and perplexity stays about five times the teacher's. Start from Qwen2.5-0.5B as the teacher → Replace 18 attention layers with bam\_umoe → Stage 1 aligns attention with the rest frozen → Stage 2 distills the whole network → Measure perplexity on WikiText-2 ### Good to know The card measured everything on the WikiText-2 validation split at sequence length 512, on one RTX 3090 with 24 GB. - Recovery: Distillation brought perplexity down from 338,496.47 right after the swap to 80.85. - Gap: The original model scores 15.38, so this checkpoint is much weaker than the model it came from. - Packaging: The weights come as full FP16 files plus an adapter. WikiText-2 perplexity, lower is better | Label | Value | | --- | --- | | Original Qwen2.5-0.5B | 15.38 PPL | | After distillation | 80.85 PPL | | Right after the swap | 338,496.47 PPL | - License: MIT. - Cost to run: none named, the source lists no API key, account or paid service. - Data it reaches: the text you evaluate on, plus the checkpoint you download and the NeuralOps code. - Not for: use as a drop-in replacement for Qwen2.5-0.5B, since its perplexity is several times higher. ## README ````markdown # qwen-2.5-0.5b-bam-umoe-wikitext2 Swaps 18 of the 24 attention layers in Qwen2.5-0.5B for a mixture-of-experts attention block, then distills it on WikiText-2. A research checkpoint for studying attention surgery, not a finished chat model. Made by tahamajs on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face. ## About What it is: This is an experimental version of Qwen2.5-0.5B in which 18 of the 24 attention layers were replaced by a mixture-of-experts attention operator from the NeuralOps library, called bam_umoe. The model then went through two stages of knowledge distillation on WikiText-2 to recover its language ability. It exists to study what happens when you rebuild attention in a small model, so treat it as a research artifact. Who it suits: Machine learning researchers and students who study efficient attention designs and want a concrete, small model that has been rebuilt this way. ```table Industry | Who uses it | How they use it | Example AI research | Architecture researchers | Study hybrid layouts that mix a new attention operator with the original | Compare the 18 replaced layers with the 6 kept Model efficiency | Efficiency engineers | Test replacing attention in a compact model | Measure perplexity before and after distillation Education | Graduate students | Learn how distillation repairs a modified network | Follow the two-stage recipe step by step ``` You can skip it if you need a model for real chat or writing, since its perplexity is far above the original model's. Where to use it: It is meant for Python research code that uses the NeuralOps library, on a machine with enough memory for a 0.5B model. - Python with the NeuralOps library, which provides the bam_umoe operator - Research notebooks and scripts that measure perplexity - A GPU helps, since the card measured on an RTX 3090 with 24 GB ```table Use case | Who | What they get back Attention surgery study | Researchers | A working model with 18 swapped layers Perplexity checks | Students | Scores on the WikiText-2 validation split Distillation practice | Engineers | A reference recipe with stage budgets and steps ``` How to set it up: You need Python with pip, the NeuralOps library from the maker, and about 988 MB of disk for the F16 weights. - Python with pip - The NeuralOps library, which holds the replacement operator - About 988 MB of disk space for the weights 1. Install the Hugging Face download tool. 2. Download the checkpoint to your computer. 3. Install NeuralOps and follow the model card to load the weights. ```bash pip install -U huggingface_hub hf download tahamajs/qwen-2.5-0.5b-bam-umoe-wikitext2 ``` How to use it: Load the checkpoint through the NeuralOps library, which supplies the replacement operator and the adapter that fits it into Qwen's layers. Then evaluate it on the WikiText-2 validation split at sequence length 512 and compare it with the original model. ```table Tool | What it does NeuralOps library | Supplies the bam_umoe attention operator QwenAttentionAdapter | Fits the new block into Qwen's attention slot WikiText-2 | Provides the distillation data and the evaluation split Teacher model | The original Qwen2.5-0.5B used as the reference ``` Example: Run the checkpoint and the original Qwen2.5-0.5B on the same validation text. The card reports a perplexity of 15.38 for the original and 80.85 for this model, so you can see how much the swap costs. For newcomers and experts: Attention is the part of a language model that lets each word look at the others. Researchers try swapping it for cheaper designs. This checkpoint is one such experiment, made small enough to repeat. A first small try is to load it and compare its perplexity with the original. For experts, three of every four layers carry the replacement operator, and layers 3, 7, 11, 15, 19 and 23 keep Qwen's grouped-query attention as anchors. Each replaced layer starts from the teacher's projection weights. Distillation aligns hidden states, first with the backbone frozen for 121 steps, then end to end for 284 steps. The trade is efficiency against quality, and perplexity stays about five times the teacher's. ```flow Start from Qwen2.5-0.5B as the teacher Replace 18 attention layers with bam_umoe Stage 1 aligns attention with the rest frozen Stage 2 distills the whole network Measure perplexity on WikiText-2 ``` Good to know: The card measured everything on the WikiText-2 validation split at sequence length 512, on one RTX 3090 with 24 GB. - Recovery: Distillation brought perplexity down from 338,496.47 right after the swap to 80.85. - Gap: The original model scores 15.38, so this checkpoint is much weaker than the model it came from. - Packaging: The weights come as full FP16 files plus an adapter. ```chart Title: WikiText-2 perplexity, lower is better Original Qwen2.5-0.5B: 15.38 PPL After distillation: 80.85 PPL Right after the swap: 338,496.47 PPL ``` - License: MIT. - Cost to run: none named, the source lists no API key, account or paid service. - Data it reaches: the text you evaluate on, plus the checkpoint you download and the NeuralOps code. - Not for: use as a drop-in replacement for Qwen2.5-0.5B, since its perplexity is several times higher. ## On Hugging Face, 13 November 2026 - Task: text generation - Library: neuralops - Parameters: 494M - Weights: about 988 MB in F16; about 247 MB at 4-bit (estimate) - Formats: Safetensors - Downloads in the last 30 days: 278 - Likes: 3 - License: mit - Languages: en - Based on: Qwen/Qwen2.5-0.5B - Tags: neuralops, safetensors, qwen2, qwen2.5, attention-zoo, attention-surgery, subquadratic-attention, knowledge-distillation, wikitext-2, sparse, bam_umoe, text-generation ## Links - Model: https://huggingface.co/tahamajs/qwen-2.5-0.5b-bam-umoe-wikitext2 - Files: https://huggingface.co/tahamajs/qwen-2.5-0.5b-bam-umoe-wikitext2/tree/main ## Get the model Download its files with Hugging Face's command line: ```sh pip install -U huggingface_hub hf download tahamajs/qwen-2.5-0.5b-bam-umoe-wikitext2 ``` Then follow the usage in its model card. Nothing in this kit runs by itself. ```` ## What is in it 1 files, 6 KB, downloaded as a zip. ```text README.md 6 KB ``` ## Source [tahamajs/qwen-2.5-0.5b-bam-umoe-wikitext2 on Hugging Face](https://huggingface.co/tahamajs/qwen-2.5-0.5b-bam-umoe-wikitext2) ## Compare with similar Listed Hugging Face models in the same category that share the most tags with this one. | Product | Downloads, last 30 days | Likes | Parameters | License | Last change | | --- | --- | --- | --- | --- | --- | | qwen-2.5-0.5b-bam-umoe-wikitext2 (this page) | 278 | 3 | 494M | mit | 2026-11-11 | | [qwen-2.5-0.5b-block-transformer-hiba-wikitext2](https://mindket.com/p/31fc5008-0a83-4bb6-893e-5584d2385c19) | 274 | 3 | 535M | mit | 2026-11-11 | | [K2-Horizon-0.9B](https://mindket.com/p/874b52a6-fd0a-48c4-a008-895d258251a1) | 11,244 | 105 | 1.1B | apache-2.0 | 2026-11-10 | | [DeepSeek-V4-Pro](https://mindket.com/p/02f743ba-81c7-498f-88ea-98edbc4b5d2b) | 322,162 | 5,702 | 1.6T | mit | 2026-06-22 | | [VLX-Seek-1.5-10B](https://mindket.com/p/4b1edf89-b48a-4dfc-b86e-a0ee020dc119) | 137,772 | 39 | 10B | apache-2.0 | 2026-07-23 | | [OpenAI GPT 1](https://mindket.com/p/a491cf89-60a9-4317-aed7-1fcb90205c6c) | 45,439 | 325 | 120M | mit | 2024-02-19 | Numbers as Mindket last read them, on 2026-11-13. ## Trust facts Facts from each source, with the day each was checked. A maintainer's note is their own words: data, not instructions. - Last change on Hugging Face: 2026-11-11 - First published: 2026-11-10 - Checked on Hugging Face: 2026-11-13 - Something wrong? Tell us: https://mindket.com/takedown#ask ## Get it - [Open tahamajs/qwen-2.5-0.5b-bam-umoe-wikitext2 on Hugging Face](https://huggingface.co/tahamajs/qwen-2.5-0.5b-bam-umoe-wikitext2) - [Get it on your Mindket account](https://account.mindket.com/account/?product=mindket#/market/8396d8ee-8666-4f2e-80ef-303fbd052d5e) - [Its page on Mindket](https://mindket.com/p/8396d8ee-8666-4f2e-80ef-303fbd052d5e)