Qwen3.8-2B-Distill-GGUF

Qwen3.8-2B-Distill-GGUF

Runs a small 2B reasoning model on phones, laptops and single-board computers, as GGUF files in five quant levels. A distillation of a large Qwen3.8 model, packaged by Empero for llama.cpp.

Made by empero-ai on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This repository holds GGUF files of Qwen3.8-2B from empero-ai, a small reasoning model with about 1.9B parameters. The model was distilled from a much larger Qwen3.8 teacher into the Qwen3.5-2B architecture. The files come in five sizes, from a 1.312 GB Q4_K_M file to a 3.897 GB BF16 reference.

Who it suits: People who want a small local model for chat, math and reasoning on modest hardware, and developers who need something light for apps.

Industry | Who uses it | How they use it | Example
Software | Developers | Embed a small local model in an app | A chat feature that works offline
Education | Students and teachers | Try a reasoning model on a school laptop | A math helper that runs locally
Hobby electronics | Makers | Run a model on a single-board computer | A home assistant prototype
Research | ML researchers | Study a distilled small model against its base | A comparison on math word problems

If you want the largest and most capable model, you can skip it, since this one is built for small devices.

Where to use it:

• llama.cpp, Ollama, LM Studio, Jan and KoboldCpp
• Other stock GGUF runtimes on Windows, macOS or Linux
• Phones and single-board computers, with the smaller quants

Use case | Who | What they get back
Local chat | Individuals | Reasoned answers produced on a phone or laptop
Math and logic | Students | Worked solutions with a visible thinking block
Offline assistants | Developers | A small model without a network connection

How to set it up:

• Python, to run the Hugging Face download tool
• A llama.cpp release new enough to know the Qwen3.5 design and its Gated DeltaNet layers, because older ones cannot open the file
• Q4_K_M and Q5_K_M suit phones and any modern laptop, even CPU-only
• Q6_K and Q8_0 need a GPU with 4 GB or more, or a CPU with 8 GB of RAM
• BF16 needs a GPU with 6 GB or more

1. Install the Hugging Face hub tools with pip.
2. Download the repository with the hf command, or pick one GGUF file.
3. Load the file in llama.cpp, Ollama, LM Studio, Jan or KoboldCpp.

pip install -U huggingface_hub
hf download empero-ai/Qwen3.8-2B-Distill-GGUF

How to use it: Start a chat with llama-cli and the file you chose. Use the built-in chat template, which the -cnv flag turns on, and allow a long output since the model thinks before it answers.

llama-cli -m Qwen3.8-2B-Q4_K_M.gguf \
  --temp 0.6 --top-p 0.95 --top-k 20 \
  -n 16384 -cnv

Tool | What it does
Qwen3.8-2B-Q4_K_M.gguf | The recommended balance of quality and size, 1.312 GB
Qwen3.8-2B-Q5_K_M.gguf | Higher quality for a modest size increase, 1.455 GB
Qwen3.8-2B-Q6_K.gguf | Close to lossless, 1.606 GB
Qwen3.8-2B-Q8_0.gguf | The highest-quality quantization, 2.077 GB
Qwen3.8-2B-BF16.gguf | The full precision reference, 3.897 GB

Example: You start llama-cli with the Q4_K_M file and ask "What is 17 times 23, and show your steps". The model first writes out its thinking, then gives the answer. Your app should hide that thinking block before showing the reply to users.

For newcomers and experts: Begin with the Q4_K_M file, which the card recommends, and keep the card's sampling values. If answers get cut off, raise the output limit.

The base family mixes three Gated DeltaNet layers with each full-attention layer. The card says the distillation used about 30,000 curated teacher traces. At long context the KV cache becomes the main memory cost, so the weight size understates what you need.

Download one GGUF file
Load it in a GGUF runtime
Set low temperature, top-p, top-k
Model thinks, then answers
Strip the thinking block for users

Good to know:

• License: Apache 2.0
• Cost to run: none named
• Data it reaches: downloads files from Hugging Face, then runs on your own device
• Not for: older llama.cpp builds, which fail to load this architecture

Title: Chain-of-thought scores in the card, base versus distilled
mmlu, Qwen3.5-2B base: 0.283
mmlu, Qwen3.8-2B: 0.548
gsm8k_cot, Qwen3.5-2B base: 0.330
gsm8k_cot, Qwen3.8-2B: 0.640

On Hugging Face, 13 November 2026

• Task: text generation
• Library: gguf
• Parameters: 1.9B
• Weights: about 971 MB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 509,820
• Likes: 184
• License: apache-2.0
• Languages: en
• Based on: empero-ai/Qwen3.8-2B
• Quantized versions: 1
• Tags: gguf, llama.cpp, quantized, empero-ai, qwen3.5, qwen3.8, distillation, reasoning, gated-deltanet, edge, text-generation, conversational

Links

• Model: https://huggingface.co/empero-ai/Qwen3.8-2B-Distill-GGUF
• Files: https://huggingface.co/empero-ai/Qwen3.8-2B-Distill-GGUF/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download empero-ai/Qwen3.8-2B-Distill-GGUF

Then follow the usage in its model card. Nothing in this kit runs by itself.