Qwen3.8-9B-Distill-GGUF

Qwen3.8-9B-Distill-GGUF

GGUF quantizations of Empero's Qwen3.8-9B, a reasoning model distilled from a larger Qwen3.8 into the Qwen3.5-9B design, for llama.cpp, Ollama, LM Studio and similar tools.

Made by empero-ai on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: Empero shrank its Qwen3.8-9B model into GGUF files, and this repository holds them in several quant types. The source model is a distillation of a larger Qwen3.8 model into the architecture of Qwen3.5-9B. The record lists 9.2B parameters and English as the language.

It is a reasoning model, so each answer starts with a thinking block. The training used about 70,000 curated traces from the larger teacher model.

Who it suits: It suits people who run models locally and want a 9B reasoning model in several file sizes, from a 5.8 GB Q4_K_M to a full-precision file. The card says the Q4 and Q5 files fit comfortably on 8 to 12 GB cards.

Industry | Who uses it | How they use it | Example
Software | Developers | Run a local reasoning model for code and logic | Working through a tricky algorithm
Education | Students and tutors | Practice with a model that shows its reasoning | Checking a math solution
Research | Analysts | Test a distilled model against its base | Comparing answers on exam style questions
Home labs | Enthusiasts | Run a 9B model on a consumer graphics card | Chatting on a 12 GB card

Skip it if you use a hosted assistant and never run models yourself.

Where to use it:

• llama.cpp, in a recent build with Qwen3.5 and Gated DeltaNet support
• Ollama, LM Studio, Jan and KoboldCpp, which load GGUF files directly
• Windows, macOS or Linux, with Python for the download step

Use case | Who | What they get back
Local chat with reasoning | Anyone with a GPU | Answers that open with a thinking block
School style questions | Students | Step by step answers
Code and logic help | Developers | Reasoned answers from a 9B model

How to set it up: You need a few things in place first.

• Python and pip, for the download command
• A recent llama.cpp build, since older builds fail to load this architecture
• Disk space for the file you pick, from 5.78 GB to 18.4 GB

1. Install the Hugging Face hub package with pip.
2. Download the repository with the hf command.
3. Pick a quant from the table below and load it in your tool.

pip install -U huggingface_hub
hf download empero-ai/Qwen3.8-9B-Distill-GGUF

Quant | Size | Note
Q4_K_M | 5.780 GB | Recommended, a balance of quality and size
Q5_K_M | 6.643 GB | A step up in quality, and an 8 GB card can hold it with a short context
Q6_K | 7.559 GB | Called near-lossless in the card
Q8_0 | 9.786 GB | The highest quality quantization
BF16 | 18.407 GB | Full precision reference

How to use it: In llama.cpp you run the command below with the built-in chat template turned on. The model opens each answer with a thinking block, so allow a large token limit and remove that block before showing the answer to end users. In the other tools, load the file and use the card's sampling settings.

llama-cli -m Qwen3.8-9B-Q4_K_M.gguf \
  --temp 0.6 --top-p 0.95 --top-k 20 \
  -n 16384 -cnv

Example: You start the command above in a terminal and type a school style math question. The model writes a thinking block first, then the final answer. You can cut the thinking part if you only want the answer.

For newcomers and experts: Start with the Q4_K_M file and the command above. It is the card's recommended pick for most people. Try Q5_K_M or Q6_K if you have spare memory.

The card reports a chain-of-thought MMLU score of 0.751 against 0.546 for the Qwen3.5-9B base, and a GSM8K score of 0.870 against 0.885. So the distillation gains on broad knowledge and slips slightly on grade school math. These are the maker's own numbers from identical settings. The Qwen3.5 family mixes three Gated DeltaNet layers with each full attention layer.

Title: Chain-of-thought scores from the card
MMLU, Qwen3.5-9B base: 0.546 score
MMLU, Qwen3.8-9B: 0.751 score
GSM8K, Qwen3.5-9B base: 0.885 score
GSM8K, Qwen3.8-9B: 0.870 score

Download a GGUF quant with hf
Load it in llama.cpp or another tool
Send a question with the chat template
Read the thinking block and the answer

Good to know:

• License: Apache 2.0
• Cost to run: none named, but the file sizes range from 5.78 GB to 18.4 GB
• Data it reaches: it downloads files from Hugging Face onto your computer
• Not for: older llama.cpp builds, which cannot load this architecture

On Hugging Face, 13 November 2026

• Task: text generation
• Library: gguf
• Parameters: 9.2B
• Weights: about 4.6 GB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 537,695
• Likes: 356
• License: apache-2.0
• Languages: en
• Based on: empero-ai/Qwen3.8-9B
• Quantized versions: 4
• Tags: gguf, llama.cpp, quantized, empero-ai, qwen3.5, qwen3.8, distillation, reasoning, gated-deltanet, text-generation, conversational

Links

• Model: https://huggingface.co/empero-ai/Qwen3.8-9B-Distill-GGUF
• Files: https://huggingface.co/empero-ai/Qwen3.8-9B-Distill-GGUF/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download empero-ai/Qwen3.8-9B-Distill-GGUF

Then follow the usage in its model card. Nothing in this kit runs by itself.