Qwen3.8-4B-Distill-GGUF

Qwen3.8-4B-Distill-GGUF

Runs a 4B reasoning model on a laptop or a mid-range graphics card, as GGUF files in five quant levels. A distillation of a large Qwen3.8 model, packaged by Empero for llama.cpp and similar apps.

Made by empero-ai on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This repository holds GGUF files of Qwen3.8-4B from empero-ai, a reasoning model with about 4.3B parameters. It was distilled from a much larger Qwen3.8 teacher into the Qwen3.5-4B design. Five files are offered, from a 2.783 GB Q4_K_M quant to an 8.666 GB BF16 reference.

Who it suits: People who want a mid-size local model for chat, math and reasoning, and developers who need a model that fits on a consumer GPU.

Industry | Who uses it | How they use it | Example
Software | Developers | Run a local model behind an app or a chat window | An offline assistant for a desktop tool
Education | Students and teachers | Practice reasoning problems on a personal computer | A study helper that works offline
Research | ML researchers | Compare a distilled model against its base | A test on knowledge and math sets
IT operations | Small teams | Keep prompts on their own hardware | A private note summarizer

If you need a model for long documents or the best possible accuracy, you can skip it, since it is a small distilled model.

Where to use it:

• llama.cpp, Ollama, LM Studio, Jan and KoboldCpp
• Other stock GGUF runtimes on Windows, macOS or Linux
• Laptops and desktops with or without a graphics card

Use case | Who | What they get back
Local chat | Individuals | Reasoned answers produced on your own computer
Knowledge questions | Students | Step-by-step answers with a visible thinking block
App features | Developers | A model that runs without a network connection

How to set it up:

• Python, to run the Hugging Face download tool
• A llama.cpp release new enough to know the Qwen3.5 design and its Gated DeltaNet layers, because older ones cannot open the file
• For the two smaller quants, a card with 4 to 6 GB works, and CPU-only runs are also a strong option
• For Q6_K and Q8_0, 6 to 8 GB is recommended, and BF16 wants 12 GB or more

1. Install the Hugging Face hub tools with pip.
2. Download the repository with the hf command, or pick one GGUF file.
3. Load the file in llama.cpp, Ollama, LM Studio, Jan or KoboldCpp.

pip install -U huggingface_hub
hf download empero-ai/Qwen3.8-4B-Distill-GGUF

How to use it: Start a chat with llama-cli and the file you picked. The -cnv flag turns on the built-in chat template, and you should allow a long output because the model thinks before it answers.

llama-cli -m Qwen3.8-4B-Q4_K_M.gguf \
  --temp 0.6 --top-p 0.95 --top-k 20 \
  -n 16384 -cnv

Tool | What it does
Qwen3.8-4B-Q4_K_M.gguf | The recommended balance of quality and size, 2.783 GB
Qwen3.8-4B-Q5_K_M.gguf | Higher quality for a modest size increase, 3.161 GB
Qwen3.8-4B-Q6_K.gguf | Close to lossless, 3.563 GB
Qwen3.8-4B-Q8_0.gguf | The highest-quality quantization, 4.611 GB
Qwen3.8-4B-BF16.gguf | The full precision reference, 8.666 GB

Example: You start llama-cli with the Q4_K_M file and ask "Explain why the sky is blue in three steps". The model writes out its thinking first, then gives the three steps. Your app should hide the thinking block before showing the reply.

For newcomers and experts: Begin with the Q4_K_M file the card recommends and keep its sampling values. If answers stop early, raise the output limit.

The card says the distillation used about 45,000 curated teacher traces. In its own table, the distilled model scores 0.553 on mmlu against 0.354 for the base, but 0.785 on gsm8k_cot against 0.850. Long contexts make the KV cache the main memory cost, so weight size understates what you need.

Download one GGUF file
Load it in a GGUF runtime
Set low temperature, top-p, top-k
Model thinks, then answers
Strip the thinking block for users

Good to know:

• License: Apache 2.0
• Cost to run: none named
• Data it reaches: downloads files from Hugging Face, then runs on your own device
• Not for: older llama.cpp builds, which cannot load this architecture
• Limits: the card's table shows a lower gsm8k_cot score than the base model

Title: Chain-of-thought scores in the card, base versus distilled
mmlu, Qwen3.5-4B base: 0.354
mmlu, Qwen3.8-4B: 0.553
gsm8k_cot, Qwen3.5-4B base: 0.850
gsm8k_cot, Qwen3.8-4B: 0.785

On Hugging Face, 13 November 2026

• Task: text generation
• Library: gguf
• Parameters: 4.3B
• Weights: about 2.2 GB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 570,974
• Likes: 208
• License: apache-2.0
• Languages: en
• Based on: empero-ai/Qwen3.8-4B
• Quantized versions: 2
• Tags: gguf, llama.cpp, quantized, empero-ai, qwen3.5, qwen3.8, distillation, reasoning, gated-deltanet, text-generation, conversational

Links

• Model: https://huggingface.co/empero-ai/Qwen3.8-4B-Distill-GGUF
• Files: https://huggingface.co/empero-ai/Qwen3.8-4B-Distill-GGUF/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download empero-ai/Qwen3.8-4B-Distill-GGUF

Then follow the usage in its model card. Nothing in this kit runs by itself.