Qwen3.8-35B-A3B-Distill-GGUF

Qwen3.8-35B-A3B-Distill-GGUF

Runs a 35B Mixture-of-Experts model with about 3B active parameters per token from GGUF files in ten sizes, plus a vision projector. For llama.cpp, Ollama, LM Studio, Jan and KoboldCpp users.

Made by empero-ai on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This repository packs empero-ai's Qwen3.8-35B-A3B-Distill into GGUF files of different sizes. That model was distilled from the Qwen3.8 frontier models into a Qwen3.6-35B-A3B mixture-of-experts network. It has 35B parameters in total, and about 3B work on each token. You can load the files in llama.cpp, in Ollama, or in desktop apps such as LM Studio, Jan and KoboldCpp.

Who it suits: People who run large models on their own machine and want a file sized for their memory, including those with only a CPU and plenty of RAM.

Industry | Who uses it | How they use it | Example
Software development | Local-model users | Run a reasoning model offline | A chat session in llama.cpp
AI tooling | Desktop app users | Load the file in LM Studio, Jan or KoboldCpp | A local chat app using the Q4_K_M file
Research | Model researchers | Compare quantization levels | Q2_K against Q8_0 on one model
Vision work | Hobbyists | Describe images with the vision projector | A photo described through llama-mtmd-cli

If you have less than 16 GB of memory, you can skip it, since even the smallest file is about 12.6 GB.

Where to use it:

• llama.cpp, Ollama, LM Studio, Jan and KoboldCpp
• Windows, macOS or Linux, after you fetch the files with Python
• A machine with enough VRAM or system RAM to hold the whole file

Use case | Who | What they get back
Reasoning chat | Local-model users | Answers that open with a think block
Image description | Hobbyists | A description of a photo, using the vision projector
CPU or partial offload runs | People without a big GPU | Practical speeds, since only about 3B parameters are active

How to set it up:

• Python and pip
• A current llama.cpp build that supports this model's design, because older builds cannot load it
• Memory for the whole file: 16 GB for the smallest, 24 GB of VRAM for Q4_K_M on a GPU, or 32 GB of RAM on a CPU

1. Install the Hugging Face hub tools.
2. Download the model files, or pick one file from the repository.
3. Check the files against the SHA256SUMS list, then run one with your runtime.

pip install -U huggingface_hub
hf download empero-ai/Qwen3.8-35B-A3B-Distill-GGUF

To check the downloads against the list of checksums in the repository:

sha256sum -c SHA256SUMS --ignore-missing

How to use it: Run the file in llama.cpp with the built-in chat template. This is a reasoning model, so every answer opens with a think block. Allow a generous token limit and strip that block before you show answers to users. Other runtimes load the file directly, and the chat template is stored inside it.

llama-cli -m Qwen3.8-35B-A3B-Q4_K_M.gguf \
  --temp 0.6 --top-p 0.95 --top-k 20 \
  -n 16384 -cnv

For images, pair the projector file with any text file.

llama-mtmd-cli -m Qwen3.8-35B-A3B-Q4_K_M.gguf \
  --mmproj mmproj-Qwen3.8-35B-A3B-F16.gguf \
  --image photo.jpg -p "Describe this image."

Tool | What it does
llama-cli | Runs a text chat with the chosen GGUF file
llama-mtmd-cli | Runs the model on an image with the projector file
sha256sum | Checks the downloaded files against the SHA256SUMS list

Example: You run the Q4_K_M file with llama-cli and ask a question. The model writes a think block first and then its answer. For a photo, you run llama-mtmd-cli with the projector and the prompt that asks it to describe the image.

For newcomers and experts: If you are new to this, start with the Q4_K_M file, which the card recommends for most people. Check that your llama.cpp build is recent, then ask a short question.

Inside, 30 of its layers are Gated DeltaNet and 10 use full attention. It holds 256 experts and sends each token to 8 of them. Only the whole file must fit in memory, not the 35B of compute. The IQ quants and the 2-bit and 3-bit K-quants use an importance matrix to stay coherent. The vision tower comes unchanged from the base, and the card says the distillation was text-only and vision was not evaluated.

Pick a file that fits your memory
Check it against SHA256SUMS
Run it in llama.cpp or another app
Strip the think block from answers

Good to know:

• Memory: the whole file must fit, and a long context adds a large cache cost
• Vision: the projector is inherited from the base, and the card says vision was not evaluated
• Reference file: the BF16 file is full precision and meant for reference only

Title: File size of selected quantizations in GB
IQ2_M: 12.558 GB
IQ3_M: 16.340 GB
Q3_K_M: 17.664 GB
IQ4_XS: 19.628 GB
Q4_K_M: 21.713 GB
Q6_K: 29.209 GB
Q8_0: 37.802 GB
BF16: 71.067 GB

• License: Apache-2.0
• Cost to run: no API key or paid service is named, only your own hardware
• Data it reaches: downloading fetches model files, and chats run locally in your runtime
• Not for: older llama.cpp builds, which fail to load this architecture

On Hugging Face, 13 November 2026

• Task: text generation
• Library: gguf
• Parameters: 35.5B
• Weights: about 17.8 GB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 1,337,097
• Likes: 277
• License: apache-2.0
• Languages: en
• Based on: empero-ai/Qwen3.8-35B-A3B-Distill
• Tags: gguf, llama.cpp, quantized, empero-ai, qwen3.6, qwen3.8, distillation, reasoning, gated-deltanet, text-generation, conversational

Links

• Model: https://huggingface.co/empero-ai/Qwen3.8-35B-A3B-Distill-GGUF
• Files: https://huggingface.co/empero-ai/Qwen3.8-35B-A3B-Distill-GGUF/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download empero-ai/Qwen3.8-35B-A3B-Distill-GGUF

Then follow the usage in its model card. Nothing in this kit runs by itself.