Qwen3.5-9B-MTP-GGUF

Qwen3.5-9B-MTP-GGUF

GGUF copies of Qwen3.5 9B that support multi-token-prediction speculative decoding in llama.cpp for faster generation. For people running the model locally who can build llama.cpp.

Made by unsloth on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: Qwen3.5-9B-MTP-GGUF is Unsloth's GGUF version of Qwen3.5-9B, an image-and-text model with 9.2B parameters. It supports MTP speculative decoding in llama.cpp, which the card says speeds up generation by about 1.5 to 2 times. The card also notes developer role support and better tool calling.

Who it suits: People who run Qwen3.5 locally with llama.cpp and are willing to build it themselves for the faster mode.

Industry | Who uses it | How they use it | Example
Software | Developers | Run a local model behind coding tools such as Codex or OpenCode | A local endpoint for a coding client
Research | AI hobbyists | Test speculative decoding on a 9B model | Compare speed with and without MTP
Education | Students | Run a current open model on one GPU | A local chat with Qwen3.5

You can skip it if you cannot build llama.cpp, or if you need several parallel requests, since the card says more than one slot is not yet supported with MTP.

Where to use it:

• llama.cpp, built from the MTP branch or a release with MTP
• Linux, where the card's build commands run, with options for CPU and Metal
• Coding clients such as Codex and OpenCode, per the card's developer role note

Use case | Who | What they get back
Faster local chat | Developers | Text generation that is about 1.5 to 2 times faster, per the card
Tool calling | Developers | Better parsing of nested objects in tool calls
Fine-tuning | Tinkerers | A model you can also fine-tune locally with Unsloth

How to set it up:

• A Linux machine with build tools, cmake and an NVIDIA GPU for the CUDA build
• llama.cpp built from a version that has MTP
• Python and pip, if you download with the hf command

1. Install the build tools.
2. Clone the llama.cpp repository, as the card shows.
3. Build with CUDA on, or set it off for CPU or Metal.
4. Start llama-server with the MTP flags.

apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp

pip install -U huggingface_hub
hf download unsloth/Qwen3.5-9B-MTP-GGUF

How to use it: Start llama-server with the repository and the UD-Q4_K_XL quant, and the draft MTP flags. Set the CUDA option to OFF in the build for CPU or Metal. The card says more than one slot and the mmproj file are not yet supported with MTP.

export LLAMA_CACHE="unsloth/Qwen3.5-9B-MTP-GGUF"
./llama.cpp/llama-server \
    -hf unsloth/Qwen3.5-9B-MTP-GGUF:UD-Q4_K_XL \
    -ngl 99 -c 8192 -fa on -np 1 \
    --spec-type draft-mtp --spec-draft-n-max 6

Example: You start the server with those flags and chat with it. Generation runs faster than plain decoding, and the card gives about 1.5 to 2 times as its figure.

For newcomers and experts: If you are new, the build step is the hard part. Start with the card's guide, and keep the context at 8192 as in the example.

If you know the field, MTP uses the model's own multi-token prediction head as a draft for speculative decoding. The card notes llama.cpp renamed the flag from mtp to draft-mtp in May 2026, so use the newer name. The base model uses a hybrid of Gated Delta Networks and mixture-of-experts layers.

Build llama.cpp with MTP
Download the GGUF quant
Start llama-server with draft-mtp
Faster text generation

Good to know:

• License: Apache 2.0
• Cost to run: your own hardware, with about 4.6 GB for a 4-bit copy by the record's estimate
• Data it reaches: the files you download and the prompts you send to your local server
• Not for: several parallel requests or the mmproj image file when MTP is on

On Hugging Face, 13 November 2026

• Task: image text to text
• Library: transformers
• Parameters: 9.2B
• Weights: about 4.6 GB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 105,866
• Likes: 230
• License: apache-2.0
• Based on: Qwen/Qwen3.5-9B
• Quantized versions: 1
• Tags: transformers, gguf, unsloth, qwen, qwen3_5, image-text-to-text, conversational

Links

• Model: https://huggingface.co/unsloth/Qwen3.5-9B-MTP-GGUF
• Files: https://huggingface.co/unsloth/Qwen3.5-9B-MTP-GGUF/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download unsloth/Qwen3.5-9B-MTP-GGUF

Then follow the usage in its model card. Nothing in this kit runs by itself.