Qwen3.5-4B-MTP-GGUF

Qwen3.5-4B-MTP-GGUF

Unsloth's GGUF files of Qwen3.5-4B with multi-token prediction, which lets llama.cpp use speculative decoding for roughly 1.5 to 2 times faster generation. For people who run a small multimodal model locally.

Made by unsloth on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This repository holds Unsloth's GGUF files of Qwen3.5-4B that include multi-token prediction, or MTP. With a llama.cpp build that supports MTP, the model drafts several tokens at a time and verifies them together. The card says this makes generation about 1.5 to 2 times faster. The record lists 4.3B parameters and about 2.2 GB at 4-bit.

Who it suits: People who run a small model on their own machine and want faster text generation, and developers who use it as a local backend for coding tools.

Industry | Who uses it | How they use it | Example
Software | Developers with a local model | Serve the model through llama-server | Point a coding tool at a local server
Local AI | Hobbyists with limited hardware | Run a small model faster | Chat with quicker replies on one GPU
Agents | People using Codex or OpenCode | Use the developer role the card says is now supported | Run a coding agent against a local model

You can skip it if you use a llama.cpp version without MTP support, since you would not get the speed-up.

Where to use it:

• A llama.cpp build with MTP support, which the card says was merged on 16 May 2026
• Unsloth, which the card says can also fine-tune the model locally
• Windows, macOS or Linux, with CUDA, CPU or Metal builds

Use case | Who | What they get back
Faster local generation | Anyone running llama-server | Text replies produced with speculative decoding
Tool calling | Agent builders | Better handling of nested objects in tool calls
Coding agents | Developers | A model that accepts the developer role

How to set it up: You need to build llama.cpp with MTP support and download the model through it. The card's build steps are for Linux with CUDA. Set CUDA to OFF for CPU or Metal.

• Python and pip, to download files yourself
• A C++ build toolchain and CMake
• A GPU for the CUDA build, or use CPU or Metal

1. Clone the llama.cpp repository.
2. Build it with the commands below.
3. Start llama-server with the model and MTP settings.

apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp

To fetch the repository files yourself, Hugging Face gives these steps:

pip install -U huggingface_hub
hf download unsloth/Qwen3.5-4B-MTP-GGUF

How to use it: Start llama-server with the model from this repository and the MTP drafting options. The card notes that more than one parallel slot and the mmproj option are not yet supported with MTP.

export LLAMA_CACHE="unsloth/Qwen3.5-4B-MTP-GGUF"
./llama.cpp/llama-server \
    -hf unsloth/Qwen3.5-4B-MTP-GGUF:UD-Q4_K_XL \
    -ngl 99 -c 8192 -fa on -np 1 \
    --spec-type draft-mtp --spec-draft-n-max 6

Tool | What it does
llama-server | Serves the model with speculative decoding on
--spec-type draft-mtp | Turns on MTP drafting
--spec-draft-n-max | Sets how many tokens the draft proposes at once
-np 1 | Keeps one parallel slot, as MTP needs

Example: You start the server with the command above and send a chat request from your tool. The model replies with text, produced with MTP drafting. The card claims it generates faster than without MTP, though it gives no timing for this exact setup.

For newcomers and experts: If you are new, build llama.cpp as shown and start the server with the one command, then chat with a short question. The card also links an Unsloth guide for running Qwen3.5.

For experts, MTP is speculative decoding where the model's own extra heads draft tokens that the main model checks together. The card notes that llama.cpp renamed the option from mtp to draft-mtp on 13 May 2026. The trade-off is that the speed-up depends on how often the drafts are accepted.

Prompt in
MTP head drafts several tokens
Main model verifies them together
Accepted tokens out

Good to know:

• Speed: the card claims about 1.5 to 2 times faster generation with MTP
• Limits: parallel slots above one and the mmproj option are not yet supported with MTP
• Quant: UD-Q4_K_XL is the file the sample command loads
• License: Apache 2.0
• Cost to run: no API key or paid service named
• Data it reaches: downloads files from Hugging Face, nothing else is named
• Not for: llama.cpp builds without MTP support, or several parallel requests with MTP

On Hugging Face, 13 November 2026

• Task: image text to text
• Library: transformers
• Parameters: 4.3B
• Weights: about 2.2 GB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 109,500
• Likes: 112
• License: apache-2.0
• Based on: Qwen/Qwen3.5-4B
• Quantized versions: 1
• Tags: transformers, gguf, unsloth, qwen, qwen3_5, image-text-to-text, conversational

Links

• Model: https://huggingface.co/unsloth/Qwen3.5-4B-MTP-GGUF
• Files: https://huggingface.co/unsloth/Qwen3.5-4B-MTP-GGUF/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download unsloth/Qwen3.5-4B-MTP-GGUF

Then follow the usage in its model card. Nothing in this kit runs by itself.