Qwen3.5-0.8B-MTP-GGUF

Qwen3.5-0.8B-MTP-GGUF

Runs the small Qwen3.5 0.8B vision-language model in llama.cpp with multi-token prediction for faster generation, as Unsloth's GGUF files. For developers testing speculative decoding.

Made by unsloth on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This repository holds Unsloth's GGUF files for Qwen3.5-0.8B with multi-token prediction, or MTP, so llama.cpp can use speculative decoding. Qwen3.5-0.8B is a small vision-language model, listed at 773M parameters. Unsloth says MTP makes generation roughly 1.5 to 2 times faster.

Who it suits: It suits developers who want to try speculative decoding on a tiny model, and people who prototype or fine-tune small models. The Qwen team says a model this small is meant for prototyping, task-specific fine-tuning and research.

Industry | Who uses it | How they use it | Example
Software | Developers | Test MTP speculative decoding in llama-server | Start the card's server command and send a prompt
Research | ML researchers | Study a small multimodal model | Compare generation speed with and without MTP
Product teams | Prototypers | Prototype a feature on a model that is only about 386 MB at 4-bit | Run a task-specific prototype locally

Anyone who needs a large model's depth, or who needs image input through mmproj, can skip it, because MTP does not support mmproj yet.

Where to use it:

• llama.cpp, built with MTP support
• A CUDA GPU build, or a CPU or Metal build with CUDA switched off
• Codex and OpenCode, since the card adds developer role support for them
• Windows, macOS or Linux, where the files download with Python

Use case | Who | What they get back
Faster local generation | Developers | Text produced about 1.5 to 2 times faster, per Unsloth
Tool calling | Agent builders | Better parsing of nested objects in tool calls
Local fine-tuning | Researchers | A small base to adapt with Unsloth

How to set it up:

• Python with pip, for the download
• Build tools and CMake, plus the CUDA toolkit for a GPU build
• A llama.cpp build that includes MTP, which the card says has been merged

1. Install the build tools.
2. Clone the llama.cpp repository.
3. Configure and build llama.cpp, then copy the binaries beside it.
4. Start llama-server with the MTP flags.

apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp

pip install -U huggingface_hub
hf download unsloth/Qwen3.5-0.8B-MTP-GGUF

How to use it: Run llama-server with the card's command, which names the repository and the UD-Q4_K_XL tier. Set the CUDA option to OFF in the build for CPU or Metal. llama.cpp renamed the speculative type from mtp to draft-mtp, so use the new name.

export LLAMA_CACHE="unsloth/Qwen3.5-0.8B-MTP-GGUF"
./llama.cpp/llama-server \
    -hf unsloth/Qwen3.5-0.8B-MTP-GGUF:UD-Q4_K_XL \
    -ngl 99 -c 8192 -fa on -np 1 \
    --spec-type draft-mtp --spec-draft-n-max 6

Tool | What it does
--spec-type draft-mtp | Turns on MTP speculative decoding
--spec-draft-n-max 6 | Sets the draft limit to 6
-np 1 | Keeps one slot, since values above 1 are not yet supported with MTP
--mmproj | Not yet supported together with MTP

Example: You start llama-server with the command above and send it a short prompt. The card says MTP speeds up generation by roughly 1.5 to 2 times.

For newcomers and experts: This is a tiny model, so it fits on modest hardware, but the setup builds llama.cpp from source. Run the build once, start the server and ask a short question.

Qwen3.5 pairs Gated Delta Networks with sparse mixture-of-experts layers and is trained on multimodal tokens from the start. MTP lets the model draft several tokens ahead, which llama.cpp then checks.

Build llama.cpp with MTP support
Start llama-server with draft-mtp
The model drafts several tokens
The drafts are checked and kept
Text comes out faster

Good to know:

• Limits: MTP cannot yet be combined with more than one slot or with mmproj
• Languages: the Qwen3.5 family covers 201 languages and dialects
• Size: 773M parameters, with about 386 MB at 4-bit as an estimate
• License: Apache 2.0
• Cost to run: none named
• Data it reaches: the prompts you send to your own server, and the -hf flag pulls the files from the Hugging Face repository
• Not for: batching with np above 1 or image input through mmproj, while MTP is on

On Hugging Face, 13 November 2026

• Task: image text to text
• Library: transformers
• Parameters: 773M
• Weights: about 386 MB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 14,886
• Likes: 43
• License: apache-2.0
• Based on: Qwen/Qwen3.5-0.8B
• Tags: transformers, gguf, unsloth, qwen, qwen3_5, image-text-to-text, conversational

Links

• Model: https://huggingface.co/unsloth/Qwen3.5-0.8B-MTP-GGUF
• Files: https://huggingface.co/unsloth/Qwen3.5-0.8B-MTP-GGUF/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download unsloth/Qwen3.5-0.8B-MTP-GGUF

Then follow the usage in its model card. Nothing in this kit runs by itself.