Qwen3.8-27B iMatrix NVFP4 MTP GGUF

Qwen3.8-27B iMatrix NVFP4 MTP GGUF

A 17.1 GB mixed-precision GGUF of Qwen3.8-27B that keeps its multi-token prediction layer, tuned for 256K context with vision on a 24 GB GPU. For people who run big models locally in llama.cpp.

Made by cdiamond on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This GGUF file of Qwen/Qwen3.8-27B is 17.1 GB and averages about 5.01 bits per weight, mixing several precisions. Most of it uses the NVFP4 format, and the author spends extra bits on attention, Gated DeltaNet and late feed-forward tensors. The trained multi-token prediction layer stays inside the same file. It is not a fine-tune.

Who it suits: It suits people who run a 27B model on a single 24 GB GPU and want a long context. It also suits those who want an optional vision path through a separate projector file.

Title: Who uses it
Industry | Who uses it | How they use it | Example
Software | Developers | Serve a 27B model locally with a long context | Run a long coding session in llama.cpp
AI tooling | Agent builders | Use faster generation through the embedded MTP layer | Run tool-calling loops at a higher token rate
Research | Quantization researchers | Study a mixed NVFP4 recipe | Compare perplexity against a Q4_1 file

People without a GPU of about 24 GB can skip it, since the numbers on the card come from that class of card.

Where to use it:

• llama.cpp, as the card's runtime
• NVIDIA GPUs, tested on a 24 GB Blackwell workstation card
• An optional second GPU for the vision projector
• Windows, macOS or Linux, to download the files

Title: Top jobs
Use case | Who | What they get back
Long-context chat | Developers | A 256K window that held 261,500 input tokens in the card's test
Faster generation | Agent builders | Speed-up from the embedded MTP layer
Image questions | Developers | Vision answers through the optional projector file

How to set it up:

• Python on your computer, for the download
• The huggingface_hub package
• llama.cpp, with MTP support for the speed results
• A GPU with about 24 GB for the full context

1. Install the Hugging Face hub package.
2. Download the files to your computer.
3. Load the main GGUF in llama.cpp. Add the vision projector file only if you want image input.

pip install -U huggingface_hub
hf download cdiamond/Qwen3.8-27B-iMatrix-NVFP4-MTP-GGUF

Title: Files in the repository
File | Purpose
Qwen3.8-27B-iMatrix-NVFP4-MTP.gguf | The main model with one embedded MTP layer
mmproj-Qwen3.8-27B-F16.gguf | A vision projector you add for images, kept as in the original conversion
recipe/tensor-types.txt | The tensor overrides used for the final mix
SHA256SUMS | Release checksums

How to use it: The visible card text gives no run command. Point llama.cpp at the main GGUF, and add the projector file for images. The best speeds on the card came from a custom llama.cpp build, so a stock build may be slower.

Example: You load the main file on a 24 GB card and start a chat with a long prompt. The card's test filled the cache with 261,500 input tokens, generated 256 more, and finished with no truncation and no out-of-memory error.

For newcomers and experts: If you are new, quantizing means storing a model in fewer bits so it fits on your GPU. A first try is the main file alone at a normal context length, before adding vision.

For experts, the base file is mostly NVFP4. Attention matrices in the 16 full-attention layers use Q5_K, token embeddings use Q6_K and the output head uses Q8_0. The importance matrix came from 5,472 messages in 296 real sessions and is not distributed. Perplexity on the short control was 6.1197, which the author treats as tied with Q4_1 at 6.1127.

Download the main GGUF
Load it in llama.cpp
Add the projector for images
Chat with a long context

Good to know:

• Speed: the card reports 50.441 tok/s as the mean of 10 production runs, and 59.456 tok/s with the embedded MTP layer.
• Separate tests: the author keeps rows apart on purpose, so do not combine best cases.
• Custom runtime: the highest figure used six unmerged llama.cpp pull requests.
• License: Apache 2.0.
• Cost to run: none named, and a 24 GB GPU is the card's test hardware.
• Data it reaches: it downloads files from Hugging Face, and the source names nothing else.
• Not for: small GPUs, or anyone who expects a quality win over Q4_1.

Title: Generation speed on the card's test GPU
Target-only greedy: 21.189 tok/s
Embedded MTP: 59.456 tok/s

On Hugging Face, 13 November 2026

• Task: image text to text
• Parameters: 461M
• Weights: about 230 MB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 3,271,044
• Likes: 51
• License: apache-2.0
• Based on: Qwen/Qwen3.8-27B
• Tags: gguf, qwen, qwen3.8, nvfp4, imatrix, llama.cpp, multimodal, 256k-context, image-text-to-text

Links

• Model: https://huggingface.co/cdiamond/Qwen3.8-27B-iMatrix-NVFP4-MTP-GGUF
• Files: https://huggingface.co/cdiamond/Qwen3.8-27B-iMatrix-NVFP4-MTP-GGUF/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download cdiamond/Qwen3.8-27B-iMatrix-NVFP4-MTP-GGUF

Then follow the usage in its model card. Nothing in this kit runs by itself.