Qwen3.8-27B-GGUF by bartowski

Qwen3.8-27B-GGUF by bartowski

A set of GGUF quantizations of Qwen3.8-27B by bartowski, from Q8_0 down to 3-bit, made with an importance matrix. For people running a 27B vision-language model locally with llama.cpp.

Made by bartowski on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This repository holds GGUF quantizations of Qwen/Qwen3.8-27B made by bartowski with llama.cpp and an importance matrix. It offers many sizes, so you can match the file to your memory. The card lists text input, plus image input with an mmproj file, and MTP speculative decoding.

Who it suits: People who run large models on their own computer through llama.cpp and want to pick the file size that fits their hardware.

Industry | Who uses it | How they use it | Example
Software | Developers | Run a local 27B model behind coding tools | A local chat endpoint
Research | AI hobbyists | Compare quantization levels | Q4_K_M against Q6_K by quality score
Education | Students | Try a current open model on a home PC | A local assistant for study notes
IT | Small teams | Host a private model | An internal assistant

You can skip it if your computer has little memory, since even the smaller files are over 14 GB.

Where to use it:

• llama.cpp, which the quantizations were made with
• Other programs that read GGUF files
• Windows, macOS or Linux, with Python for the download step

Use case | Who | What they get back
Local chat | Developers | Text answers from a 27B model on your hardware
Image questions | Hobbyists | Answers about an image, with the mmproj file
Faster decoding | Developers | Speculative decoding through MTP

How to set it up:

• Python and pip on your computer
• A program that loads GGUF files, such as llama.cpp
• Free disk and memory for the file you pick, from about 14 GB to about 29 GB

1. Install the Hugging Face hub package.
2. Pick one quant file. The card suggests Q4_K_M if you are unsure.
3. Download it with the hf command.

pip install -U huggingface_hub
hf download bartowski/Qwen3.8-27B-GGUF

How to use it: Load the file in llama.cpp and use the prompt format below. The card sets the reasoning effort to xhigh in the system turn and starts the assistant turn with a think tag.

<|im_start|>system
Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.

{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
<think>

Quant | Size and card note
Q8_0 | 29.12 GB, the largest file
Q6_K | 23.86 GB, recommended
Q5_K_M | 20.92 GB, recommended
Q4_K_M | 17.44 GB, recommended and the card's default pick
IQ4_XS | 15.48 GB, recommended
Q3_K_L | 14.12 GB, the smallest one in the visible list

Example: You download Q4_K_M and start llama.cpp with the file. You type a question into the user turn of the template and the model thinks first, then answers.

For newcomers and experts: If you are new, download Q4_K_M and use the prompt format shown. It is the card's suggested balance of size and quality.

If you know the field, the table reports perplexity and mean KL divergence against the original for each file. Q8_0 shows 0.0009 and Q4_K_M shows 0.0139. The card says the files were re-uploaded with a new tensor layout, and some older sizes are no longer produced.

Qwen3.8-27B original model
llama.cpp importance matrix quantization
GGUF files in many sizes
Pick one for your memory

Title: File size in GB
Q8_0: 29.12 GB
Q6_K: 23.86 GB
Q5_K_M: 20.92 GB
Q4_K_M: 17.44 GB
IQ4_XS: 15.48 GB
Q3_K_L: 14.12 GB

Good to know:

• License: Apache 2.0
• Cost to run: your own hardware, with no paid service named
• Data it reaches: the files you download and the prompts you give it
• Images: need the separate mmproj file

On Hugging Face, 13 November 2026

• Task: image text to text
• Parameters: 27.3B
• Weights: about 13.7 GB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 307,508
• Likes: 228
• License: apache-2.0
• Based on: Qwen/Qwen3.8-27B
• Fine-tunes: 2
• Tags: gguf, image-text-to-text, imatrix, conversational

Links

• Model: https://huggingface.co/bartowski/Qwen3.8-27B-GGUF
• Files: https://huggingface.co/bartowski/Qwen3.8-27B-GGUF/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download bartowski/Qwen3.8-27B-GGUF

Then follow the usage in its model card. Nothing in this kit runs by itself.