Qwen3.5-4B GGUF

Qwen3.5-4B GGUF

Imatrix GGUF quantizations of Qwen3.5-4B, a 4.3B image-text model with MTP layers, with the 4-bit file near 3 GB. For people who run a small model locally.

Made by bartowski on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This repository holds GGUF quantizations of Qwen3.5-4B from Qwen, a model of about 4.3B parameters that Hugging Face tags as image text to text. A third party made the files with llama.cpp and an importance matrix, so the model runs in local tools. The card also adds MTP layers.

Who it suits: It suits people who want a small multimodal-tagged model on a laptop or a modest desktop and can pick a file that fits their memory. The 4-bit file is about 3 GB.

Industry | Who uses it | How they use it | Example
Local AI | Enthusiasts | Chat with a small model offline | Trying Q4_K_M on a laptop
Software | Developers | Test prompts on a light local model | Checking a system prompt
Education | Students | Learn how quantization trades size for quality | Comparing Q8_0 and Q4_K_M
Speed testing | Engineers | Try MTP drafting in llama.cpp | Comparing speed with and without the flag

If you need strong answers on hard tasks, you can skip a model this small. If you want to send images, first check that your app supports image input for it, since the card shows only a text layout.

Where to use it: The card says you can run the files in several tools. A newly supported model may need an update from the app's developers first.

• llama.cpp and ramalama
• LM Studio, koboldcpp and Jan AI
• Text Generation Web UI and LoLLMs
• Windows, macOS or Linux with Python, to download the files

Use case | Who | What they get back
Local chat | Enthusiasts | Replies from a small model on your own computer
Prompt testing | Developers | Quick answers from a light model
MTP drafting | Engineers | A speculative decoding setup to time in llama.cpp

How to set it up: You need an app that loads GGUF files and memory for the size you choose. Every file here is small, so memory is rarely the limit. Download one file, not the whole repository.

• One of the apps listed above
• A file that fits your memory, with Q4_K_M as the card's default size
• Python, only for the Hugging Face download command

1. Pick a file from the table.
2. Download it from the repository.
3. Load it and use the prompt format below.

Title: Some files in the card
Quant | Size | Use
bf16 | 8.67 GB | Full weights
Q8_0 | 4.62 GB | Top quality, more than most people need
Q6_K | 3.81 GB | Near-perfect quality, recommended
Q5_K_M | 3.44 GB | High quality, recommended
Q4_K_M | 3.01 GB | Good quality, the card's default pick
IQ4_XS | 2.67 GB | Decent quality in a smaller file

pip install -U huggingface_hub
hf download bartowski/Qwen_Qwen3.5-4B-GGUF

How to use it: Wrap each chat in the layout below. The assistant turn opens with a think tag, so the model reasons before it answers.

<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
<think>

The card says MTP layers work in llama.cpp from release b9180. To use them, add this flag when you start the program.

--spec-type draft-mtp

Example: You load the Q4_K_M file in LM Studio, set a short system prompt, and ask for a plan for a two-day trip. The model starts with its think block, then writes the plan in the chat window.

For newcomers and experts: A quantized file stores a model's numbers more coarsely to save space. Start with Q4_K_M and one short prompt, then try a bigger or smaller file if needed.

For experts, the quants use an importance matrix and were built with llama.cpp release b9222. Q6_K_L, Q5_K_L, Q4_K_L and Q3_K_XL keep embedding and output weights at Q8_0. Q4_0 and IQ4_NL offer online repacking for CPU inference.

The original model is converted to GGUF
An importance matrix guides the quantization
Several sizes are written, from bf16 down
You download one file that fits your memory
You load it in an app and send prompts

Good to know: The Q3 file is usable but lower in quality and suits low-memory machines. Hugging Face estimates the 4-bit weights near 2.2 GB, while the Q4_K_M file in the card is 3.01 GB.

Title: File size by quant type
bf16: 8.67 GB
Q8_0: 4.62 GB
Q6_K: 3.81 GB
Q5_K_M: 3.44 GB
Q4_K_M: 3.01 GB
IQ4_XS: 2.67 GB

• License: Apache 2.0, as Hugging Face lists it
• Cost to run: none named
• Data it reaches: your prompts, run locally, and the files you download
• Not for: hard reasoning tasks that need a large model, since this one has about 4.3B parameters

On Hugging Face, 13 November 2026

• Task: image text to text
• Parameters: 4.3B
• Weights: about 2.2 GB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 137,218
• Likes: 85
• License: apache-2.0
• Based on: Qwen/Qwen3.5-4B
• Tags: gguf, image-text-to-text, conversational

Links

• Model: https://huggingface.co/bartowski/Qwen_Qwen3.5-4B-GGUF
• Files: https://huggingface.co/bartowski/Qwen_Qwen3.5-4B-GGUF/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download bartowski/Qwen_Qwen3.5-4B-GGUF

Then follow the usage in its model card. Nothing in this kit runs by itself.