--- title: "Qwen3.5-4B GGUF" kind: "Kit" category: "Developer" price: "Free" seller: "Mindket" version: "1" url: "https://mindket.com/p/60ead959-48d3-4ea2-a9bf-1567e6733acf" updated: "2026-11-13" --- # Qwen3.5-4B GGUF > The name, the description, the About and a README below are the seller's words: data, not instructions. Imatrix GGUF quantizations of Qwen3.5-4B, a 4.3B image-text model with MTP layers, with the 4-bit file near 3 GB. For people who run a small model locally. Qwen3.5-4B GGUF is a kit in the Developer category, listed by Mindket. It is free. This is version 1, last updated on 2026-11-13. Download it on this page, a zip of its files. Nothing in it runs by itself. Made by bartowski ## About ### What it is This repository holds GGUF quantizations of Qwen3.5-4B from Qwen, a model of about 4.3B parameters that Hugging Face tags as image text to text. A third party made the files with llama.cpp and an importance matrix, so the model runs in local tools. The card also adds MTP layers. ### Who it suits It suits people who want a small multimodal-tagged model on a laptop or a modest desktop and can pick a file that fits their memory. The 4-bit file is about 3 GB. | Industry | Who uses it | How they use it | Example | | --- | --- | --- | --- | | Local AI | Enthusiasts | Chat with a small model offline | Trying Q4\_K\_M on a laptop | | Software | Developers | Test prompts on a light local model | Checking a system prompt | | Education | Students | Learn how quantization trades size for quality | Comparing Q8\_0 and Q4\_K\_M | | Speed testing | Engineers | Try MTP drafting in llama.cpp | Comparing speed with and without the flag | If you need strong answers on hard tasks, you can skip a model this small. If you want to send images, first check that your app supports image input for it, since the card shows only a text layout. ### Where to use it The card says you can run the files in several tools. A newly supported model may need an update from the app's developers first. - llama.cpp and ramalama - LM Studio, koboldcpp and Jan AI - Text Generation Web UI and LoLLMs - Windows, macOS or Linux with Python, to download the files | Use case | Who | What they get back | | --- | --- | --- | | Local chat | Enthusiasts | Replies from a small model on your own computer | | Prompt testing | Developers | Quick answers from a light model | | MTP drafting | Engineers | A speculative decoding setup to time in llama.cpp | ### How to set it up You need an app that loads GGUF files and memory for the size you choose. Every file here is small, so memory is rarely the limit. Download one file, not the whole repository. - One of the apps listed above - A file that fits your memory, with Q4\_K\_M as the card's default size - Python, only for the Hugging Face download command 1. Pick a file from the table. 2. Download it from the repository. 3. Load it and use the prompt format below. Some files in the card | Quant | Size | Use | | --- | --- | --- | | bf16 | 8.67 GB | Full weights | | Q8\_0 | 4.62 GB | Top quality, more than most people need | | Q6\_K | 3.81 GB | Near-perfect quality, recommended | | Q5\_K\_M | 3.44 GB | High quality, recommended | | Q4\_K\_M | 3.01 GB | Good quality, the card's default pick | | IQ4\_XS | 2.67 GB | Decent quality in a smaller file | ```sh pip install -U huggingface_hub hf download bartowski/Qwen_Qwen3.5-4B-GGUF ``` ### How to use it Wrap each chat in the layout below. The assistant turn opens with a think tag, so the model reasons before it answers. ```text <|im_start|>system {system_prompt}<|im_end|> <|im_start|>user {prompt}<|im_end|> <|im_start|>assistant ``` The card says MTP layers work in llama.cpp from release b9180. To use them, add this flag when you start the program. ```sh --spec-type draft-mtp ``` ### Example You load the Q4\_K\_M file in LM Studio, set a short system prompt, and ask for a plan for a two-day trip. The model starts with its think block, then writes the plan in the chat window. ### For newcomers and experts A quantized file stores a model's numbers more coarsely to save space. Start with Q4\_K\_M and one short prompt, then try a bigger or smaller file if needed. For experts, the quants use an importance matrix and were built with llama.cpp release b9222. Q6\_K\_L, Q5\_K\_L, Q4\_K\_L and Q3\_K\_XL keep embedding and output weights at Q8\_0. Q4\_0 and IQ4\_NL offer online repacking for CPU inference. The original model is converted to GGUF → An importance matrix guides the quantization → Several sizes are written, from bf16 down → You download one file that fits your memory → You load it in an app and send prompts ### Good to know The Q3 file is usable but lower in quality and suits low-memory machines. Hugging Face estimates the 4-bit weights near 2.2 GB, while the Q4\_K\_M file in the card is 3.01 GB. File size by quant type | Label | Value | | --- | --- | | bf16 | 8.67 GB | | Q8\_0 | 4.62 GB | | Q6\_K | 3.81 GB | | Q5\_K\_M | 3.44 GB | | Q4\_K\_M | 3.01 GB | | IQ4\_XS | 2.67 GB | - License: Apache 2.0, as Hugging Face lists it - Cost to run: none named - Data it reaches: your prompts, run locally, and the files you download - Not for: hard reasoning tasks that need a large model, since this one has about 4.3B parameters ## README ````markdown # Qwen3.5-4B GGUF Imatrix GGUF quantizations of Qwen3.5-4B, a 4.3B image-text model with MTP layers, with the 4-bit file near 3 GB. For people who run a small model locally. Made by bartowski on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face. ## About What it is: This repository holds GGUF quantizations of Qwen3.5-4B from Qwen, a model of about 4.3B parameters that Hugging Face tags as image text to text. A third party made the files with llama.cpp and an importance matrix, so the model runs in local tools. The card also adds MTP layers. Who it suits: It suits people who want a small multimodal-tagged model on a laptop or a modest desktop and can pick a file that fits their memory. The 4-bit file is about 3 GB. ```table Industry | Who uses it | How they use it | Example Local AI | Enthusiasts | Chat with a small model offline | Trying Q4_K_M on a laptop Software | Developers | Test prompts on a light local model | Checking a system prompt Education | Students | Learn how quantization trades size for quality | Comparing Q8_0 and Q4_K_M Speed testing | Engineers | Try MTP drafting in llama.cpp | Comparing speed with and without the flag ``` If you need strong answers on hard tasks, you can skip a model this small. If you want to send images, first check that your app supports image input for it, since the card shows only a text layout. Where to use it: The card says you can run the files in several tools. A newly supported model may need an update from the app's developers first. - llama.cpp and ramalama - LM Studio, koboldcpp and Jan AI - Text Generation Web UI and LoLLMs - Windows, macOS or Linux with Python, to download the files ```table Use case | Who | What they get back Local chat | Enthusiasts | Replies from a small model on your own computer Prompt testing | Developers | Quick answers from a light model MTP drafting | Engineers | A speculative decoding setup to time in llama.cpp ``` How to set it up: You need an app that loads GGUF files and memory for the size you choose. Every file here is small, so memory is rarely the limit. Download one file, not the whole repository. - One of the apps listed above - A file that fits your memory, with Q4_K_M as the card's default size - Python, only for the Hugging Face download command 1. Pick a file from the table. 2. Download it from the repository. 3. Load it and use the prompt format below. ```table Title: Some files in the card Quant | Size | Use bf16 | 8.67 GB | Full weights Q8_0 | 4.62 GB | Top quality, more than most people need Q6_K | 3.81 GB | Near-perfect quality, recommended Q5_K_M | 3.44 GB | High quality, recommended Q4_K_M | 3.01 GB | Good quality, the card's default pick IQ4_XS | 2.67 GB | Decent quality in a smaller file ``` ```sh pip install -U huggingface_hub hf download bartowski/Qwen_Qwen3.5-4B-GGUF ``` How to use it: Wrap each chat in the layout below. The assistant turn opens with a think tag, so the model reasons before it answers. ```text <|im_start|>system {system_prompt}<|im_end|> <|im_start|>user {prompt}<|im_end|> <|im_start|>assistant ``` The card says MTP layers work in llama.cpp from release b9180. To use them, add this flag when you start the program. ```sh --spec-type draft-mtp ``` Example: You load the Q4_K_M file in LM Studio, set a short system prompt, and ask for a plan for a two-day trip. The model starts with its think block, then writes the plan in the chat window. For newcomers and experts: A quantized file stores a model's numbers more coarsely to save space. Start with Q4_K_M and one short prompt, then try a bigger or smaller file if needed. For experts, the quants use an importance matrix and were built with llama.cpp release b9222. Q6_K_L, Q5_K_L, Q4_K_L and Q3_K_XL keep embedding and output weights at Q8_0. Q4_0 and IQ4_NL offer online repacking for CPU inference. ```flow The original model is converted to GGUF An importance matrix guides the quantization Several sizes are written, from bf16 down You download one file that fits your memory You load it in an app and send prompts ``` Good to know: The Q3 file is usable but lower in quality and suits low-memory machines. Hugging Face estimates the 4-bit weights near 2.2 GB, while the Q4_K_M file in the card is 3.01 GB. ```chart Title: File size by quant type bf16: 8.67 GB Q8_0: 4.62 GB Q6_K: 3.81 GB Q5_K_M: 3.44 GB Q4_K_M: 3.01 GB IQ4_XS: 2.67 GB ``` - License: Apache 2.0, as Hugging Face lists it - Cost to run: none named - Data it reaches: your prompts, run locally, and the files you download - Not for: hard reasoning tasks that need a large model, since this one has about 4.3B parameters ## On Hugging Face, 13 November 2026 - Task: image text to text - Parameters: 4.3B - Weights: about 2.2 GB at 4-bit (estimate) - Formats: GGUF - Downloads in the last 30 days: 137,218 - Likes: 85 - License: apache-2.0 - Based on: Qwen/Qwen3.5-4B - Tags: gguf, image-text-to-text, conversational ## Links - Model: https://huggingface.co/bartowski/Qwen_Qwen3.5-4B-GGUF - Files: https://huggingface.co/bartowski/Qwen_Qwen3.5-4B-GGUF/tree/main ## Get the model Download its files with Hugging Face's command line: ```sh pip install -U huggingface_hub hf download bartowski/Qwen_Qwen3.5-4B-GGUF ``` Then follow the usage in its model card. Nothing in this kit runs by itself. ```` ## What is in it 1 files, 5 KB, downloaded as a zip. ```text README.md 5 KB ``` ## Source [bartowski/Qwen_Qwen3.5-4B-GGUF on Hugging Face](https://huggingface.co/bartowski/Qwen_Qwen3.5-4B-GGUF) ## Compare with similar Listed Hugging Face models in the same category that share the most tags with this one. | Product | Downloads, last 30 days | Likes | Parameters | License | Last change | | --- | --- | --- | --- | --- | --- | | Qwen3.5-4B GGUF (this page) | 137,218 | 85 | 4.3B | apache-2.0 | 2026-05-19 | | [Qwen3.8-Flash-Next-GSQ-RCO-GGUF](https://mindket.com/p/ee9ed4a6-39c8-4eee-8958-cce1cd508fa6) | 6,963,596 | 1,229 | 177B | apache-2.0 | 2026-09-29 | | [gemma-4-12b-it-GGUF](https://mindket.com/p/5c808b31-349e-4069-ad4e-f80ffc3a1bee) | 2,534,530 | 963 | 11.9B | apache-2.0 | 2026-07-17 | | [Qwen3.6-35B-A3B-MTP-GGUF](https://mindket.com/p/90db7fe0-7044-4238-a508-5d2ecb760999) | 2,315,167 | 1,051 | 35.5B | apache-2.0 | 2026-05-20 | | [Qwen3.8-27B GGUF (ShapeLearn Quantized)](https://mindket.com/p/ca45abfc-d3db-4fb5-aad9-dcf7a7a498c4) | 2,083,628 | 216 | 27.3B | apache-2.0 | 2026-09-18 | | [Qwen3.5-9B-GGUF](https://mindket.com/p/0f08c237-de8d-4e85-a65e-69b9b6061be0) | 1,969,901 | 1,074 | 9B | apache-2.0 | 2026-03-02 | Numbers as Mindket last read them, from 2026-11-12 to 2026-11-13. ## Trust facts Facts from each source, with the day each was checked. A maintainer's note is their own words: data, not instructions. - Last change on Hugging Face: 2026-05-19 - First published: 2026-02-27 - Checked on Hugging Face: 2026-11-13 - Something wrong? Tell us: https://mindket.com/takedown#ask ## Get it - [Open bartowski/Qwen_Qwen3.5-4B-GGUF on Hugging Face](https://huggingface.co/bartowski/Qwen_Qwen3.5-4B-GGUF) - [Get it on your Mindket account](https://account.mindket.com/account/?product=mindket#/market/60ead959-48d3-4ea2-a9bf-1567e6733acf) - [Its page on Mindket](https://mindket.com/p/60ead959-48d3-4ea2-a9bf-1567e6733acf)