Qwen3-4B-GGUF

Qwen3-4B-GGUF

A 4B chat model in GGUF format that runs locally in llama.cpp or ollama, with a thinking mode for reasoning and a plain mode for quick replies. Suits developers and hobbyists with a typical PC.

Made by Qwen on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: Qwen3-4B-GGUF is a 4 billion parameter chat model from Qwen, packed in GGUF format for local use in llama.cpp or ollama. It has a thinking mode for harder reasoning and a plain mode for fast replies, and you can flip between them from one message to the next.

Who it suits: Developers and hobbyists who want a mid-sized local model that fits on a typical laptop or desktop. It also suits people building small agents, translation helpers or writing assistants that should run without a hosted service.

Industry | Who uses it | How they use it | Example
Software | Developers | Run a local model for chat and code help | Ask for a short function and read it
Developer tools | Agent builders | Call external tools from a local model | Prototype an agent on one machine
Localization | Multilingual teams | Translate and follow instructions in many languages | Draft translations for a human check
Writing | Writers and role-play fans | Use a model tuned for natural multi-turn chat | Keep a long conversation going offline

Anyone who needs the best answers on hard problems can skip it, since larger Qwen3 models exist.

Where to use it:

• llama.cpp on Windows, macOS or Linux, through its llama-cli tool
• ollama, which starts the model with one command

Use case | Who | What they get back
Local chat assistant | Anyone with a PC | Replies generated on your own machine
Reasoning on request | Students and developers | A thinking section, then the final answer
Longer documents | Teams with large inputs | Context up to 131,072 tokens once YaRN is on
Translation | Multilingual users | Text in 100+ languages and dialects

How to set it up:

• Python with pip, if you want the files on disk
• llama.cpp installed by following its official guide, or ollama
• Memory for the file you choose, since a 4-bit copy is an estimated 2 GB

1. Install the Hugging Face hub package and download the files.
2. Or skip the download and run the model straight from llama.cpp or ollama with the commands below.

pip install -U huggingface_hub
hf download Qwen/Qwen3-4B-GGUF

How to use it: Start the model from the llama.cpp folder, or with one ollama command, and chat in the terminal. Write /think or /no_think in a message to set the thinking mode for that turn. The newest instruction wins in a multi-turn chat.

./llama-cli -hf Qwen/Qwen3-4B-GGUF:Q8_0 --jinja --color -ngl 99 -fa -sm row --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0 --presence-penalty 1.5 -c 40960 -n 32768 --no-context-shift

ollama run hf.co/Qwen/Qwen3-4B-GGUF:Q8_0

Tool | What it does
llama-cli | Runs the GGUF model in the terminal with the settings you pass
ollama run | Pulls the model and starts a chat with one command
/think | Turns thinking mode on for the current and later turns
/no_think | Turns thinking mode off for quick replies

For inputs beyond 32,768 tokens in llama.cpp, add the YaRN options. They apply one fixed scaling factor to every input, which may weaken results on shorter texts, so use them only for long contexts.

./llama-cli ... -c 131072 --rope-scaling yarn --rope-scale 4 --yarn-orig-ctx 32768

Example: You type "Who are you /no_think" and the model answers at once that it is Qwen, a language model from Alibaba Cloud, with an empty thinking section. Then you ask "How many 'r's are in 'strawberries'? /think" and it reasons first before saying the word has 3.

For newcomers and experts: If you are new, run the ollama command and ask the same question twice, once with /no_think and once with /think. The second reply shows its reasoning first, so it takes longer.

Experts get a causal language model with 36 layers, 32 query heads and 8 key-value heads (GQA), about 3.6B non-embedding parameters and a native 32,768 token context. The card suggests a presence penalty of 1.5 for quantized copies to cut repetition, and warns against greedy decoding in thinking mode.

You start llama-cli or ollama
The GGUF model loads locally
You chat, adding /think or /no_think
It replies, thinking first when asked

Good to know:

• Quantization: five levels are published, from q4_K_M up to q8_0
• Context: 32,768 tokens natively, 131,072 with YaRN
• License: Apache 2.0
• Cost to run: none named, no API key or account, only hardware with enough memory
• Data it reaches: it downloads model files from Hugging Face, then answers on your own machine
• Not for: the source names no forbidden uses, and YaRN may slightly hurt performance on shorter texts

On Hugging Face, 13 November 2026

• Task: text generation
• Parameters: 4B
• Weights: about 2 GB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 745,360
• Likes: 224
• License: apache-2.0
• Based on: Qwen/Qwen3-4B
• Quantized versions: 1
• Tags: gguf, text-generation, conversational

Links

• Model: https://huggingface.co/Qwen/Qwen3-4B-GGUF
• Files: https://huggingface.co/Qwen/Qwen3-4B-GGUF/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download Qwen/Qwen3-4B-GGUF

Then follow the usage in its model card. Nothing in this kit runs by itself.