# Quantization explained: what Q4_K_M, GGUF, AWQ and GPTQ mean

> Quantization stores a model's weights with fewer bits, so the file shrinks and fits in less memory, at some cost in accuracy. A GGUF name such as Q4_K_M spells out the method: about 4 bits, the k-quant family and the M mix. Ollama picks Q4_K_M by default when a repository has it. AWQ and GPTQ are separate methods with their own repositories.

Mindket editors · Published 2026-11-06 · Updated 2026-11-06 · https://mindket.com/learn/open-weight-models/quantization-explained

## In short

- Read a file name before you download it: the number after Q is the bits, and the last letter is the mix.
- Start from one 4-bit file, and pull a higher level only if the answers disappoint you.
- Quantize from a 16-bit original, never from a file that is already quantized.
- Check which method a repository uses, because each one has its own library.

## Before you start

Hugging Face's [overview of quantization](https://huggingface.co/docs/transformers/quantization/overview) says it lowers the memory needed to load and use a model by storing the weights in a lower precision, while trying to keep as much accuracy as possible. Weights are usually stored as 16-bit or 32-bit numbers. Many quantized files store them as small integers such as int8 or int4.

The [hardware lesson](https://mindket.com/learn/local-ai/local-llm-hardware) turns bits into gigabytes and has a size table, so this lesson skips the sizes. GGUF is a binary file format built for quick loading and saving, and a GGUF file name carries its quantization level. The lesson covers those names, the cost of a smaller file, the other formats and how to run or make one. Keep the five prompts from lesson 3 close, because they are the test at the end.

## Read a GGUF name: Q4_K_M letter by letter

Quantizers publish one model at several levels, and the level sits in the file name. The llama.cpp [pull request that added the K names](https://github.com/ggml-org/llama.cpp/pull/1684) defines the parts.

````text
Q4_K_M
│ │ └ M: the mix, which tensors get extra bits
│ └── K: the k-quant family, weights in super-blocks
└──── 4: about 4 bits per weight
````

1. Read the number after Q as the rough bits per weight. Q4 is about 4 bits and Q6 about 6.
2. Read K as the k-quant family. Q4_K, its 4-bit type, keeps weights in super-blocks of 8 blocks with 32 weights each, quantizes the scales and mins with 6 bits, and works out at 4.5 bits per weight.
3. Read the last letter as the mix, meaning which tensors get more bits. A tensor is one block of the model's weights. Q4_K_S uses the Q4_K type for all tensors, with one exception the pull request notes: every variant gives the output.weight tensor 6 bits. Q4_K_M uses the 6-bit Q6_K type for half of the attention.wv and feed_forward.w2 tensors and Q4_K for the rest.
4. Treat other prefixes as other families. Hugging Face's [GGUF page](https://huggingface.co/docs/hub/gguf) lists older types without a K, such as the 8-bit Q8_0, as legacy methods that are not widely used today. It lists names such as IQ4_XS, a 4-bit type at 4.25 bits per weight, as types built with an importance matrix.

## What a smaller file costs in accuracy

Perplexity scores how well a model predicts real text, and lower is better. The k-quants pull request measured LLaMA 7B on the wikitext dataset. The 16-bit original scores 5.9066, and the chart shows how much each quantized file adds to that.

Q6_K adds 0.004, and the author notes that its result is within 0.1% of the original. Q4_K_M adds 0.054, and Q2_K adds 0.870, about 16 times as much. The author also found that bigger models do not escape the loss. Relative error went back to the 7B level at 30B and 65B.

LLaMA 7B is a 2023 model, so read the shape as a guide and not as a forecast for the model you run. Perplexity also does not say whether your task still works, and only your own prompts answer that.

**Perplexity added over the 16-bit original, LLaMA 7B** Each file's wikitext perplexity from the pull request minus the 5.9066 of the 16-bit original, with the file size as the pull request writes it (13.0G for the original). Lower is better. The gold bar is Q4_K_M, the level Ollama picks by default.

- Q2_K, 2.67G: +0.870
- Q3_K_S, 2.75G: +0.551
- Q3_K_M, 3.06G: +0.244
- Q3_K_L, 3.35G: +0.180
- Q4_K_S, 3.56G: +0.115
- Q4_K_M, 3.80G: **+0.054**
- Q5_K_S, 4.33G: +0.035
- Q5_K_M, 4.45G: +0.014
- Q6_K, 5.15G: +0.004

Source: [GitHub: llama.cpp pull request 1684, k-quants](https://github.com/ggml-org/llama.cpp/pull/1684) (read 2026-11-06)

## AWQ, GPTQ and the other formats

GGUF is one family. Hugging Face's overview lists many methods. Some need calibration for greater accuracy, and others quantize on the fly as the model loads. Maarten Grootendorst's [video on GPTQ, GGUF and AWQ](https://www.youtube.com/watch?v=mNE_d-C82lI) is about three of them, and the table below compares four.

AWQ, short for activation-aware weight quantization, starts from the finding that not all weights matter equally. [Its 2023 paper](https://huggingface.co/papers/2306.00978) reports that protecting only 1% of the salient weights can greatly reduce quantization error. Hugging Face's [AWQ page](https://huggingface.co/docs/transformers/quantization/awq) says the method keeps a small fraction of the important weights and compresses a model to 4 bits with minimal performance degradation.

GPTQ is a one-shot method. In Hugging Face's [GPTQ page](https://huggingface.co/docs/transformers/quantization/gptq), each row of the weight matrix is quantized independently to find the version that minimizes the error. The weights are stored as int4 and restored to fp16 on the fly during inference. [Its 2022 paper](https://huggingface.co/papers/2210.17323) reports quantizing GPT models with 175 billion parameters in approximately four GPU hours, and the GPTQ page puts a 175B model at about four hours on an NVIDIA A100.

Transformers no longer supports AutoGPTQ, and the docs point to GPT-QModel instead. Installing AutoAWQ downgrades Transformers to version 4.47.1. To tell which method a repository uses, open its config.json and read `quant_method` inside `quantization_config`. An AWQ model shows `"quant_method": "awq"`.

**Four quantization methods side by side**

| | GGUF (llama.cpp) | AWQ | GPTQ (GPT-QModel) | bitsandbytes |
| --- | --- | --- | --- | --- |
| Bits, as the table writes them | 1/8 | 4 | 2/3/4/8 | 4/8 |
| Quantizes the model as it loads | ✗ No | ✗ No | ✗ No | ✓ Yes |
| Runs on a CPU and on NVIDIA GPUs | ✓ Yes | ✓ Yes | ✓ Yes | ✓ Yes |
| Runs on Apple silicon (Metal) | ✓ Yes | ✗ No | ✓ Yes | Marked yellow |
| Fine-tune it with PEFT | ✗ No | ✓ Yes | ✓ Yes | ✓ Yes |

Sources: [Hugging Face Docs: Transformers quantization overview](https://huggingface.co/docs/transformers/quantization/overview) (read 2026-11-06), [Hugging Face Docs: GGUF](https://huggingface.co/docs/hub/gguf) (read 2026-11-06)

[Which Quantization Method is Right for You? (GPTQ vs. GGUF vs. AWQ)](https://www.youtube.com/watch?v=mNE_d-C82lI)

Maarten Grootendorst on YouTube, a video comparing GPTQ, GGUF and AWQ. It is a second view of the methods in the table, and this lesson does not rely on its claims.

> The bits row is copied with the slashes the table uses. Hugging Face's table marks bitsandbytes on Apple silicon in yellow and does not say what yellow means.

## How three makers ship their weights

A few makers quantize before release, so the files you download are the ones they tested. gpt-oss's card says all its evaluations ran with the same MXFP4 quantization. Hugging Face's [GGUF page](https://huggingface.co/docs/hub/gguf) calls MXFP4 4-bit Microscaling Block Floating Point, and its overview lists NVFP4 as a 4-bit method for CUDA GPUs.

| Model | What the card says it ships | What the card says about memory |
| --- | --- | --- |
| gpt-oss-20b | MoE weights post-trained with MXFP4 quantization | Runs within 16GB of memory |
| gpt-oss-120b | The same MXFP4 quantization | Runs on a single 80GB GPU |
| Llama 4 Scout | BF16 weights | Fits within a single H100 GPU with on-the-fly int4 quantization |
| Mistral Large 4 | A separate NVFP4 checkpoint | A smaller memory footprint, faster inference and the same performance, for Blackwell GPUs |

## Pick a file and run it

Ollama can pull a GGUF repository straight from Hugging Face, in the forms its [Ollama page](https://huggingface.co/docs/hub/ollama) shows below. With no tag, Ollama uses Q4_K_M when the repository has one and otherwise picks one reasonable type from the repository. The tag is not case-sensitive, and a full file name works too.

````bash
ollama run hf.co/{username}/{repository}
ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0
````

1. Find a GGUF repository for the model. Search its name with GGUF, or filter on the `library=gguf` tag.
2. To copy the command, open the GGUF viewer on a file in the Files and versions tab and choose ollama from the Use this model dropdown. The Ollama option has to be on in your Local Apps settings.
3. Start with Q4_K_M. Run your five prompts from lesson 3 on it. If the answers fall short, pull Q5_K_M or Q6_K and run them again.
4. Keep the smallest file that passes. Write down the repository, the file name and the date.

## Quantize a model yourself

Someone has usually quantized the model already, and Hugging Face's GPTQ page advises checking the Hub for an existing version first. For your own GGUF, llama.cpp's [quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) gives two steps: convert the model to a 16-bit GGUF, then run `llama-quantize` on it from the llama.cpp folder.

Start from the 16-bit file. The README warns that the `--allow-requantize` flag, which re-quantizes tensors that are already quantized, can severely reduce quality compared with starting from 16-bit or 32-bit. It also names an importance matrix, passed with `--imatrix`, as a way to cut the accuracy loss. For a build with no setup, the GGUF-my-repo space on Hugging Face makes quants for you.

````bash
# 1. Convert the Hugging Face model to a 16-bit GGUF
python convert_hf_to_gguf.py --outfile gemma-4-E2B-it-bf16.gguf --outtype bf16 --remote google/gemma-4-E2B-it

# 2. Quantize that file to Q4_K_M
./build/bin/llama-quantize gemma-4-E2B-it-bf16.gguf gemma-4-E2B-it-Q4_K_M.gguf Q4_K_M
````

> Copied from the quantize README read on 6 November 2026, with its Gemma 4 E2B example. Mindket did not run these commands.

## Common mistakes

- Judging a level by perplexity alone. The chart shows how well a 2023 model predicted wikitext, and it does not show whether your task works.
- Mixing up the formats. A GGUF file, an AWQ repository and a GPTQ repository need different libraries, so check the file extension or `quant_method` before you download.
- Expecting vision to survive conversion. The README says the converter handles the language part, and a model that takes images needs a separate mmproj file.
- Copying a file name from an old tutorial. The K names date from June 2023, and newer types such as IQ4_XS and MXFP4 follow their own rules.

## Try it yourself

- Open a GGUF repository for a model you like and list its files. Write down what each name says about bits and mix.
- Pull the model at Q4_K_M with Ollama and again one level up, then put the same five prompts to both.
- Open the config.json of an AWQ or GPTQ repository and find its `quant_method`.
- Use the chart to work out how much perplexity Q5_K_M saves over Q4_K_M, and decide whether the larger file is worth it for your job.

## Next course: Local AI

You can now read a file name, weigh what a smaller file costs and run one. The [Local AI course](https://mindket.com/learn/local-ai/ollama-tutorial) starts with Ollama, the tool this lesson used, then covers its API, LM Studio and the memory a model needs.

## Check yourself

1. In the file name Q4_K_M, what does the K stand for?

   A. Kilobytes per weight
   B. The k-quant family, whose types store weights in super-blocks
   C. The context length
   D. A key that unlocks the file

   Answer: B. The pull request that added these types describes them as super-blocks made of blocks of weights, and K marks that family.

2. What does Ollama pull when you give a Hugging Face repository no tag?

   A. The smallest file in the repository
   B. Always the 16-bit file
   C. Q4_K_M when the repository has it
   D. Always Q8_0

   Answer: C. Hugging Face's Ollama page says Q4_K_M is the default when it is present, and otherwise Ollama picks one reasonable type.

3. In the chart of perplexity added to LLaMA 7B, which statement holds?

   A. Q4_K_M adds far less perplexity than Q2_K
   B. Q2_K adds the least perplexity
   C. Q6_K adds the most perplexity
   D. Every level adds the same perplexity

   Answer: A. Q4_K_M adds about 0.054 and Q2_K about 0.870, so the 2-bit file loses about 16 times as much.

4. A 4-bit file gives answers that fall short on your prompts. What should you try first?

   A. Re-quantize the 4-bit file to 2 bits
   B. Quantize the same file again with --allow-requantize
   C. Switch to a smaller model
   D. Pull the level above it, such as Q5_K_M, and run your prompts again

   Answer: D. The chart shows Q5_K_M adding less perplexity than Q4_K_M, and the README warns that re-quantizing already quantized tensors can severely reduce quality.

## Frequently asked questions

### Which quantization level should I start with?

Q4_K_M. Ollama picks it when a repository has it and you give no tag, and it adds about 0.05 perplexity to LLaMA 7B in the pull request's table. Move up one level if the answers fall short.

### Does quantization make a model faster?

Often. The README says it can speed up inference, and in the pull request's 7B table every quantized file generated a token faster than the 16-bit one on all three machines it tested.

### Why does one model come in so many files?

Quantizers publish a file per level, so one repository can hold many. Download only the one whose name has the bits and the mix you chose.

### Is an AWQ or GPTQ model better than a GGUF one?

Neither wins in general. They differ in where they run, so pick by your hardware and the tool you use, then test on your prompts.

### Do I need to quantize a model myself?

Usually not. Qwen3.8-27B's model page counted over 1,600 quantized models on 6 November 2026, so check the Hub for the file you need first.

## Sources

- [GitHub: llama.cpp pull request 1684, k-quants](https://github.com/ggml-org/llama.cpp/pull/1684) (read 2026-11-06)
- [GitHub: llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) (read 2026-11-06)
- [Hugging Face Docs: GGUF](https://huggingface.co/docs/hub/gguf) (read 2026-11-06)
- [Hugging Face Docs: Use Ollama with any GGUF model](https://huggingface.co/docs/hub/ollama) (read 2026-11-06)
- [Hugging Face Docs: Transformers quantization overview](https://huggingface.co/docs/transformers/quantization/overview) (read 2026-11-06)
- [Hugging Face Docs: AWQ](https://huggingface.co/docs/transformers/quantization/awq) (read 2026-11-06)
- [Hugging Face Docs: GPTQ](https://huggingface.co/docs/transformers/quantization/gptq) (read 2026-11-06)
- [Hugging Face Papers: AWQ, Activation-aware Weight Quantization](https://huggingface.co/papers/2306.00978) (read 2026-11-06)
- [Hugging Face Papers: GPTQ, Accurate Post-Training Quantization](https://huggingface.co/papers/2210.17323) (read 2026-11-06)
- [Hugging Face: openai/gpt-oss-20b model card](https://huggingface.co/openai/gpt-oss-20b) (read 2026-11-06)
- [GitHub: Meta Llama 4 model card](https://github.com/meta-llama/llama-models/blob/main/models/llama4/MODEL_CARD.md) (read 2026-11-06)
- [Hugging Face: mistralai/Mistral-Large-4-1T-A52B model card](https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B) (read 2026-11-06)
- [Hugging Face: Qwen/Qwen3.8-27B model page](https://huggingface.co/Qwen/Qwen3.8-27B) (read 2026-11-06)
- [YouTube: Which Quantization Method is Right for You? (GPTQ vs. GGUF vs. AWQ)](https://www.youtube.com/watch?v=mNE_d-C82lI) (read 2026-11-06)

[Browse automations](https://mindket.com/automations)