gemma-4-31B-it-qat-GGUF

gemma-4-31B-it-qat-GGUF

Runs Google's Gemma 4 31B instruction model from Unsloth's 4-bit QAT GGUF files, with an included drafter for faster speculative decoding in llama.cpp. For people serving a large multimodal model locally.

Made by unsloth on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This is a GGUF build of Gemma 4 31B instruction-tuned, from Google DeepMind and packaged by Unsloth. It comes from a Quantization-Aware Training checkpoint in Q4_0, which the card says keeps quality close to bfloat16 while cutting the memory the model needs. Gemma 4 takes text and images in, writes text out, and has a context of up to 256K tokens. The repository also ships a drafter file for speculative decoding.

Who it suits: It suits developers and teams who want a 30.7B multimodal model on their own hardware, served through llama.cpp, and researchers who study quantization-aware checkpoints.

Industry | Who uses it | How they use it | Example
Software | Developers | Serve the model with llama-server and speculative decoding | Run the card's command and send prompts to it
Research | ML researchers | Study a Q4_0 checkpoint trained with quantization in the loop | Compare it with a bfloat16 run
Self-hosting | Teams with large-memory machines | Keep a 31B model on their own servers | Serve text and image prompts without a hosted API

Anyone without room for about 15.3 GB of weights can skip it, and the QAT family also has smaller sizes such as E2B, E4B and 12B.

Where to use it:

• llama.cpp's llama-server
• Unsloth Studio, where the card says Gemma 4 can be run and fine-tuned
• Windows, macOS or Linux, where the files download with Python

Use case | Who | What they get back
Serve a large local model | Developers | A text answer to text or image prompts
Faster generation with a drafter | Developers | The same output as without the drafter, since the main model checks every drafted token
Try a QAT checkpoint | Researchers | Q4_0 weights built to hold up under quantization

How to set it up:

• Python with pip, for the download
• A recent llama.cpp build that includes llama-server
• Memory for about 15.3 GB of weights at 4-bit, as an estimate

1. Install the Hugging Face hub client.
2. Optionally download the repository's files, since the -hf flag in the command below can name the repository itself.
3. Start llama-server with the card's command.

pip install -U huggingface_hub
hf download unsloth/gemma-4-31B-it-qat-GGUF

How to use it: Run the card's llama-server command from your llama.cpp checkout. A recent llama.cpp finds the drafter file in the repository root on its own, so you do not pass a draft model.

./build/bin/llama-server \
  -hf unsloth/gemma-4-31B-it-qat-GGUF:UD-Q4_K_XL \
  --spec-type draft-mtp --spec-draft-n-max 4 \
  -ngl 999 -fa on

Tool | What it does
llama-server | Serves the model for your prompts
-hf | Names the repository and the UD-Q4_K_XL tier to load
--spec-type draft-mtp | Turns on speculative decoding with the multi-token prediction drafter
--spec-draft-n-max 4 | Sets the draft limit to 4

Example: You start the server with the command above and send it a prompt. The drafter proposes tokens and the 31B model checks each one, so the text you get is the same as it would be without the drafter.

For newcomers and experts: Quantization-aware training means the model learned with the 4-bit squeeze in view, so less quality is lost. Run the card's command as written, then send a simple question.

The drafter is a near-lossless Q4_0 file named mtp-gemma-4-31B-it.gguf, and it shares the main model's KV cache. The card lists four QAT checkpoint families across the Gemma 4 sizes.

Variant | Format | Meant for
Unquantized QAT checkpoints | Q4_0 in half precision | Custom compilation and research
GGUF | Q4_0 | Ready-to-run files for many tools, as in this repository
Mobile-optimized | wNa8o8 | Mobile hardware, for E2B and E4B
Compressed Tensors | w4a16 | Optimized inference with vLLM

The drafter proposes several tokens
The 31B model checks each one
Accepted tokens join the reply
The output stays unchanged

Good to know:

• Context: up to 256K tokens
• Inputs: text and images, while audio is supported only on E2B, E4B and 12B
• Size: 30.7B parameters, with about 15.3 GB at 4-bit as an estimate
• Languages: more than 140
• License: Apache 2.0
• Cost to run: none named
• Data it reaches: the Hugging Face repository the -hf flag names, plus the prompts and images you send
• Not for: nothing named in the card

On Hugging Face, 13 November 2026

• Task: image text to text
• Library: transformers
• Parameters: 30.7B
• Weights: about 15.3 GB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 79,933
• Likes: 226
• License: apache-2.0
• Based on: google/gemma-4-31B-it-qat-q4_0-unquantized
• Quantized versions: 1
• Tags: transformers, gguf, gemma4, image-text-to-text, unsloth, gemma, google, conversational

Links

• Model: https://huggingface.co/unsloth/gemma-4-31B-it-qat-GGUF
• Files: https://huggingface.co/unsloth/gemma-4-31B-it-qat-GGUF/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download unsloth/gemma-4-31B-it-qat-GGUF

Then follow the usage in its model card. Nothing in this kit runs by itself.