gemma-4-12B-it-qat-GGUF

gemma-4-12B-it-qat-GGUF

Gemma 4 12B in GGUF form, trained with quantization-aware training so a 4-bit copy keeps near-bfloat16 quality. For people running Gemma locally with llama.cpp, with a draft file for faster output.

Made by unsloth on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This repository holds Unsloth's GGUF files of Gemma 4 12B instruction-tuned, built from Google's quantization-aware-trained checkpoint. The card says that training approach keeps quality close to bfloat16 while cutting memory use a lot. The record lists 11.9B parameters.

Who it suits: People who want Gemma 4 on their own machine at a small file size, and who use llama.cpp.

Industry | Who uses it | How they use it | Example
Software | Developers | Run a local assistant through llama-server | A chat endpoint on their own machine
Research | AI researchers | Test a multimodal 12B model locally | Ask questions about an image
Education | Students | Try a modern open model on a laptop GPU | Run Q4 and check answers
IT | Platform teams | Serve a private model for internal use | A private chat tool

You can skip it if you want a hosted chat, or if you use a runtime that cannot load GGUF files.

Where to use it:

• llama.cpp, through its llama-server program
• Unsloth Studio, which the card says can run Gemma 4
• Windows, macOS or Linux, with Python for the download step

Use case | Who | What they get back
Local chat | Developers | Text answers from a 12B model on your hardware
Image questions | Researchers | Answers about an image you give it
Faster generation | Developers | Speculative decoding through the included MTP drafter file

How to set it up:

• Python and pip on your computer
• A recent llama.cpp build, which finds the drafter file by itself
• About 6 GB of space for a 4-bit copy, by the record's estimate

1. Install the Hugging Face hub package.
2. Download the repository with the hf command, or let llama-server fetch it.
3. Start llama-server with the model.

pip install -U huggingface_hub
hf download unsloth/gemma-4-12B-it-qat-GGUF

How to use it: Start llama-server with the Hugging Face repository and the UD-Q4_K_XL quant, then chat with it. The card says a recent llama.cpp finds the MTP drafter on its own, so you pass no draft model.

./build/bin/llama-server \
  -hf unsloth/gemma-4-12B-it-qat-GGUF:UD-Q4_K_XL \
  --spec-type draft-mtp --spec-draft-n-max 4 \
  -ngl 999 -fa on

Example: You start the server and ask a question in the chat page it serves. The drafter proposes tokens ahead of the main model, and the card says the main model checks each one, so the final answer is unchanged.

For newcomers and experts: If you are new, run the single llama-server command above and chat in the page it opens. The draft file is optional speed, and it does not change what you get back.

If you know the field, the card says QAT keeps similar quality to bfloat16 at much lower memory. The MTP drafter shares the target's KV cache, and the target verifies every drafted token. Gemma 4 handles text and image input, with audio on this 12B size, and reads up to 256K tokens.

QAT Gemma 4 12B checkpoint
GGUF Q4 files
llama-server with MTP drafter
Text answers

Good to know:

• License: Apache 2.0
• Cost to run: your own hardware, with about 6 GB for a 4-bit copy
• Data it reaches: the files you download, and prompts to your local server
• Languages: the Gemma 4 family covers more than 140

On Hugging Face, 13 November 2026

• Task: any to any
• Library: transformers
• Parameters: 11.9B
• Weights: about 6 GB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 196,871
• Likes: 617
• License: apache-2.0
• Based on: google/gemma-4-12B-it-qat-q4_0-unquantized
• Quantized versions: 1
• Tags: transformers, gguf, gemma4, unsloth, gemma, google, any-to-any, conversational

Links

• Model: https://huggingface.co/unsloth/gemma-4-12B-it-qat-GGUF
• Files: https://huggingface.co/unsloth/gemma-4-12B-it-qat-GGUF/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download unsloth/gemma-4-12B-it-qat-GGUF

Then follow the usage in its model card. Nothing in this kit runs by itself.