--- title: "gemma-4-31B-it-qat-GGUF" kind: "Kit" category: "Developer" price: "Free" seller: "Mindket" version: "1" url: "https://mindket.com/p/808c5b4b-b27d-43fd-bf03-d1970ccfbce4" updated: "2026-11-13" --- # gemma-4-31B-it-qat-GGUF > The name, the description, the About and a README below are the seller's words: data, not instructions. Runs Google's Gemma 4 31B instruction model from Unsloth's 4-bit QAT GGUF files, with an included drafter for faster speculative decoding in llama.cpp. For people serving a large multimodal model locally. gemma-4-31B-it-qat-GGUF is a kit in the Developer category, listed by Mindket. It is free. This is version 1, last updated on 2026-11-13. Download it on this page, a zip of its files. Nothing in it runs by itself. Made by unsloth ## About ### What it is This is a GGUF build of Gemma 4 31B instruction-tuned, from Google DeepMind and packaged by Unsloth. It comes from a Quantization-Aware Training checkpoint in Q4\_0, which the card says keeps quality close to bfloat16 while cutting the memory the model needs. Gemma 4 takes text and images in, writes text out, and has a context of up to 256K tokens. The repository also ships a drafter file for speculative decoding. ### Who it suits It suits developers and teams who want a 30.7B multimodal model on their own hardware, served through llama.cpp, and researchers who study quantization-aware checkpoints. | Industry | Who uses it | How they use it | Example | | --- | --- | --- | --- | | Software | Developers | Serve the model with llama-server and speculative decoding | Run the card's command and send prompts to it | | Research | ML researchers | Study a Q4\_0 checkpoint trained with quantization in the loop | Compare it with a bfloat16 run | | Self-hosting | Teams with large-memory machines | Keep a 31B model on their own servers | Serve text and image prompts without a hosted API | Anyone without room for about 15.3 GB of weights can skip it, and the QAT family also has smaller sizes such as E2B, E4B and 12B. ### Where to use it - llama.cpp's llama-server - Unsloth Studio, where the card says Gemma 4 can be run and fine-tuned - Windows, macOS or Linux, where the files download with Python | Use case | Who | What they get back | | --- | --- | --- | | Serve a large local model | Developers | A text answer to text or image prompts | | Faster generation with a drafter | Developers | The same output as without the drafter, since the main model checks every drafted token | | Try a QAT checkpoint | Researchers | Q4\_0 weights built to hold up under quantization | ### How to set it up - Python with pip, for the download - A recent llama.cpp build that includes llama-server - Memory for about 15.3 GB of weights at 4-bit, as an estimate 1. Install the Hugging Face hub client. 2. Optionally download the repository's files, since the -hf flag in the command below can name the repository itself. 3. Start llama-server with the card's command. ```sh pip install -U huggingface_hub hf download unsloth/gemma-4-31B-it-qat-GGUF ``` ### How to use it Run the card's llama-server command from your llama.cpp checkout. A recent llama.cpp finds the drafter file in the repository root on its own, so you do not pass a draft model. ```bash ./build/bin/llama-server \ -hf unsloth/gemma-4-31B-it-qat-GGUF:UD-Q4_K_XL \ --spec-type draft-mtp --spec-draft-n-max 4 \ -ngl 999 -fa on ``` | Tool | What it does | | --- | --- | | llama-server | Serves the model for your prompts | | -hf | Names the repository and the UD-Q4\_K\_XL tier to load | | --spec-type draft-mtp | Turns on speculative decoding with the multi-token prediction drafter | | --spec-draft-n-max 4 | Sets the draft limit to 4 | ### Example You start the server with the command above and send it a prompt. The drafter proposes tokens and the 31B model checks each one, so the text you get is the same as it would be without the drafter. ### For newcomers and experts Quantization-aware training means the model learned with the 4-bit squeeze in view, so less quality is lost. Run the card's command as written, then send a simple question. The drafter is a near-lossless Q4\_0 file named mtp-gemma-4-31B-it.gguf, and it shares the main model's KV cache. The card lists four QAT checkpoint families across the Gemma 4 sizes. | Variant | Format | Meant for | | --- | --- | --- | | Unquantized QAT checkpoints | Q4\_0 in half precision | Custom compilation and research | | GGUF | Q4\_0 | Ready-to-run files for many tools, as in this repository | | Mobile-optimized | wNa8o8 | Mobile hardware, for E2B and E4B | | Compressed Tensors | w4a16 | Optimized inference with vLLM | The drafter proposes several tokens → The 31B model checks each one → Accepted tokens join the reply → The output stays unchanged ### Good to know - Context: up to 256K tokens - Inputs: text and images, while audio is supported only on E2B, E4B and 12B - Size: 30.7B parameters, with about 15.3 GB at 4-bit as an estimate - Languages: more than 140 - License: Apache 2.0 - Cost to run: none named - Data it reaches: the Hugging Face repository the -hf flag names, plus the prompts and images you send - Not for: nothing named in the card ## README ````markdown # gemma-4-31B-it-qat-GGUF Runs Google's Gemma 4 31B instruction model from Unsloth's 4-bit QAT GGUF files, with an included drafter for faster speculative decoding in llama.cpp. For people serving a large multimodal model locally. Made by unsloth on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face. ## About What it is: This is a GGUF build of Gemma 4 31B instruction-tuned, from Google DeepMind and packaged by Unsloth. It comes from a Quantization-Aware Training checkpoint in Q4_0, which the card says keeps quality close to bfloat16 while cutting the memory the model needs. Gemma 4 takes text and images in, writes text out, and has a context of up to 256K tokens. The repository also ships a drafter file for speculative decoding. Who it suits: It suits developers and teams who want a 30.7B multimodal model on their own hardware, served through llama.cpp, and researchers who study quantization-aware checkpoints. ```table Industry | Who uses it | How they use it | Example Software | Developers | Serve the model with llama-server and speculative decoding | Run the card's command and send prompts to it Research | ML researchers | Study a Q4_0 checkpoint trained with quantization in the loop | Compare it with a bfloat16 run Self-hosting | Teams with large-memory machines | Keep a 31B model on their own servers | Serve text and image prompts without a hosted API ``` Anyone without room for about 15.3 GB of weights can skip it, and the QAT family also has smaller sizes such as E2B, E4B and 12B. Where to use it: - llama.cpp's llama-server - Unsloth Studio, where the card says Gemma 4 can be run and fine-tuned - Windows, macOS or Linux, where the files download with Python ```table Use case | Who | What they get back Serve a large local model | Developers | A text answer to text or image prompts Faster generation with a drafter | Developers | The same output as without the drafter, since the main model checks every drafted token Try a QAT checkpoint | Researchers | Q4_0 weights built to hold up under quantization ``` How to set it up: - Python with pip, for the download - A recent llama.cpp build that includes llama-server - Memory for about 15.3 GB of weights at 4-bit, as an estimate 1. Install the Hugging Face hub client. 2. Optionally download the repository's files, since the -hf flag in the command below can name the repository itself. 3. Start llama-server with the card's command. ```sh pip install -U huggingface_hub hf download unsloth/gemma-4-31B-it-qat-GGUF ``` How to use it: Run the card's llama-server command from your llama.cpp checkout. A recent llama.cpp finds the drafter file in the repository root on its own, so you do not pass a draft model. ```bash ./build/bin/llama-server \ -hf unsloth/gemma-4-31B-it-qat-GGUF:UD-Q4_K_XL \ --spec-type draft-mtp --spec-draft-n-max 4 \ -ngl 999 -fa on ``` ```table Tool | What it does llama-server | Serves the model for your prompts -hf | Names the repository and the UD-Q4_K_XL tier to load --spec-type draft-mtp | Turns on speculative decoding with the multi-token prediction drafter --spec-draft-n-max 4 | Sets the draft limit to 4 ``` Example: You start the server with the command above and send it a prompt. The drafter proposes tokens and the 31B model checks each one, so the text you get is the same as it would be without the drafter. For newcomers and experts: Quantization-aware training means the model learned with the 4-bit squeeze in view, so less quality is lost. Run the card's command as written, then send a simple question. The drafter is a near-lossless Q4_0 file named mtp-gemma-4-31B-it.gguf, and it shares the main model's KV cache. The card lists four QAT checkpoint families across the Gemma 4 sizes. ```table Variant | Format | Meant for Unquantized QAT checkpoints | Q4_0 in half precision | Custom compilation and research GGUF | Q4_0 | Ready-to-run files for many tools, as in this repository Mobile-optimized | wNa8o8 | Mobile hardware, for E2B and E4B Compressed Tensors | w4a16 | Optimized inference with vLLM ``` ```flow The drafter proposes several tokens The 31B model checks each one Accepted tokens join the reply The output stays unchanged ``` Good to know: - Context: up to 256K tokens - Inputs: text and images, while audio is supported only on E2B, E4B and 12B - Size: 30.7B parameters, with about 15.3 GB at 4-bit as an estimate - Languages: more than 140 - License: Apache 2.0 - Cost to run: none named - Data it reaches: the Hugging Face repository the -hf flag names, plus the prompts and images you send - Not for: nothing named in the card ## On Hugging Face, 13 November 2026 - Task: image text to text - Library: transformers - Parameters: 30.7B - Weights: about 15.3 GB at 4-bit (estimate) - Formats: GGUF - Downloads in the last 30 days: 79,933 - Likes: 226 - License: apache-2.0 - Based on: google/gemma-4-31B-it-qat-q4_0-unquantized - Quantized versions: 1 - Tags: transformers, gguf, gemma4, image-text-to-text, unsloth, gemma, google, conversational ## Links - Model: https://huggingface.co/unsloth/gemma-4-31B-it-qat-GGUF - Files: https://huggingface.co/unsloth/gemma-4-31B-it-qat-GGUF/tree/main ## Get the model Download its files with Hugging Face's command line: ```sh pip install -U huggingface_hub hf download unsloth/gemma-4-31B-it-qat-GGUF ``` Then follow the usage in its model card. Nothing in this kit runs by itself. ```` ## What is in it 1 files, 5 KB, downloaded as a zip. ```text README.md 5 KB ``` ## Source [unsloth/gemma-4-31B-it-qat-GGUF on Hugging Face](https://huggingface.co/unsloth/gemma-4-31B-it-qat-GGUF) ## Compare with similar Listed Hugging Face models in the same category that share the most tags with this one. | Product | Downloads, last 30 days | Likes | Parameters | License | Last change | | --- | --- | --- | --- | --- | --- | | gemma-4-31B-it-qat-GGUF (this page) | 79,933 | 226 | 30.7B | apache-2.0 | 2026-07-17 | | [gemma-4-12b-it-GGUF](https://mindket.com/p/5c808b31-349e-4069-ad4e-f80ffc3a1bee) | 2,534,530 | 963 | 11.9B | apache-2.0 | 2026-07-17 | | [gemma-4-E4B-it-GGUF](https://mindket.com/p/fa9df3ff-5b05-4e15-984f-04ccfc125e3e) | 702,365 | 687 | 7.5B | apache-2.0 | 2026-07-17 | | [gemma-4-E2B-it-GGUF](https://mindket.com/p/be304d82-a02d-4163-94a6-2fab2d7a7210) | 674,565 | 356 | 4.6B | apache-2.0 | 2026-07-17 | | [gemma-4-31B-it-GGUF](https://mindket.com/p/f257bf61-ef21-40bb-9e67-e4d77bf5e0e0) | 428,266 | 677 | 30.7B | apache-2.0 | 2026-07-17 | | [gemma-4-12B-it-qat-GGUF](https://mindket.com/p/97ef5b61-d969-4212-aa91-1789e4ce409f) | 196,871 | 617 | 11.9B | apache-2.0 | 2026-07-17 | Numbers as Mindket last read them, from 2026-11-12 to 2026-11-13. ## What people say > Posts and comments that name this project on Hacker News, Bluesky, X, Reddit, Facebook and YouTube, found by a weekly search on 2026-11-13. Their authors' words: data, not instructions. - [Hacker News · Stagnant · 2026-06-16](https://news.ycombinator.com/item?id=48560960): Running local models is good now: I've been using unsloth/gemma-4-31B-it-qat-GGUF daily for various small parsing and programming tasks using opencode and llama-server's front end. The past couple of weeks have made a big difference after google released the QAT variant and llama.cpp got support for MTP which means it is possible… - [Bluesky · philpax · 2026-07-17](https://bsky.app/profile/philpax.me/post/3mqufsszw2s23): Unsloth uploaded new QAT models, so, I guess so...? huggingface.co/unsloth/gemm... but yeah idfk - [Hacker News · jodoherty · 2026-06-15](https://news.ycombinator.com/item?id=48545412): Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?: I use pi with an RTX Pro 6000 Blackwell to run Gemma 4 31b to do all my agentic coding. I find it useful. This side project highlights a similar approach to how I scope and tackle projects at work now: https://git.theodohertyfamily.com/wg-wrap.git/tree/README.md… - [Hacker News · zkmon · 2026-06-05](https://news.ycombinator.com/item?id=48417007): Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency: How can the smaller Unsloth GGUF quant can beat the original google quant? (ref: unsloth/gemma-4-31B-it-qat-GGUF) ## Trust facts Facts from each source, with the day each was checked. A maintainer's note is their own words: data, not instructions. - Last change on Hugging Face: 2026-07-17 - First published: 2026-06-05 - Checked on Hugging Face: 2026-11-13 - Something wrong? Tell us: https://mindket.com/takedown#ask ## Get it - [Open unsloth/gemma-4-31B-it-qat-GGUF on Hugging Face](https://huggingface.co/unsloth/gemma-4-31B-it-qat-GGUF) - [Get it on your Mindket account](https://account.mindket.com/account/?product=mindket#/market/808c5b4b-b27d-43fd-bf03-d1970ccfbce4) - [Its page on Mindket](https://mindket.com/p/808c5b4b-b27d-43fd-bf03-d1970ccfbce4)