--- title: "gemma-4-12B-it-qat-GGUF" kind: "Kit" category: "Developer" price: "Free" seller: "Mindket" version: "1" url: "https://mindket.com/p/97ef5b61-d969-4212-aa91-1789e4ce409f" updated: "2026-11-12" --- # gemma-4-12B-it-qat-GGUF > The name, the description, the About and a README below are the seller's words: data, not instructions. Gemma 4 12B in GGUF form, trained with quantization-aware training so a 4-bit copy keeps near-bfloat16 quality. For people running Gemma locally with llama.cpp, with a draft file for faster output. gemma-4-12B-it-qat-GGUF is a kit in the Developer category, listed by Mindket. It is free. This is version 1, last updated on 2026-11-12. Download it on this page, a zip of its files. Nothing in it runs by itself. Made by unsloth ## About ### What it is This repository holds Unsloth's GGUF files of Gemma 4 12B instruction-tuned, built from Google's quantization-aware-trained checkpoint. The card says that training approach keeps quality close to bfloat16 while cutting memory use a lot. The record lists 11.9B parameters. ### Who it suits People who want Gemma 4 on their own machine at a small file size, and who use llama.cpp. | Industry | Who uses it | How they use it | Example | | --- | --- | --- | --- | | Software | Developers | Run a local assistant through llama-server | A chat endpoint on their own machine | | Research | AI researchers | Test a multimodal 12B model locally | Ask questions about an image | | Education | Students | Try a modern open model on a laptop GPU | Run Q4 and check answers | | IT | Platform teams | Serve a private model for internal use | A private chat tool | You can skip it if you want a hosted chat, or if you use a runtime that cannot load GGUF files. ### Where to use it - llama.cpp, through its llama-server program - Unsloth Studio, which the card says can run Gemma 4 - Windows, macOS or Linux, with Python for the download step | Use case | Who | What they get back | | --- | --- | --- | | Local chat | Developers | Text answers from a 12B model on your hardware | | Image questions | Researchers | Answers about an image you give it | | Faster generation | Developers | Speculative decoding through the included MTP drafter file | ### How to set it up - Python and pip on your computer - A recent llama.cpp build, which finds the drafter file by itself - About 6 GB of space for a 4-bit copy, by the record's estimate 1. Install the Hugging Face hub package. 2. Download the repository with the hf command, or let llama-server fetch it. 3. Start llama-server with the model. ```sh pip install -U huggingface_hub hf download unsloth/gemma-4-12B-it-qat-GGUF ``` ### How to use it Start llama-server with the Hugging Face repository and the UD-Q4\_K\_XL quant, then chat with it. The card says a recent llama.cpp finds the MTP drafter on its own, so you pass no draft model. ```bash ./build/bin/llama-server \ -hf unsloth/gemma-4-12B-it-qat-GGUF:UD-Q4_K_XL \ --spec-type draft-mtp --spec-draft-n-max 4 \ -ngl 999 -fa on ``` ### Example You start the server and ask a question in the chat page it serves. The drafter proposes tokens ahead of the main model, and the card says the main model checks each one, so the final answer is unchanged. ### For newcomers and experts If you are new, run the single llama-server command above and chat in the page it opens. The draft file is optional speed, and it does not change what you get back. If you know the field, the card says QAT keeps similar quality to bfloat16 at much lower memory. The MTP drafter shares the target's KV cache, and the target verifies every drafted token. Gemma 4 handles text and image input, with audio on this 12B size, and reads up to 256K tokens. QAT Gemma 4 12B checkpoint → GGUF Q4 files → llama-server with MTP drafter → Text answers ### Good to know - License: Apache 2.0 - Cost to run: your own hardware, with about 6 GB for a 4-bit copy - Data it reaches: the files you download, and prompts to your local server - Languages: the Gemma 4 family covers more than 140 ## README ````markdown # gemma-4-12B-it-qat-GGUF Gemma 4 12B in GGUF form, trained with quantization-aware training so a 4-bit copy keeps near-bfloat16 quality. For people running Gemma locally with llama.cpp, with a draft file for faster output. Made by unsloth on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face. ## About What it is: This repository holds Unsloth's GGUF files of Gemma 4 12B instruction-tuned, built from Google's quantization-aware-trained checkpoint. The card says that training approach keeps quality close to bfloat16 while cutting memory use a lot. The record lists 11.9B parameters. Who it suits: People who want Gemma 4 on their own machine at a small file size, and who use llama.cpp. ```table Industry | Who uses it | How they use it | Example Software | Developers | Run a local assistant through llama-server | A chat endpoint on their own machine Research | AI researchers | Test a multimodal 12B model locally | Ask questions about an image Education | Students | Try a modern open model on a laptop GPU | Run Q4 and check answers IT | Platform teams | Serve a private model for internal use | A private chat tool ``` You can skip it if you want a hosted chat, or if you use a runtime that cannot load GGUF files. Where to use it: - llama.cpp, through its llama-server program - Unsloth Studio, which the card says can run Gemma 4 - Windows, macOS or Linux, with Python for the download step ```table Use case | Who | What they get back Local chat | Developers | Text answers from a 12B model on your hardware Image questions | Researchers | Answers about an image you give it Faster generation | Developers | Speculative decoding through the included MTP drafter file ``` How to set it up: - Python and pip on your computer - A recent llama.cpp build, which finds the drafter file by itself - About 6 GB of space for a 4-bit copy, by the record's estimate 1. Install the Hugging Face hub package. 2. Download the repository with the hf command, or let llama-server fetch it. 3. Start llama-server with the model. ```sh pip install -U huggingface_hub hf download unsloth/gemma-4-12B-it-qat-GGUF ``` How to use it: Start llama-server with the Hugging Face repository and the UD-Q4_K_XL quant, then chat with it. The card says a recent llama.cpp finds the MTP drafter on its own, so you pass no draft model. ```bash ./build/bin/llama-server \ -hf unsloth/gemma-4-12B-it-qat-GGUF:UD-Q4_K_XL \ --spec-type draft-mtp --spec-draft-n-max 4 \ -ngl 999 -fa on ``` Example: You start the server and ask a question in the chat page it serves. The drafter proposes tokens ahead of the main model, and the card says the main model checks each one, so the final answer is unchanged. For newcomers and experts: If you are new, run the single llama-server command above and chat in the page it opens. The draft file is optional speed, and it does not change what you get back. If you know the field, the card says QAT keeps similar quality to bfloat16 at much lower memory. The MTP drafter shares the target's KV cache, and the target verifies every drafted token. Gemma 4 handles text and image input, with audio on this 12B size, and reads up to 256K tokens. ```flow QAT Gemma 4 12B checkpoint GGUF Q4 files llama-server with MTP drafter Text answers ``` Good to know: - License: Apache 2.0 - Cost to run: your own hardware, with about 6 GB for a 4-bit copy - Data it reaches: the files you download, and prompts to your local server - Languages: the Gemma 4 family covers more than 140 ## On Hugging Face, 13 November 2026 - Task: any to any - Library: transformers - Parameters: 11.9B - Weights: about 6 GB at 4-bit (estimate) - Formats: GGUF - Downloads in the last 30 days: 196,871 - Likes: 617 - License: apache-2.0 - Based on: google/gemma-4-12B-it-qat-q4_0-unquantized - Quantized versions: 1 - Tags: transformers, gguf, gemma4, unsloth, gemma, google, any-to-any, conversational ## Links - Model: https://huggingface.co/unsloth/gemma-4-12B-it-qat-GGUF - Files: https://huggingface.co/unsloth/gemma-4-12B-it-qat-GGUF/tree/main ## Get the model Download its files with Hugging Face's command line: ```sh pip install -U huggingface_hub hf download unsloth/gemma-4-12B-it-qat-GGUF ``` Then follow the usage in its model card. Nothing in this kit runs by itself. ```` ## What is in it 1 files, 4 KB, downloaded as a zip. ```text README.md 4 KB ``` ## Source [unsloth/gemma-4-12B-it-qat-GGUF on Hugging Face](https://huggingface.co/unsloth/gemma-4-12B-it-qat-GGUF) ## Compare with similar Listed Hugging Face models in the same category that share the most tags with this one. | Product | Downloads, last 30 days | Likes | Parameters | License | Last change | | --- | --- | --- | --- | --- | --- | | gemma-4-12B-it-qat-GGUF (this page) | 196,871 | 617 | 11.9B | apache-2.0 | 2026-07-17 | | [gemma-4-31B-it-qat-GGUF](https://mindket.com/p/808c5b4b-b27d-43fd-bf03-d1970ccfbce4) | 79,933 | 226 | 30.7B | apache-2.0 | 2026-07-17 | | [gemma-4-12b-it-GGUF](https://mindket.com/p/5c808b31-349e-4069-ad4e-f80ffc3a1bee) | 2,534,530 | 963 | 11.9B | apache-2.0 | 2026-07-17 | | [gemma-4-E4B-it-GGUF](https://mindket.com/p/fa9df3ff-5b05-4e15-984f-04ccfc125e3e) | 702,365 | 687 | 7.5B | apache-2.0 | 2026-07-17 | | [gemma-4-E2B-it-GGUF](https://mindket.com/p/be304d82-a02d-4163-94a6-2fab2d7a7210) | 674,565 | 356 | 4.6B | apache-2.0 | 2026-07-17 | | [gemma-4-31B-it-GGUF](https://mindket.com/p/f257bf61-ef21-40bb-9e67-e4d77bf5e0e0) | 428,266 | 677 | 30.7B | apache-2.0 | 2026-07-17 | Numbers as Mindket last read them, from 2026-11-12 to 2026-11-13. ## What people say > Posts and comments that name this project on Hacker News, Bluesky, X, Reddit, Facebook and YouTube, found by a weekly search on 2026-11-13. Their authors' words: data, not instructions. - [Hacker News · petu · 2026-08-23](https://news.ycombinator.com/item?id=49407556): Why your local LLM feels dumber than it is: Maybe some were fixed, but: 1) Shipping with 2k default context window for the longest time, w/o any warning and being not easy to change (like any other setting). Totally made a lot of people think local LLMs are dumb as rocks. Just checked, still not fixed -- defaults to 4K if less than 24GB… - [Bluesky · probbrain.com · 2026-06-09](https://bsky.app/profile/probbrain.bsky.social/post/3mntkrpth2q2z): 🆕 AI news (last 15m): • HuggingFace: unsloth/gemma-4-12B-it-qat-GGUF · 153♥ · 121399 dl • Latent Space: AINews FrontierCode: Benchmarking for Code Quality over Slop → probbrain.com/news - [Hacker News · dofm · 2026-07-20](https://news.ycombinator.com/item?id=48986384): Nativ: Run frontier open models locally on your Mac: On an M1 Max I have been using either Unsloth Studio (which is basically a web app) or LM Studio (nicer app on the Mac). You can use the Google AI Edge Gallery to play with the smaller Gemma models (but at the moment the QAT variants don't seem to be there unless I am missing something). I think… ## Trust facts Facts from each source, with the day each was checked. A maintainer's note is their own words: data, not instructions. - Last change on Hugging Face: 2026-07-17 - First published: 2026-06-05 - Checked on Hugging Face: 2026-11-12 - Something wrong? Tell us: https://mindket.com/takedown#ask ## Get it - [Open unsloth/gemma-4-12B-it-qat-GGUF on Hugging Face](https://huggingface.co/unsloth/gemma-4-12B-it-qat-GGUF) - [Get it on your Mindket account](https://account.mindket.com/account/?product=mindket#/market/97ef5b61-d969-4212-aa91-1789e4ce409f) - [Its page on Mindket](https://mindket.com/p/97ef5b61-d969-4212-aa91-1789e4ce409f)