google_gemma-4-31B-it-GGUF

google_gemma-4-31B-it-GGUF

Quantized GGUF files of Google's Gemma 4 31B instruction model, from 17.15 GB to 61.41 GB, made by bartowski with an importance matrix. Reads images and text. Run in llama.cpp, LM Studio and similar tools.

Made by bartowski on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This repository holds GGUF quantizations of Google's gemma-4-31B-it, made by bartowski. He built them with llama.cpp release b8746 and an importance matrix, which helps keep quality as files shrink. The model has 30.7B parameters and takes images and text. There are many file sizes, from a 61.41 GB full BF16 copy down to files near 17 GB. The card says its latest update brought fixes and a chat template.

Who it suits: It suits people who want to run Gemma 4 31B on their own computer and need a file size that fits their memory.

Industry | Who uses it | How they use it | Example
Local AI | People running models offline | Load a quantized file in a desktop runner | Run the 19.60 GB Q4_K_M file in LM Studio
Software | Developers testing local models | Serve the model from llama.cpp or a similar tool | Try several quant levels on the same prompts
Research | Hobbyists who compare quantization levels | Test two file sizes on the same questions | Compare Q6_K with Q5_K_M on one set of prompts

You can skip it if you want the original Google weights for training, since these are compressed files for running the model.

Where to use it:

• llama.cpp, ramalama, koboldcpp or LM Studio
• Jan AI, Text Generation Web UI or LoLLMs
• Windows, macOS and Linux with Python, to download the files

Use case | Who | What they get back
Local chat | Hobbyists | Replies from a 31B model on their own machine
Image questions | Users of vision capable runners | Answers that take a picture into account
Size choice | People with limited memory | A file that fits the RAM or VRAM they have

How to set it up: You need these first:

• Python, to use Hugging Face's download client
• Disk space and memory that fit the file you pick
• One of the runners listed above

1. Install or update the Hugging Face download client.
2. Choose one file from the card's table, since the card says not to fetch the whole branch.
3. Download that file to your computer.
4. Load it in your runner and use the prompt format below.

pip install -U huggingface_hub
hf download bartowski/google_gemma-4-31B-it-GGUF

How to use it: You load one file in your runner and chat with the model. If your runner does not apply the chat template itself, the card gives the prompt format below. The text available here does not show run commands.

<bos><|turn>system
{system_prompt}<turn|>
<|turn>user
{prompt}<turn|>
<|turn>model
<|channel>thought
<channel|>

Tool | What it does
Q4_K_M | A 19.60 GB file that the card names as the default size for most uses
Q6_K | A 26.73 GB file with higher quality
IQ4_XS | A 17.16 GB file that saves space compared with the Q4 files
Q3_K_XL | A 17.15 GB file with lower quality, for low memory
bf16 | The full weights at 61.41 GB, stored in several parts

Example: You have a machine with about 24 GB of memory, so you take the 19.60 GB Q4_K_M file. You load it in LM Studio and send a prompt. The card suggests this size for most people.

For newcomers and experts: Quantizing shrinks a model so it fits in less memory, and the suffix tells how much was shrunk. A good first try is the Q4_K_M file, then a larger or smaller one if it does not fit.

For readers who know the field, the files marked L keep the embedding and output weights at Q8_0. Q4_0 and IQ4_NL offer online repacking for some ARM and AVX CPUs. Q4_1 is a legacy format that the card says improves tokens per watt on Apple silicon.

Pick a file that fits your memory
Download that single file
Open it in a runner
Apply the prompt format
Chat with the model

Good to know:

• Single file: the card says to download one file, not the whole branch, and the repository holds many large files.
• New models: tools may need an update before a newly supported model runs.
• Cut: the file table in the text available here stops after 15 rows, so the smaller files are missing.
• Counts: Hugging Face lists 30.7B parameters and about 15.3 GB at 4-bit as an estimate.
• Versions: Hugging Face lists no other quantized versions and 1 fine tuned version.

Title: File size by quantization, from the card's table
bf16: 61.41 GB
Q8_0: 32.64 GB
Q6_K: 26.73 GB
Q5_K_M: 22.61 GB
Q4_K_M: 19.60 GB
IQ4_XS: 17.16 GB
Q3_K_XL: 17.15 GB

• License: Apache 2.0.
• Cost to run: none named. You need memory that fits the file.
• Data it reaches: the prompts and images you give it, on your own machine. The text available here names no network use.
• Not for: none named in the text available here.

On Hugging Face, 13 November 2026

• Task: image text to text
• Parameters: 30.7B
• Weights: about 15.3 GB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 44,117
• Likes: 98
• License: apache-2.0
• Based on: google/gemma-4-31B-it
• Fine-tunes: 1
• Tags: gguf, image-text-to-text, imatrix, conversational

Links

• Model: https://huggingface.co/bartowski/google_gemma-4-31B-it-GGUF
• Files: https://huggingface.co/bartowski/google_gemma-4-31B-it-GGUF/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download bartowski/google_gemma-4-31B-it-GGUF

Then follow the usage in its model card. Nothing in this kit runs by itself.