gemma-4-31b-it-exl3-optimized

gemma-4-31b-it-exl3-optimized

Holds the measurements and recipes for building an optimized EXL3 quant of Gemma 4 31B-it at any bitrate. For people who quantize models with exllamav3 and want to skip the slow measuring steps.

Made by Dino0040 on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This repository is not a ready model. It holds the pipeline outputs from exllamav3's optimized quantization recipes for google/gemma-4-31B-it. With these files you can build an optimized EXL3 quant at an arbitrary bitrate. The slow part, a sensitivity measurement of about an hour that needed roughly 120 GB of VRAM, is already done. A 3.00bpw quant is available on a separate branch for anyone who only wants to test it.

Who it suits: People who quantize large models with exllamav3 and want better bit allocation without redoing the measurements.

Industry | Who uses it | How they use it | Example
Model hosting | Engineers who serve large models | Build a quant at the bitrate that fits their GPU | Making a 3 bit quant that fits a single card
Open model community | Quantization hobbyists | Reuse the measurements to try other bitrates | Generating a recipe for a different bits per weight target
Research | People who study quantization error | Compare uniform and optimized bit allocation | Reproducing the card's KL divergence comparison

Skip it if you want a model to chat with right away, or if you have no GPU with more than 20 GB of VRAM for the conversion step.

Where to use it: It works with the exllamav3 project and its scripts. The files download on Windows, macOS or Linux.

• exllamav3: its optimize and convert scripts use these files
• A GPU with more than 20 GB of VRAM, for the conversion
• The original google/gemma-4-31B-it weights, as the input to convert

Use case | Who | What they get back
Custom bitrate quant | Quantizers | A quant made from the recipe at your target bitrate
Quick test | Anyone curious | The ready made 3.00bpw quant on its branch
Method comparison | Researchers | Numbers comparing uniform and optimized quants

How to set it up: You need the exllamav3 scripts and the original Gemma weights, plus the files from this repository.

• Python with pip
• exllamav3 and its scripts
• The original google/gemma-4-31B-it weights
• A GPU with more than 20 GB of VRAM

1. Install the Hugging Face hub library.
2. Download the files from this repository.
3. Run the two steps below with your own paths and bitrate.

pip install -U huggingface_hub
hf download Dino0040/gemma-4-31b-it-exl3-optimized

How to use it: First make a recipe for your target bitrate, which is nearly instant. Then convert the model with that recipe and the trace as calibration data, which takes an hour or two depending on the GPU.

# 1. Recipe for your target bitrate (basically instant)
python sc_optimize.py -m noise_attrib.json -rr rfn_2.00bpw.json -b <bpw> -hb 6 -al 2.0 -mink 2 -ks 0.5 -maxk 4 -o recipe.yaml

# 2. Convert with the recipe and the trace as calibration data (>20GB VRAM, An hour or two depending on GPU)
python convert.py -i <gemma-4-31B-it> -o <out> -w <work> -rcp recipe.yaml -cd cal_trace.safetensors

Tool | What it does
sc_optimize.py | Builds a recipe for a target bitrate from the measurements
convert.py | Converts the model using the recipe and the calibration trace
-ks 0.5 | Allows half bit rates, and needs -maxk 4 with it
-mink 2 | Keeps every tensor at 2 bits per weight or more

Example: You pick a target of 3.5 bits per weight, replace the placeholder in the first command and run both steps. You end up with an EXL3 quant of Gemma 4 31B-it made from the optimized recipe.

For newcomers and experts: If you are new to this, a quant is a smaller copy of a model that trades some accuracy for less memory. If you only want to try one, use the 3.00bpw branch and skip the scripts.

If you know the field, the files include a self sampled calibration trace, per tensor sensitivity measurements and error anchors from a 2 bpw probe. The recipe decides how many bits each tensor gets, and the trace is used as calibration data during conversion.

Sensitivity measured on the full model
Recipe sets bits for each tensor
Convert uses recipe and trace
Optimized EXL3 quant results

Good to know: The author describes this as a one evening experiment out of curiosity, run on roughly five dollars of cloud time, and claims no expertise. The figures below are theirs, from a 3.00 bpw test against the bf16 model.

Title: Card test at 3.00 bpw, lower KL is better
Quant | Mean KL | Perplexity
Uniform quant from July | 0.0561 | 1.2497
Optimized recipe | 0.0381 | 1.2276
Recipe plus trace calibration | 0.0301 | 1.2187

The card puts the total drop in KL divergence at 46 percent at the same size as the old quant.

• License: Apache 2.0
• Cost to run: none named for the files. Converting needs a GPU with more than 20 GB of VRAM
• Data it reaches: the model weights and the calibration trace you point the scripts at
• Not for: use as a ready to run model, since the repository holds recipe and measurement files

On Hugging Face, 13 November 2026

• Downloads in the last 30 days: 41
• Likes: 2
• License: apache-2.0
• Based on: google/gemma-4-31B-it
• Tags: exl3

Links

• Model: https://huggingface.co/Dino0040/gemma-4-31b-it-exl3-optimized
• Files: https://huggingface.co/Dino0040/gemma-4-31b-it-exl3-optimized/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download Dino0040/gemma-4-31b-it-exl3-optimized

Then follow the usage in its model card. Nothing in this kit runs by itself.