GLM-5.3-Flash EXL3 3.05bpw

GLM-5.3-Flash EXL3 3.05bpw

Runs GLM-5.3-Flash locally on two RTX 3090 cards with 128 GB of RAM and a 262,144-token context, using a verified 3.05 bpw EXL3 quantization. For people hosting a large model at home.

Made by albucino on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: A compressed copy of the GLM-5.3-Flash text generation model, set up to run on a home machine with two 24 GB graphics cards while keeping the full 262,144-token context. The repository holds the model card, a snapshot of the runtime and a lock file of checksums. The weights are not stored here. They come unchanged from turboderp's 3.05 bits-per-weight EXL3 release, and the runtime pulls them down and checks them.

Who it suits: People who want a large long-context model on their own hardware instead of a hosted service, and who already own two RTX 3090s with 128 GB of RAM.

Industry | Who uses it | How they use it | Example
Software | Developers building agents | Point an agent client at the local OpenAI-compatible server | A coding agent that reads a whole project in one prompt
Research | Long-document analysts | Send huge inputs and look for facts buried deep inside them | Retrieve a fact hidden near the 260K-token mark
Privacy-minded teams | Small teams with a GPU workstation | Keep prompts on a machine they own | Draft internal text without a hosted service
Hardware testing | Builders of local AI rigs | Measure what two consumer GPUs can do | Compare first-token time at 131K and 260K tokens

You can skip it if you lack two RTX 3090s and 128 GB of RAM, since the setup was validated on one such machine only.

Where to use it: It runs on a Linux machine with Docker and the NVIDIA Container Toolkit. Any client that speaks the OpenAI API format can talk to the local server it starts.

• Linux with Docker and the NVIDIA Container Toolkit
• Two RTX 3090 GPUs and 128 GB of RAM
• Agent tools and apps that can call an OpenAI-compatible API on your own machine
• Windows, macOS or Linux with Python, only to fetch this repository's files

Use case | Who | What they get back
Long-document questions | Researchers | Answers over prompts up to 262,144 tokens
Agent work | Developers | Tool-calling turns, with reasoning limited to 4,096 tokens and prefix caching on
Local serving | Hobbyists | An API on their own machine that decodes at 49.4 tokens per second in the card's test

How to set it up: Prepare the hardware first, then fetch the files, then build and start the runtime. The weights are large, so leave room on disk.

• Linux, Docker and the NVIDIA Container Toolkit
• Two RTX 3090s and 128 GB of RAM
• Python, for the Hugging Face download commands
• About 117 GiB of disk space for the verified weights

1. Install the Hugging Face tools and fetch this repository's files.
2. Get the runtime project named glm53-flash-2x3090 by DominikBucko from GitHub.
3. Run its download script to fetch and verify the weights.
4. Copy the example environment file and set MODEL_DIR in it.
5. Build the image, run the preflight check, then start the server.

pip install -U huggingface_hub
hf download albucino/GLM-5.3-Flash-EXL3-3.05bpw

scripts/download_model.sh /models/glm53-flash-exl3-3.05bpw
cp .env.example .env
make build-image && make preflight && make serve

How to use it: Start the server with make serve. It exposes an OpenAI-compatible API on the local machine, so you point your client or agent tool at it and send prompts as usual. The card describes two profiles, one tuned for long prefill and one for agent work with prefix caching.

Tool | What it does
scripts/download_model.sh | Downloads the weights and verifies them
scripts/install_model.py | Adds the 22 missing indexer entries and checks the files against the lock
make build-image | Builds the Docker image
make preflight | Runs the preflight check before you serve
make serve | Starts the OpenAI-compatible API

Example: Send the server a prompt of 131,072 tokens, such as a large document, in the prefill profile. The card's own run saw the first token after 113.5 seconds, then text at 49.4 tokens per second.

For newcomers and experts: GLM-5.3-Flash is a large language model, and quantizing it stores its numbers in fewer bits so it fits on home hardware. This page is the recipe, not the weights. A first small try is to run the two download commands, which fetch this repository's files before you commit to the large weights download.

For experts, the routed experts sit in host memory while each GPU keeps an LRU cache of them in VRAM, with the hottest 20% of each layer pinned. For cold experts the GPU pulls 30% over PCIe and a CPU thread pool computes the rest, inside one CUDA graph per decode step. The upstream 3.05 bpw branch leaves 22 sparse-attention indexer tensors out of its index file, so a loader that trusts the index loses long-context retrieval. The install script adds them and checks all 29 files against the lock. The trade-offs are single-request serving, no speculative decoding and some quality lost against the BF16 original, with no size given.

Fetch the 3.05 bpw EXL3 weights
Add 22 indexer entries to the index
Check all 29 files against the lock
Build the image and run preflight
Serve an OpenAI-compatible API

Good to know: The timings below come from the card's benchmark on one machine, one request at a time.

Title: Seconds to the first token
131,072 prefill profile: 113.5 s
260,096 prefill profile: 235.3 s
131,072 agent profile: 236.2 s
8,192 agent profile: 9.5 s

• Hardware: validated on one machine only, with two RTX 3090s, a Threadripper PRO 5975WX and 128 GB of DDR4-3200 memory.
• Speed: decode depends heavily on CPU memory bandwidth, and prefill depends on PCIe bandwidth.
• Real agent turns: decoded at 37.7 tokens per second on average.
• Cache: across 15 agent tasks, prefix caching served 72.9% of the prompt tokens.
• License: MIT for GLM-5.3-Flash and for turboderp's quantization, Apache-2.0 for the runtime. Review the upstream terms before use.
• Cost to run: none named, no API key, account or paid service, apart from the hardware.
• Data it reaches: it downloads the weights from Hugging Face and serves an API on your own machine.
• Not for: several requests at once, speculative decoding, or machines without two 24 GB GPUs.

On Hugging Face, 13 November 2026

• Task: text generation
• Downloads in the last 30 days: 0
• Likes: 2
• License: mit
• Based on: zai-org/GLM-5.3-Flash
• Tags: glm-5.3, exl3, exllamav3, vllm, 256k-context, rtx-3090, dual-gpu, 24gb-vram, 128gb-ram, consumer-gpu, cpu-offload, local-llm

Links

• Model: https://huggingface.co/albucino/GLM-5.3-Flash-EXL3-3.05bpw
• Files: https://huggingface.co/albucino/GLM-5.3-Flash-EXL3-3.05bpw/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download albucino/GLM-5.3-Flash-EXL3-3.05bpw

Then follow the usage in its model card. Nothing in this kit runs by itself.