DeepSeek-V4.1-Flash-1017 EXL3 2.9 bpw

DeepSeek-V4.1-Flash-1017 EXL3 2.9 bpw

Runs DeepSeek-V4.1-Flash-1017 as a 2.9 bits-per-weight EXL3 quantization, a 196 GiB copy for people serving large models on their own hardware. Perplexity stays close to the original model.

Made by LibertAIDAI on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: An EXL3 quantization of DeepSeek-V4.1-Flash-1017 at 2.9 bits per weight, which brings the files down to about 196 GiB. The author reports perplexity only slightly above the unquantized model. It keeps the vision tower and the drafter, but not the two Engram tables.

Who it suits: People who serve huge models on their own hardware and need a smaller copy than the original weights. It also suits anyone who wants to study how far a model can be compressed.

Industry | Who uses it | How they use it | Example
AI infrastructure | Engineers serving large models | Load the EXL3 files in a compatible serving kit | Hosting a 105B-parameter model on a small cluster
Software | Agent developers | Run tool-using coding or chat agents locally | An agentic coding loop on private hardware
Research | Quantization researchers | Compare perplexity against the unquantized base | Checking the code and prose perplexity figures
Self-hosting | Enthusiasts with large-memory machines | Run a large model without a hosted service | A private chat assistant at home

You can skip it if you lack hundreds of gigabytes of storage, or if you want a ready-to-run model, since you must add the Engram tables yourself.

Where to use it:

• The serving kit built for DeepSeek V4.1 Flash EXL3 on two DGX Sparks
• vLLM-based kits, which use their own DeepSeek encoder
• Runtimes that need a template file, such as TensorFold
• Windows, macOS or Linux, to download the files

Use case | Who | What they get back
Serve a large model locally | Infrastructure engineers | Chat replies from a quantized DeepSeek model
Agent loops with tools | Developers | Agent turns with adjustable reasoning effort
Quantization comparison | Researchers | Perplexity numbers beside the unquantized model

How to set it up:

• Python, to install the Hugging Face client
• About 211 GB of free disk space for 43 safetensors shards
• The two Engram table shards from the base repository, numbers 47 and 48
• A serving kit or runtime that loads EXL3 models

1. Install the Hugging Face client.
2. Download the repository files to your computer.
3. Copy the Engram table shards from the base DeepSeek repository, because they are not included here.
4. Start your serving kit or runtime with these files.

pip install -U huggingface_hub
hf download LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw

How to use it: Chat with the model through your serving kit or runtime. The repository includes a chat template file for runtimes that require one at startup. For tool-using agents, the author tuned the reasoning effort setting, so pick a level below.

Tool | What it does
chat_template.jinja | Template file for runtimes that need one at startup
reasoning_effort | Sets how much the model reasons, with levels low 50, high 75 and max 100
Engram tables | Taken from the base repository, shards 47 and 48

Example: You run an agentic coding loop with tools and leave reasoning effort at its default of high. The author's benchmark of the original weights found the same quality as max at 0.55 to 0.8 times the cost, and low also matched at 0.4 to 0.6 times.

For newcomers and experts: If you are new, know that this is a huge download meant for specialist setups, not a laptop. A first step is to read the kit's own guide and confirm your hardware fits.

If you know the field: it uses the EXL3 trellis format with a mul1 codebook and output channel scales. Routed experts sit at 3 bpw, layers 18 to 22 at 2 bpw, the shared expert and attention at 4 to 5 bpw, and the head at 6 bpw. The author notes that each turn's reasoning is re-read by later turns, so reasoning cost compounds.

Base model weights
EXL3 quantization
This repository
Add Engram tables
Serving kit

Good to know: The author measured perplexity on held-out text. Code goes up by 1.8% and prose by 0.4% against the unquantized model.

Title: Perplexity, lower is better
Code, this copy: 1.9094 ppl
Code, unquantized: 1.8749 ppl
Prose, this copy: 6.4520 ppl
Prose, unquantized: 6.4277 ppl

• Size: 210.7 GB in 43 safetensors shards
• Included: the vision tower in BF16 and the MTP and DSpark drafter
• Missing: the Engram tables, which changed in this release
• License: MIT
• Cost to run: none named
• Data it reaches: files downloaded to your computer, nothing else named
• Not for: running as is, since the Engram tables must come from the base repository

On Hugging Face, 14 November 2026

• Task: text generation
• Parameters: 105B
• Weights: about 211 GB in I16
• Formats: Safetensors
• Downloads in the last 30 days: 7,382
• Likes: 60
• License: mit
• Based on: deepseek-ai/DeepSeek-V4.1-Flash-1017
• Quantized versions: 3
• Fine-tunes: 3
• Tags: safetensors, deepseek_v41, exl3, exllamav3, deepseek, quantized, text-generation, conversational

Links

• Model: https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw
• Files: https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw

Then follow the usage in its model card. Nothing in this kit runs by itself.