DeepSeek-V4.1-Flash-EXL3-3bpw-2x-RTX-PRO-6000

DeepSeek-V4.1-Flash-EXL3-3bpw-2x-RTX-PRO-6000

Serves a 3-bit compressed copy of DeepSeek-V4.1-Flash-1017 on two 96 GB RTX PRO 6000 Blackwell cards with vLLM, for developers who host a large coding and agent model on their own hardware.

Made by diffbot on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This is a 3-bit compressed copy of DeepSeek-V4.1-Flash-1017, a text generation model with 219B parameters. The author calibrated the EXL3 3-bit routed experts themselves. vLLM runs it across a pair of RTX PRO 6000 Blackwell cards, and about a third of the experts wait in pinned host RAM.

Who it suits: It suits engineers and teams who host a large coding or agent model on their own GPU machines and want an OpenAI-compatible API in front of it.

Industry | Who uses it | How they use it | Example
Software development | Engineers with two 96 GB GPUs | Serve the model for coding agents | Point an agent at port 8000 with the API key you set
AI infrastructure | Teams with an 8-GPU node | Run one copy on each half of the node, keeping all experts in VRAM | Send requests to port 8000 or 8001
Model research | People who study quantization | Compare the pack against the original model | Read its mean KL and benchmark tables

Anyone without the GPUs, memory and disk listed below can skip it.

Where to use it:

• Machines with Docker, the NVIDIA container toolkit and two Blackwell sm_120 cards
• vLLM on a pair of RTX PRO 6000 cards, split with tensor parallelism of 2
• An 8x RTX PRO 6000 node, run as two replicas of four GPUs each
• Clients of an OpenAI-compatible API served on port 8000
• Windows, macOS or Linux with Python, only to fetch the files

Use case | Who | What they get back
Coding agent backend | Developers | Fast replies in multi-turn work with tool calls
Long context work | Developers | A 1M-token window with an fp8 KV cache
Quality checking | Researchers | Published KL, GSM8K-200 and HumanEval numbers for the pack

How to set it up: What you need first:

• Two Blackwell sm_120 GPUs with 96 GB each
• A CUDA 13 driver on the host
• Docker with the NVIDIA container toolkit
• Around 260 GiB of spare host RAM
• 340 GB of disk

1. Install the Hugging Face tools with pip.
2. Download the repository into a local folder.
3. Open the recipe folder and build the serving image.
4. Build the kernels, which takes about 4 minutes.
5. Start the server with an API key you choose.

pip install -U huggingface_hub

hf download diffbot/DeepSeek-V4.1-Flash-EXL3-3bpw-2x-RTX-PRO-6000 --local-dir ~/models/DeepSeek-V4.1-Flash-1017-EXL3-3bpw
cd ~/models/DeepSeek-V4.1-Flash-1017-EXL3-3bpw/recipe
bash docker/build.sh                      # serving image dsv41-flash-1017-exl3-3bpw-sm120
bash kernels/build.sh                     # the xmoe, xmoe_do, xmoe_do_i8, fp8ar and rowtopk kernels, about 4 minutes

Put your own value in API_KEY first.

An x86_64 CPU with AVX-512 support (VNNI, VBMI, VBMI2), plus many cores and memory channels, lets you add the optional CPU runtime. Build it and run the host check, then switch it on.

bash cpuexp/build.sh && bash cpuexp/check_host.sh

Put your own value in API_KEY first.

On an 8x RTX PRO 6000 node, download the whole repo the same way, including the drafter-code-k7 folder, then run these.

bash docker/build.sh                      # the same image, now with the chunked-indexer patch
bash kernels/build.sh 8x                  # xmoe_p8k, xmoe_do_i8, fp8ar, fp8arg and rowtopk

Put your own value in API_KEY first.

How to use it: After serve.sh starts, any client can reach its OpenAI-compatible API on port 8000 and ask for the model named deepseek-v4.1-flash. The scripts below check the running server.

python3 bench/needle-test.py 200 && python3 bench/test-toolcall.py && python3 bench/test-image.py

Put your own value in API_KEY first.

For a single stream of mostly code on two cards, you can switch on the code-tuned DSpark head with k set to 7.

Put your own value in API_KEY first.

Tool | What it does
serve/serve.sh | Starts the server on two cards, with every setting documented inside
serve/serve-8x.sh | Starts two replicas on an 8-GPU node
bench/needle-test.py | Runs the needle test against the server
bench/test-toolcall.py | Runs the tool call test
bench/test-image.py | Runs the image test

Example: You start the server with the default command, then send a chat request to port 8000 with your API key and the model name deepseek-v4.1-flash. The quantized model's reply comes back through the OpenAI-compatible API.

For newcomers and experts: This is a huge model, with about 437 GB of weights, so it needs a GPU server and not a laptop. A first small step is to download the files and read the recipe README, which explains the stack. Then run the build and start commands one by one.

Experts will want the details. The author quantized the routed experts to 3 bits with EXL3, using their own calibration and a method credited to coolbho3k. vLLM splits the model over two cards with tensor parallelism of 2. About a third of the experts stay in pinned host RAM, and custom sm_120 kernels or a CPU runtime that needs AVX-512, which is optional, serve them. The 8x profile uses TP4 with EP4 per replica. The trade is fidelity: mean KL to the original is 0.0522 on neutral text, against a noise floor of 0.0082, while GSM8K-200 reaches 98.5% and HumanEval 97.0%.

Download the repo with hf download
Build the Docker image and the kernels
Start serve.sh with an API key
vLLM splits the model across two GPUs
Some experts wait in pinned host RAM
Clients call the API on port 8000

Good to know:

Title: Output speed, agentic replay, one stream
2x default: 224.1 tok/s
2x CPU runtime: 261.8 tok/s
Previous release: 176 tok/s

• Speed: on the agentic replay benchmark, the default setup is 28% quicker at decoding than the last V4.1-Flash release.
• Context: the default setup offers a 1M window with room for 1.07M tokens in an fp8 KV cache.
• Hardware: the CPU runtime needs an x86_64 CPU with AVX-512, many cores and many memory channels.
• License: mit
• Cost to run: no paid service named. You supply the GPUs and pick your own API key value.
• Data it reaches: it downloads files from Hugging Face, and the server answers requests on port 8000 of your machine.
• Not for: setups below the listed GPU, RAM and disk needs.

On Hugging Face, 13 November 2026

• Task: text generation
• Parameters: 219B
• Weights: about 437 GB in I16
• Formats: Safetensors
• Downloads in the last 30 days: 2,514
• Likes: 42
• License: mit
• Based on: deepseek-ai/DeepSeek-V4.1-Flash-1017
• Quantized versions: 1
• Tags: safetensors, deepseek_v41, exl3, exllamav3, vllm, deepseek, deepseek-v4.1-flash, deepseek-v4.1-flash-1017, mixture-of-experts, 3-bit, rtx-pro-6000, blackwell

Links

• Model: https://huggingface.co/diffbot/DeepSeek-V4.1-Flash-EXL3-3bpw-2x-RTX-PRO-6000
• Files: https://huggingface.co/diffbot/DeepSeek-V4.1-Flash-EXL3-3bpw-2x-RTX-PRO-6000/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download diffbot/DeepSeek-V4.1-Flash-EXL3-3bpw-2x-RTX-PRO-6000

Then follow the usage in its model card. Nothing in this kit runs by itself.