DeepSeek-R1-GGUF

DeepSeek-R1-GGUF

GGUF copies of the 671B DeepSeek-R1 reasoning model, with 1.58-bit and 2-bit dynamic quants, for running it in llama.cpp on a local machine with enough memory.

Made by unsloth on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: DeepSeek-R1-GGUF is a set of GGUF files of the 671B-parameter DeepSeek-R1 model, made by unsloth. They shrink the model so llama.cpp can load it on your own hardware.

The card says its 1.58-bit and 2-bit dynamic quants quantize layers selectively, which keeps accuracy higher than plain 1-bit or 2-bit versions.

Who it suits: Developers and hobbyists who want to run a large reasoning model locally, on a machine with a lot of memory and ideally an Nvidia GPU. You also need to be comfortable building llama.cpp from source.

Industry | Who uses it | How they use it | Example
Software | Developers | Run a reasoning model offline for coding help | Ask it to write a small Python game
Research | Machine learning researchers | Compare extreme low-bit quants of one large model | Try the 1.58-bit files on your own prompts
Hardware enthusiasts | Home lab owners | Test how far a big model fits on one box | Offload some layers to a 24GB GPU

People without a large-memory machine can skip it, since even the 4-bit weights are estimated at about 336 GB.

Where to use it:

• llama.cpp, built from source with CUDA on, as the card's steps show
• A Linux machine, since the build steps use apt-get
• Windows, macOS or Linux with Python, for downloading the files
• A computer with a 24GB GPU such as an RTX 4090, if you want to offload layers

Use case | Who | What they get back
Run a reasoning model locally | Developers | Answers that start with a think section
Write code from a prompt | Programmers | Code for a request like the card's Flappy Bird game in Python
Test a 1.58-bit quant | Researchers | A view of how a tiny quant behaves on your prompts

How to set it up: Plan for a big download and a lot of memory before you start.

• Python and pip, for the Hugging Face download tools
• llama.cpp built from source, with the compiler toolchain, cmake and curl development packages installed
• Memory for the weights, about 336 GB at 4-bit as an estimate
• Optional: a GPU with 24GB to offload some layers

1. Get the latest llama.cpp and build it with cmake. The card installs the build packages with apt-get first and turns CUDA on.
2. Build the llama-quantize, llama-cli and llama-gguf-split targets.
3. Download only the quant type you want. The card's snippet picks UD-IQ1_S, the 1.58-bit one.
4. Run llama-cli on the first GGUF file.

The plain command from Hugging Face fetches every file in the repository, which is why the pattern-based download below is the lighter route.

pip install -U huggingface_hub
hf download unsloth/DeepSeek-R1-GGUF

# pip install huggingface_hub hf_transfer
# import os # Optional for faster downloading
# os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"

from huggingface_hub import snapshot_download
snapshot_download(
  repo_id = "unsloth/DeepSeek-R1-GGUF",
  local_dir = "DeepSeek-R1-GGUF",
  allow_patterns = ["*UD-IQ1_S*"], # Select quant type UD-IQ1_S for 1.58bit
)

How to use it: You run the model with llama-cli. Put your prompt between the User and Assistant tokens, or let a chat template tool add them for you. The card suggests a min-p of 0.05 to curb rare token picks, and its author found that works well on the 1.58-bit model. The flag -no-cnv turns off the automatic conversation mode.

./llama.cpp/llama-cli \
  --model DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S-00001-of-00003.gguf \
  --cache-type-k q4_0 \
  --threads 12 -no-cnv --prio 2 \
  --temp 0.6 \
  --ctx-size 8192 \
  --seed 3407 \
  --prompt "<｜User｜>Create a Flappy Bird game in Python.<｜Assistant｜>"

With a GPU, add the layer offload flag to move some layers onto it. With several GPUs you can probably offload more. To join the split files into one, run the merge tool.

./llama.cpp/llama-cli \
  --model DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S-00001-of-00003.gguf \
  --cache-type-k q4_0 \
  --threads 12 -no-cnv --prio 2 \
  --n-gpu-layers 7 \
  --temp 0.6 \
  --ctx-size 8192 \
  --seed 3407 \
  --prompt "<｜User｜>Create a Flappy Bird game in Python.<｜Assistant｜>"

./llama.cpp/llama-gguf-split --merge \
  DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ1_S-00001-of-00003.gguf \
  merged_file.gguf

Tool | What it does
llama-cli | Loads a GGUF file and answers your prompt in the terminal
llama-gguf-split | Merges the split GGUF files into a single file
snapshot_download | Fetches only the files that match a pattern, such as one quant type

Example: You ask for "Create a Flappy Bird game in Python." inside the User and Assistant tokens. The card's sample output shows the model opening with a think section, where it talks itself through the question before it answers.

For newcomers and experts: This is a huge model squeezed into much smaller files, and it still needs a lot of memory. A good first step is to read the build steps and download just one quant type before anything else.

Underneath, it is DeepSeek-R1 at 671B parameters in GGUF. The dynamic quants keep accuracy up by treating layers unevenly, and the trade-off is that the lowest bit counts still cost quality against the full model. The card gives no benchmark numbers in the part shown here.

Build llama.cpp with CUDA
Download one quant type
Run llama-cli on the first file
Read the think section, then the answer

Good to know: The files are large and the setup is manual, so check your memory and disk before you download.

• Size: 671B parameters in GGUF format only, listed with English as its language
• Variants: Hugging Face shows 7 fine-tuned versions made by other people
• License: MIT
• Cost to run: no API key or paid service is named, but the hardware is the real cost, with about 336 GB for the 4-bit weights
• Data it reaches: the files you download from Hugging Face and the prompts you type into llama-cli
• Not for: the card names no restrictions beyond its license

On Hugging Face, 15 November 2026

• Task: text generation
• Library: transformers
• Parameters: 671B
• Weights: about 336 GB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 46,994
• Likes: 1,128
• License: mit
• Languages: en
• Based on: deepseek-ai/DeepSeek-R1
• Fine-tunes: 7
• Tags: transformers, gguf, deepseek_v3, text-generation, deepseek, unsloth, conversational

Links

• Model: https://huggingface.co/unsloth/DeepSeek-R1-GGUF
• Files: https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download unsloth/DeepSeek-R1-GGUF

Then follow the usage in its model card. Nothing in this kit runs by itself.