bitnet.cpp

What it is

bitnet.cpp is Microsoft’s official inference framework for 1-bit LLMs such as BitNet b1.58. The README says it offers optimized kernels for fast and lossless inference of 1.58-bit models on CPU and GPU, with NPU support coming next. It is based on llama.cpp, and its kernels build on the lookup table methods of T-MAC. You build it from source, download a model from Hugging Face and run it with Python scripts. The README reports speedups of 1.37x to 5.07x on ARM CPUs and 2.37x to 6.17x on x86 CPUs, against full-precision models.

1. Clone the repo with its llama.cpp code
2. Install the Python packages in a conda environment
3. Download a 1-bit model from Hugging Face
4. setup_env.py builds the project, with i2_s or tl1
5. run_inference.py starts a chat or one prompt

Who it suits

Developers and researchers who want to run a small 1-bit model on a laptop or server CPU with no GPU. People who study low-bit inference also get the kernels and a benchmark script. It is a C++ project with Python scripts, so expect a compiler and a build step. Skip it if you want a finished chat app. The README says it uses existing 1-bit models to demonstrate the framework, and the largest model in its table is 10B.

Where to use it

bitnet.cpp is a framework you build and run yourself, so this is about systems rather than coding agents. The README documents no coding-agent integration.

• Windows: Visual Studio 2022 with the clang tools, and a Developer Command Prompt or PowerShell for every command
• Debian and Ubuntu: clang 18 or newer from the automatic installation script at apt.llvm.org
• x86 CPUs with AVX2, and ARM CPUs with NEON or the DOTPROD extension
• NVIDIA GPUs: a separate W2A8 CUDA kernel in the gpu folder, built by a bash script into libbitnet.so, with speeds measured on an A100 40GB
• Other Linux systems and macOS: the README’s build steps carry no OS label, but its clang install line is for Debian and Ubuntu only. For macOS it shows only a demo of a BitNet b1.58 3B model on an Apple M2

How to set it up

Get it saves a zip of Minda’s setup notes in your downloads folder, with steps for Windows, Linux, the GPU kernel, the embedding models and tuning. They are taken from Microsoft’s README and docs, with Minda’s own notes marked as Minda’s. The main route is a clone, a conda environment, a model download and one build script.

1. Open a shell that can build. On Windows, a Developer Command Prompt or PowerShell for VS2022. On Linux, a shell where clang 18 or newer works.
2. Clone the repo with --recursive. Then go into its folder.
3. Make a Python 3.10 conda environment. Then install requirements.txt.
4. Download the model with huggingface-cli. Then run setup_env.py to build the project.
5. Run run_inference.py with the model file.

git clone --recursive https://github.com/microsoft/BitNet.git
cd BitNet
conda create -n bitnet-cpp python=3.10
conda activate bitnet-cpp
pip install -r requirements.txt

huggingface-cli download microsoft/BitNet-b1.58-2B-4T-gguf --local-dir models/BitNet-b1.58-2B-4T
python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s

The README does not list huggingface-cli as a requirement or say how to install it.

How to use it

Run the model with run_inference.py. With -cnv it starts a chat, and the text after -p becomes the system prompt.

python run_inference.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -p "You are a helpful assistant" -cnv

The other flags the README lists are -n for the number of tokens to predict, -t for threads, -c for the prompt context size and -temp for the temperature. To measure speed, the README ships a benchmark script.

python utils/e2e_benchmark.py -m /path/to/model -n 200 -p 256 -t 4

Example

You want a vector for a search query. The embedding guide has you check out a llama.cpp branch, run setup_env.py with the 0.6B model, and then call llama-embedding:

cd 3rdparty/llama.cpp && git checkout release-bitnet-embedding-0.6b-270m && cd ../..
python setup_env.py -hr microsoft/bitnet-embedding-0.6b -md /path/to/save/model
./build/bin/llama-embedding \
  -m /path/to/save/model/bitnet-embedding-0.6b/ggml-model-i2_s.gguf \
  -p "query: What is BitNet?" \
  --embd-normalize 2 \
  --embd-output-format array

The guide says the result is a 1024-dimensional, L2-normalized embedding, printed as an array that starts [[0.0239517, 0.6826404. The guide’s FAQ says to add an instruction to the query, a one-sentence instruction that describes the task, because that is how the model is trained. Without it you will see a performance degradation. The document side needs no instruction. The guide’s own example passes “query: What is BitNet?”.

For newcomers and experts

If you are new. A language model keeps what it learned in numbers called weights. A 1-bit model such as BitNet b1.58 limits each weight to -1, 0 or +1, a ternary value, which is where the 1.58 bits come from. Small weights make the model smaller and the arithmetic cheaper, so a CPU can run it. bitnet.cpp is the program that runs such models. The embedding guide says its I2_S conversion needs natively trained BitNet models with ternary weights. Its F16 conversion does not. Start with BitNet-b1.58-2B-4T, the model the README builds first.

For experts. The README’s table marks which of three kernels, I2_S, TL1 and TL2, work for each model on x86 and on ARM. I2_S stores ternary weights in 2 bits. The embedding guide packs every 128 values into 32 bytes, four values to a byte, and appends a float32 scale to the buffer. The README’s 15 January 2026 CPU update adds weight-parallel and activation-parallel kernels, native I2_S GEMM and GEMV in ggml, tiling set in include/gemm-config.h, and Q6_K embedding quantization. The CPU page recommends activation parallel for I2_S, and the current kernel defaults to it.

The usage text offers only i2_s and tl1 as --quant-type values, so the README does not say how to choose TL2. It also says bitnet.cpp can run a 100B BitNet b1.58 model on one CPU at 5 to 7 tokens per second, yet no model that size is in its table. Check what that claim rests on before you plan around it.

Good to know

Most of it runs on your machine, and a few paths reach out.

• What it fetches: git clone --recursive pulls llama.cpp code from a fork that .gitmodules names, pip installs packages, and the model comes from Hugging Face. On Debian or Ubuntu the clang line downloads a script from apt.llvm.org and runs it at once
• Models: the framework is separate from its models, and each model has its own Hugging Face page. The embedding guide gives MIT for its models, so check the page of any other model you download
• The maker’s own caution: under Use in High-Risk Applications, the embedding guide says its models are not recommended for commercial or real-world applications without further testing and development
• Usage data: the README and the docs say nothing about telemetry
• Windows traps: every command needs the Developer shell, and the README prints the two PowerShell commands of its clang fix on one line. Minda’s Windows notes split them onto two lines
• The GPU kernel: its page is tailored to BitNet-b1.58-2B-4T, and its compile script builds for compute_80
• A mismatch in the docs: the README’s usage text calls --quant-embd a conversion to f16, and the CPU page says it converts embeddings to Q6_K
• Removal: the README gives no uninstall steps. Minda’s notes say to delete the clone and its model folder and to remove the conda environment
• License: MIT. The LICENSE file reads “Copyright (c) Microsoft Corporation.”