Bonsai-27B-gguf

Bonsai-27B-gguf

Runs a 27B-class reasoning model as a 1-bit GGUF file of about 3.9 GB for llama.cpp, so it fits a laptop or one GPU. For people who want local reasoning on everyday hardware.

Made by prism-ml on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: Bonsai 27B is a 1-bit build of a 27B-class reasoning model, packed as a GGUF file for llama.cpp. It derives from Qwen3.6-27B and stores its language weights as single bits with a shared scale for each group. The deployed size is about 3.9 GB, against roughly 54 GB at FP16, so it can run on a laptop or one GPU. The authors say it keeps about 90 percent of FP16 quality.

Who it suits: It suits people who want a large reasoning model on everyday hardware and are comfortable running specialized llama.cpp builds.

Industry | Who uses it | How they use it | Example
Local AI hobby | Enthusiasts | Run a 27B-class model on a laptop | Chatting with the model offline on a Mac using Metal
Software development | Developers | Test a small-footprint reasoning model inside a local app | Serving it with llama.cpp on a single GPU
Edge and mobile | App builders | Look at the MLX companion for Apple Silicon and iPhone | Trying the MLX version on a phone
Model research | Researchers | Study 1-bit weights and speculative decoding | Comparing it with the ternary companion

You can skip it if your runtime cannot load the author's 1-bit kernels, since stock builds may not read this format.

Where to use it:

• llama.cpp on CUDA, Metal or CPU, using the author's fork that holds the 1-bit kernels
• Windows, macOS or Linux with Python, to download the files
• Laptops and single GPUs, which the card names as its target
• Apple Silicon and iPhone, through the separate MLX companion

Use case | Who | What they get back
Local reasoning chat | Anyone with a laptop | Reasoning and coding answers from a file that fits in memory
Image questions | Vision users | Answers about pictures, using the optional vision pack
Faster decoding | GPU users | A lossless 1.37x speedup on CUDA from the bundled drafter layer

How to set it up: You need Python for the download tool and a llama.cpp build that includes the 1-bit kernels. The weights take about 3.9 GB, plus 0.63 GB if you want image input.

• Python on Windows, macOS or Linux
• The huggingface_hub package
• The author's llama.cpp fork, linked from the card, built for CUDA, Metal or CPU
• Room for the weights, plus about 4.3 GB of cache at the full 262K window

1. Install huggingface_hub.
2. Download the model files.
3. Build the author's llama.cpp fork and point it at the GGUF file.

pip install -U huggingface_hub
hf download prism-ml/Bonsai-27B-gguf

How to use it: The visible card gives the download step but no run command. It points to the author's demo repository for serving and benchmarking, and to a whitepaper for the method. Once your build loads the file, you chat with it like any local model. A drafter layer for speculative decoding ships with it.

Variant | Runs on | Size
GGUF 1-bit, this repository | llama.cpp on CUDA, Metal or CPU | About 3.9 GB
MLX 1-bit | Apple Silicon, including iPhone | Not stated
Ternary GGUF | llama.cpp | About 7.2 GB

Example: You load the GGUF in a build with the 1-bit kernels and ask a math or coding question. The model reasons through it and writes the answer. The authors measured roughly 44 tokens per second on an Apple M5 Pro laptop.

For newcomers and experts: If you are new to this, think of a huge model squeezed into a small file by storing each weight as a single bit. A first small try is the download step, then one short question once a compatible build is ready.

For people who know the field, each weight is a sign bit and every group of 128 shares one FP16 scale, which comes to 1.125 bits per weight. Embeddings, attention, MLP and the LM head are all binary, and the vision tower uses 4-bit HQQ. A hybrid backbone with about 75 percent linear attention keeps the cache small.

Title: Size in GB
FP16 original: 54 GB
Ternary Bonsai: 7.2 GB
1-bit Bonsai: 3.9 GB

You download the GGUF file
A build with 1-bit kernels loads the packed weights
The drafter layer proposes tokens ahead
The model checks them and writes the answer

Good to know: The quality figures come from the authors. Across 15 thinking-mode benchmarks they report an average of 76.11, which is 89.5 percent of FP16.

• License: Apache 2.0
• Cost to run: none named
• Data it reaches: the prompts you type and the downloaded files, plus images only if you load the vision pack
• Not for: runtimes without the author's 1-bit kernels, or work that needs full FP16 quality

On Hugging Face, 13 November 2026

• Task: text generation
• Library: llama.cpp
• Parameters: 26.9B
• Weights: about 13.4 GB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 179,818
• Likes: 945
• License: apache-2.0
• Based on: Qwen/Qwen3.6-27B
• Quantized versions: 8
• Fine-tunes: 7
• Tags: llama.cpp, gguf, conversational, 1-bit, llama-cpp, cuda, metal, on-device, hybrid-attention, prismml, bonsai, text-generation

Links

• Model: https://huggingface.co/prism-ml/Bonsai-27B-gguf
• Files: https://huggingface.co/prism-ml/Bonsai-27B-gguf/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download prism-ml/Bonsai-27B-gguf

Then follow the usage in its model card. Nothing in this kit runs by itself.