Bonsai-1.7B-GGUF

Bonsai-1.7B-GGUF

Generates text with a 1.7B language model packed to 1 bit per weight, about 0.24 GB, for llama.cpp on CUDA, Metal and CPU. For developers who need a tiny local model.

Made by prism-ml on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: Bonsai-1.7B is a small text generation model where every weight, including embeddings and the output head, is stored in a single bit. It uses the Qwen3-1.7B design, packed in the GGUF Q1_0 format. The weights take about 0.24 GB, against 3.44 GB at FP16, and it runs in llama.cpp on CUDA, Metal and CPU.

Who it suits: Developers who need a language model on a small device or with little memory, and who are happy to use the maker's fork of llama.cpp.

Industry | Who uses it | How they use it | Example
Mobile apps | App developers | Run a small model on a phone or tablet | The card lists Swift for iPhone and iPad, and Android
Edge computing | Embedded engineers | Fit a language model into tight memory | Load a 0.24 GB model on a small machine
Hobby projects | Local AI tinkerers | Try 1-bit models on a laptop | Chat with the model through llama.cpp on a Mac

Anyone who needs the best answer quality can skip it, since a 1.7B model at 1 bit trades quality for size.

Where to use it:

• llama.cpp on CUDA GPUs, including RTX and datacenter cards.
• llama.cpp on Metal, for Macs.
• llama.cpp on CPU.
• Swift on iPhone and iPad, and Android.
• Windows, macOS or Linux: the download steps run on all three with Python.

Use case | Who | What they get back
On-device chat | App developers | Text replies from a model of about 0.24 GB
Low-memory servers | Engineers | A language model that loads in a fraction of the FP16 memory
Speed tests | Tinkerers | The card reports up to 3.0x faster than FP16 on an RTX 4090

How to set it up: What you need first:

• Python with pip, to download the file.
• The PrismML fork of llama.cpp, which includes the Q1_0 kernels the model needs.
• A CUDA GPU, a Mac with Metal, or a CPU.

1. Install the Hugging Face hub package with pip.
2. Download this repository with the hf command.
3. Build the PrismML fork of llama.cpp for your hardware, as the card describes.
4. Run the GGUF file with that build.

pip install -U huggingface_hub
hf download prism-ml/Bonsai-1.7B-gguf

How to use it: Run the file with the fork's llama.cpp tools. A short system prompt works, and the card suggests the sampling settings below.

You are a helpful assistant

Title: Suggested generation settings
Parameter | Default | Suggested range
Temperature | 0.5 | 0.5 to 0.7
Top-k | 20 | 20 to 40
Top-p | 0.9 | 0.85 to 0.95
Repetition penalty | 1.0 | none given

Example: You start the model in the fork's llama.cpp with the system prompt above and ask a short question. It returns a plain text answer, generated from a file under 0.3 GB.

For newcomers and experts: Most language models store each number in 16 bits. This one stores each in 1 bit, which is why the file is so small. A good first step is to download the file and chat with it in the fork of llama.cpp.

In Q1_0, each weight keeps only its sign, with 0 for negative and 1 for positive. One FP16 scale is shared by every 128 weights. That works out to 1.125 effective bits per weight. The design is Qwen3-1.7B with 28 layers, grouped-query attention and a context of 32,768 tokens. The trade-off is that you need runtime kernels for this format.

Download the Q1_0 GGUF file
Build the PrismML llama.cpp fork
Load the file on CUDA, Metal or CPU
Generate text with the suggested settings

Good to know: The card lists the size of the weights alone in three formats.

Title: Weight memory by format
FP16: 3.44 GB
MLX 1-bit g128: 0.27 GB
GGUF Q1_0: 0.24 GB

• File size: the file on disk is 0.25 GB, a little above the weights alone, since text-handling data and metadata are packed in with them.
• Size: 1.7B parameters, about 1.4B excluding embeddings.
• MLX: a separate 1-bit build exists for Apple Silicon.
• License: apache-2.0
• Cost to run: no API key, account or paid service named. The card also links a Colab notebook for trying it in a browser.
• Data it reaches: files you download from Hugging Face, and the prompts you type.
• Not for: runtimes without Q1_0 kernels, since the card points to the PrismML forks.

On Hugging Face, 13 November 2026

• Task: text generation
• Library: llama.cpp
• Parameters: 1.7B
• Weights: about 860 MB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 173,963
• Likes: 100
• License: apache-2.0
• Based on: prism-ml/Bonsai-1.7B-unpacked
• Quantized versions: 3
• Fine-tunes: 3
• Tags: llama.cpp, gguf, 1-bit, llama-cpp, cuda, metal, on-device, prismml, bonsai, text-generation, conversational

Links

• Model: https://huggingface.co/prism-ml/Bonsai-1.7B-gguf
• Files: https://huggingface.co/prism-ml/Bonsai-1.7B-gguf/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download prism-ml/Bonsai-1.7B-gguf

Then follow the usage in its model card. Nothing in this kit runs by itself.