Ornith-1.5-35B-A3B-GSQ-RCO-GGUF

Ornith-1.5-35B-A3B-GSQ-RCO-GGUF

Shrinks the Ornith-1.5-35B-A3B text model to 3-bit and 3.5-bit GGUF files, about 13 and 15 GB instead of 69 GB, for people who run local models with llama.cpp.

Made by pfeifferj on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: These are compressed GGUF copies of Ornith-1.5-35B-A3B, a mixture-of-experts text model of about 35B parameters from the Qwen3.5 family. They come in 3-bit and 3.5-bit sizes, roughly 13 GB and 15 GB against 69 GB for the BF16 original. An independent community member made them with the published GSQ and RCO methods, and they are not an official release of the method authors.

Who it suits: It suits people who run local models with llama.cpp and want a large model squeezed into a smaller download.

Title: Who uses this model
Industry | Who uses it | How they use it | Example
Software | Developers | Run a large coding and chat model locally | Serve it with llama-server for an app
Research | Quantization researchers | Study low-bit compression results | Compare the 3-bit and 3.5-bit builds against BF16
Self-hosting | Hobbyists | Fit a big model on modest hardware | Pick the 3-bit file when disk space is tight

If you need the exact quality of the BF16 original, you can skip it, because the compressed builds score lower on the card's tests.

Where to use it:

• llama.cpp, through llama-server
• Other programs that open GGUF files
• Windows, macOS or Linux with Python, for downloading the files

Title: Jobs it does
Use case | Who | What they get back
Local chat | Anyone | Text replies from a model running on your machine
Local API | Developers | A server that other programs can call
Compression study | Researchers | Perplexity and MMLU-Pro figures for each size

How to set it up: You need Python for the download and a llama.cpp build. The author tested with one specific upstream llama.cpp commit, so a recent build is the safe choice.

• Python with pip
• A working llama.cpp build that includes llama-server
• About 15.2 GB of disk space for the 3.5-bit file, or 13.0 GB for the 3-bit one

1. Install huggingface_hub.
2. Download the GGUF file you want.
3. Start llama-server with the model file and the settings below.

pip install -U huggingface_hub

hf download pfeifferj/Ornith-1.5-35B-A3B-GSQ-RCO-GGUF Ornith-1.5-35B-A3B-GSQ-RCO-3.5bit.gguf --local-dir .

llama-server -m Ornith-1.5-35B-A3B-GSQ-RCO-3.5bit.gguf \
  -ngl 999 -c 8192 -fa on -ctk f32 -ctv f32 --jinja \
  --temp 0.6 --top-p 0.95 --top-k 20 \
  --chat-template-kwargs '{"preserve_thinking":true}'

How to use it: Once llama-server runs, chat with it or call it from your apps. The sampling values in the command are the ones Ornith recommends. Thinking is on by default, and you can turn it off by setting enable_thinking to false in the chat template arguments.

Title: Main command pieces
Tool | What it does
hf download | Fetches one GGUF file into the current folder
llama-server | Serves the model on your machine
--temp, --top-p, --top-k | Set the sampling values the card recommends
--chat-template-kwargs | Keeps the model's reasoning visible across turns

Example: You download the 3.5-bit file and start the server with the command above. Then you send a reasoning question. The model thinks first and then replies in text, because thinking is on by default.

For newcomers and experts: Quantization stores a model's numbers in fewer bits so the files shrink. This model is already big, so the small files make it easier to run. A first small test is the 3-bit file with the command above.

GSQ learns how each weight is rounded and scaled through a Gumbel-Softmax relaxation. RCO picks a quantization type for each tensor under an exact total size budget. The card compares against matched initializers and native imatrix builds. Against BF16, perplexity is 3.7% higher for the 3-bit build and 1.4% higher for the 3.5-bit one.

Ornith-1.5-35B-A3B in BF16
GSQ learns the rounding
RCO assigns types per tensor
3-bit and 3.5-bit GGUF files

Good to know: On MMLU-Pro the card reports 47.60% for BF16, 43.70% for the 3-bit build and 43.20% for the 3.5-bit build. On MMLU-Pro, the matched initializer builds beat both GSQ builds. The files hold the text model only, with the tokenizer and chat template.

Title: File size by build
BF16 reference: 69.4 GB
GSQ-RCO 3.5-bit: 15.2 GB
GSQ-RCO 3-bit: 13.0 GB

• License: MIT.
• Cost to run: none named.
• Data it reaches: the files come from Hugging Face, and the model then runs on your own computer.
• Not for: anyone expecting BF16 quality, or images, since these files hold the text model only.

On Hugging Face, 13 November 2026

• Task: text generation
• Library: gguf
• Parameters: 34.7B
• Weights: about 17.3 GB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 11,634
• Likes: 30
• License: mit
• Based on: ornith-ai/Ornith-1.5-35B-A3B
• Tags: gguf, quantization, mixed-precision, text-generation, conversational

Links

• Model: https://huggingface.co/pfeifferj/Ornith-1.5-35B-A3B-GSQ-RCO-GGUF
• Files: https://huggingface.co/pfeifferj/Ornith-1.5-35B-A3B-GSQ-RCO-GGUF/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download pfeifferj/Ornith-1.5-35B-A3B-GSQ-RCO-GGUF

Then follow the usage in its model card. Nothing in this kit runs by itself.