Arete-27B-GGUF

Arete-27B-GGUF

Runs Nous Research's Arete 27B, a Qwen3.8-27B fine-tune that reads text and images, from GGUF files in six sizes with measured quality losses. For people running big models on their own GPU.

Made by AtomicChat on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: Arete-27B-GGUF holds quantized GGUF builds of Arete-27B, which is Nous Research's fine-tune of Qwen3.8-27B. AtomicChat made the builds from Nous's original weights with its own importance matrix. The card says Nous trained the model to change how it answers politically sensitive questions, and points to Nous's card for the details.

Who it suits: People who run large language models locally and want a GGUF file sized to their graphics card, with measured numbers for each size.

Industry | Who uses it | How they use it | Example
Software development | Local-model users | Run a 27B model offline on their own GPU | A local chat model on a 24 GB card
Research | Model researchers | Compare quantization levels on one model | Reading the KL divergence of each file
AI tooling | App builders | Serve the model through llama.cpp | A local server answering chat requests
Education | Students | Study how quantization trades size for quality | Comparing a 16.2 GB file with a 28.6 GB one

If your graphics card has under about 16 GB of memory, you can skip it.

Where to use it:

• llama.cpp, through its llama-server program
• Atomic Chat, the maker's app for running open-weight models locally and offline
• Windows, macOS or Linux, after you fetch the files with Python

Use case | Who | What they get back
Local chat | Individuals | Answers from a 27B model without sending data away
Image reading | Individuals | Descriptions of images, with one extra vision file
Quantization study | Researchers | Size, KL divergence and top-1 agreement per file

How to set it up:

• Python and pip
• Enough disk space and graphics memory for the file you pick, from 13.6 GB to 28.6 GB
• llama.cpp, or another app that loads GGUF files

1. Install the Hugging Face hub tools.
2. Download the model files.
3. Pick a file that fits your card, as the card's table advises, and start a server with it.

pip install -U huggingface_hub
hf download AtomicChat/Arete-27B-GGUF

How to use it: Start llama-server with the file you chose. The chat template is stored inside every file, and llama.cpp applies it by itself. Thinking is on by default, and you can turn it off or shorten it with a template option.

llama-server -m Arete-27B-AD-Q4_K_M.gguf -ngl 99 -c 8192

# no thinking
llama-server -m Arete-27B-AD-Q4_K_M.gguf -ngl 99 -c 8192 --chat-template-kwargs '{"enable_thinking": false}'

# shorter thinking
llama-server -m Arete-27B-AD-Q4_K_M.gguf -ngl 99 -c 8192 --chat-template-kwargs '{"reasoning_effort": "low"}'

The reasoning effort can be xhigh, which is the default, medium or low. The value high makes the template stop with an error.

Tool | What it does
llama-server | Serves the chosen GGUF file for chat
enable_thinking | Template option that turns thinking off when set to false
reasoning_effort | Template option that sets how long the model thinks

Example: You start llama-server with the Q4_K_M file on a 24 GB card and send it a chat question. With thinking on, the model reasons first and then answers. Switching thinking off gives a direct answer.

For newcomers and experts: If you are new to GGUF, match the file to your graphics memory using the card's table, then start llama-server and chat. If two neighboring files both fit, the card suggests taking the larger one.

The card's numbers are measured, not estimated. KL divergence measures how much a file's output distribution shifts from the full model. Top-1 is the share of times the file and the full model choose the same next word. The AD files use an Atomic Dynamic layout, named after the types of the two largest tensor groups. Each token of context costs the model 64 KB of attention cache, so plan memory for the context as well as the file. It has no multi-token prediction head.

Pick a file that fits your card
Start llama-server with it
The chat template applies itself
Chat, with thinking on or off

Good to know:

• Sizes: the files run from 13.6 GB to 28.6 GB, and a larger context needs more memory
• Quality: smaller files drift further from the original weights

Title: Top-1 agreement with the original weights, in percent
Q8_0: 98.87%
AD-Q6_K: 98.58%
AD-Q5_K_M: 97.48%
AD-Q4_K_M: 95.89%
AD-IQ4_XS: 95.60%
AD-IQ3_S: 92.31%

• License: Apache-2.0
• Cost to run: no API key or paid service is named, only your own GPU
• Data it reaches: downloading fetches model files, and chats run locally on your machine
• Not for: cards under about 16 GB, and clients that send a reasoning effort of high, which gets no reply

On Hugging Face, 13 November 2026

• Task: image text to text
• Library: gguf
• Parameters: 26.9B
• Weights: about 13.4 GB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 1,933
• Likes: 15
• License: apache-2.0
• Based on: NousResearch/Arete-27B
• Tags: gguf, llama.cpp, arete, nousresearch, imatrix, quantized, conversational, atomic-chat, image-text-to-text

Links

• Model: https://huggingface.co/AtomicChat/Arete-27B-GGUF
• Files: https://huggingface.co/AtomicChat/Arete-27B-GGUF/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download AtomicChat/Arete-27B-GGUF

Then follow the usage in its model card. Nothing in this kit runs by itself.