Qwen3.8-27B-QSA-GSQ-RCO-GGUF

Qwen3.8-27B-QSA-GSQ-RCO-GGUF

Runs Qwen3.8-27B with sparse attention on a single 24 GB GPU at a 262K-token context with image input, as GGUF files for people with a custom llama.cpp engine build.

Made by pentacoxian-dev on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: The files here let you run Qwen3.8-27B with Qwen's sparse attention design on a single 24 GB graphics card. It handles a context of 262K tokens and accepts images. The files need a special build of llama.cpp from pentaCoxian, because a standard llama.cpp build will not load these weights.

Who it suits: It suits advanced users with a 24 GB GPU who want to read long documents or large codebases with a 27B model on their own machine.

Title: Who uses this model
Industry | Who uses it | How they use it | Example
Software | Developers | Query a large codebase locally | Ask where a function is defined in a large repository
Research | Local-model enthusiasts | Test sparse attention at long context | Compare sparse and dense attention recall
AI infrastructure | Self-hosters | Serve a 27B model on one consumer GPU | Run the serve script on an RTX 4090

If you cannot build the custom engine or your GPU has less than 24 GB, you can skip it.

Where to use it:

• The Qwen-QSA-llama.cpp engine built from its source
• A machine with one 24 GB GPU
• Windows, macOS or Linux with Python, for downloading the files

Title: Jobs it does
Use case | Who | What they get back
Long-context questions | Developers | Answers drawn from inputs up to 262K tokens
Image questions | Anyone with pictures to check | Text answers about an image you supply
Faster decoding | Self-hosters | Draft-assisted generation with the bundled drafter

How to set it up: You need a built checkout of the engine, and its README has the build steps. The three files total about 14 GB.

• Python with pip, for the hf command
• A built copy of the Qwen-QSA-llama.cpp engine
• One 24 GB GPU

1. Build the engine by following its README.
2. Download the model files into a models folder.
3. Run the serve script from the engine checkout.

pip install -U huggingface_hub

hf download pentacoxian-dev/Qwen3.8-27B-QSA-GSQ-RCO-GGUF --local-dir models
deploy/qsa27b-24gb/serve.sh

How to use it: Run the two commands from inside the built engine checkout. The script serves the main model, the drafter and the vision projector together. A SHA256SUMS file lists checksums so you can verify the downloads.

Title: Files in the repository
Tool | What it does
Main IQ3_S model file | The quantized 27B weights, with the sparse-attention indexers and an MTP head built in, 12.3 GB
Qwen3.8-27B-DFlash2-IQ4_XS.gguf | Drafter for speculative decoding, 1.0 GB
mmproj-Qwen3.8-27B-QSA-F16.gguf | Vision projector, 0.9 GB

Example: You start the server, then give it a long source file and ask for the definition of one function. The model answers in text from its sparse attention, and the card's recall test reports 67% of exact lookups fully correct at 255K tokens.

For newcomers and experts: This is an advanced setup. Normal attention looks at every earlier token, while sparse attention looks at a chosen few, which saves speed and memory on long inputs. A first small step is to read the engine README before downloading anything.

In each of its 16 attention layers, a small indexer chooses 2,048 tokens for the model to look at. The original indexers were trained at 16K, so at 255K their picks fell behind. The authors retrained only the indexers, about 53M parameters, against the model's own dense attention. Turning sparse attention off with LLAMA_QSA_DENSE=1 shows dense results at about half the decode speed.

Qwen3.8-27B IQ3_S base file
QSA indexers added and retrained
Drafter and vision projector alongside
Served by the custom engine

Good to know: On one RTX 4090 at 262K context the author measured about 114.7 tokens per second for greedy decoding and a peak of 22.8 GiB of video memory. Beyond 64K, sparse attention still misses some lookups, most often when the target record is in the middle of the text.

Title: Exact-lookup recall at 255K context
Previous indexers: 41%
Retrained indexers: 67%
Dense attention: 92%

• License: Apache 2.0.
• Cost to run: none named, apart from a 24 GB GPU.
• Data it reaches: the files come from Hugging Face, and the model then runs on your own computer.
• Not for: stock llama.cpp, which cannot run these sparse-attention weights.

On Hugging Face, 13 November 2026

• Parameters: 1.9B
• Weights: about 962 MB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 2,784
• Likes: 6
• License: apache-2.0
• Based on: Qwen/Qwen3.8-27B
• Tags: gguf, llama.cpp, gsq-rco, conversational

Links

• Model: https://huggingface.co/pentacoxian-dev/Qwen3.8-27B-QSA-GSQ-RCO-GGUF
• Files: https://huggingface.co/pentacoxian-dev/Qwen3.8-27B-QSA-GSQ-RCO-GGUF/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download pentacoxian-dev/Qwen3.8-27B-QSA-GSQ-RCO-GGUF

Then follow the usage in its model card. Nothing in this kit runs by itself.