Qwen3.6-35B-A3B 2-bit GSQ GGUF

Qwen3.6-35B-A3B 2-bit GSQ GGUF

Runs a 35B Qwen model in a 2-bit GSQ GGUF conversion with a separate vision projector for llama.cpp. For people who want a large model in a small file on CPU or a modest GPU.

Made by chtisgit on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This is a GGUF conversion of the Qwen3.6-35B-A3B 2-bit GSQ checkpoint from ISTA-DASLab, repackaged into stock llama.cpp types. Nothing here is newly quantized or trained, since the weights come from the DASLab checkpoint. The files are a text-model GGUF plus a separate vision-projector GGUF, so the model can read images as well as text.

Who it suits: It suits people who want to run a 35B model with routed experts in a small file, using standard llama.cpp on a CPU or a modest graphics card.

Industry | Who uses it | How they use it | Example
Local AI | Hobbyists with a mid-range GPU | Run a large model from a small 2-bit file | The card tested it with part of the layers offloaded to an RTX 3060
Software development | Teams building on llama.cpp | Try 2-bit GSQ weights without a custom runtime | The files load in a current stock llama.cpp build
Research | Quantization researchers | Study how GSQ 2-bit maps onto Q2_0 | The card documents the mapping and its small errors

You can skip it if you cannot run a current llama.cpp build, because older releases cannot load it correctly.

Where to use it:

• llama.cpp on Windows, macOS or Linux
• CPU-only runs, which the card tested and which avoid the CUDA crash
• NVIDIA GPUs with llama.cpp build b11390 or newer

Use case | Who | What they get back
Chat with a large local model | Local model users | A 35B model that fits in a small text file
Ask about an image | Users with the projector loaded | Text read or described from the picture
Check a conversion | Researchers | A documented mapping from GSQ types to llama.cpp types

How to set it up: You need a current llama.cpp build and both model files.

• A current llama.cpp build, because Q2_0 is a recent type that older releases cannot load
• On NVIDIA GPUs, build b11390 or newer
• Python with the Hugging Face tools, to download the files
• Room for the text file of 11.38 GiB plus the small projector

1. Install the Hugging Face tools with Python.
2. Download the repository files with the second command below.
3. Update llama.cpp to a current build, and use b11390 or newer on NVIDIA GPUs.
4. Load the text model together with the projector file.

pip install -U huggingface_hub
hf download chtisgit/Qwen3.6-35B-A3B-2Bit-GSQ-hybrid-GGUF

How to use it: The card shows no run command, so follow the llama.cpp documentation for loading a text model with a vision projector. Load the text GGUF for chat, and add the projector GGUF when you want the model to read images.

Tool | What it does
Qwen3.6-35B-A3B-GSQ-hybrid.gguf | The text model
mmproj-BF16.gguf | The vision projector for image input

Example: You load both files in a current llama.cpp build and show the model a newspaper test image. The card reports that runs on CPU and with partial offload to an RTX 3060 both read the headline MEN WALK ON MOON correctly.

For newcomers and experts: Think of it as a large model squeezed into a small file, with a second file that lets it see pictures. A first small try is to download both files and load them in a current llama.cpp build.

The routed experts are stored as Q2_0. The attention layers, the shared expert and the LM head are stored as Q8_0. A GSQ 2-bit code maps to Q2_0 by reversing the code order and negating the scale. Groups of 128 weights become two Q2_0 blocks of 64, with no dequantize or requantize step. The only numerical change is the BF16 scale becoming FP16. One MTP layer is left out.

Q2_0 code = 3 - c
Q2_0 scale = FP16(-s)

Take the GSQ 2-bit checkpoint
Map routed experts to Q2_0 by reversing codes
Copy INT8 weights into Q8_0 blocks
Write a text GGUF and a projector GGUF
Load both in a current llama.cpp build

Good to know: The two files are described in the card as follows.

Title: The two model files
Property | Value
Text model | Qwen3.6-35B-A3B-GSQ-hybrid.gguf
Text-model size | 11.38 GiB
Vision projector | mmproj-BF16.gguf
Projector size | 860.9986 MiB

• Crash on older builds: older CUDA builds fail on prompts of certain lengths with an illegal memory access error, and CPU-only builds are not affected
• Tests: the card ran load and short-generation checks, not benchmarks or perplexity runs
• Left out: one MTP layer for speculative decoding
• Size: 34.7B parameters
• License: Apache 2.0
• Cost to run: none named
• Data it reaches: none named beyond the files you download
• Not for: older llama.cpp releases, which cannot load or run it correctly

On Hugging Face, 13 November 2026

• Task: image text to text
• Library: llama.cpp
• Parameters: 34.7B
• Weights: about 17.3 GB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 9,309
• Likes: 18
• License: apache-2.0
• Based on: ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ
• Tags: llama.cpp, gguf, qwen, quantization, image-text-to-text, conversational

Links

• Model: https://huggingface.co/chtisgit/Qwen3.6-35B-A3B-2Bit-GSQ-hybrid-GGUF
• Files: https://huggingface.co/chtisgit/Qwen3.6-35B-A3B-2Bit-GSQ-hybrid-GGUF/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download chtisgit/Qwen3.6-35B-A3B-2Bit-GSQ-hybrid-GGUF

Then follow the usage in its model card. Nothing in this kit runs by itself.