Qwen3.6-35B-A3B-2Bit-GSQ

Qwen3.6-35B-A3B-2Bit-GSQ

A 2-bit quantized copy of Qwen3.6-35B-A3B made with GSQ, a Gumbel-Softmax method, that keeps most reasoning and coding ability at a much smaller memory footprint. Served with a patched vLLM.

Made by ISTA-DASLab on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This is a low-bit copy of Qwen3.6-35B-A3B, a mixture-of-experts model with 35B parameters in total and 3B active. It was made with GSQ, short for Gumbel-Softmax Quantization.

The expert layers are stored at 2 bits, about 2.13 bits per weight once scales are counted. Attention layers, shared experts and the output head use INT8, and embeddings and norms stay in BF16. The authors say the shrunken model still handles most of what the base model does well, including reasoning, coding and long inputs.

Who it suits: People who serve large open models and need to cut memory use, and researchers who study low-bit quantization. It needs vLLM on an Ampere or Hopper GPU.

Industry | Who uses it | How they use it | Example
AI infrastructure | Inference engineers | Serve a 35B-class model in less memory | Running the quantized copy on fewer GPUs
Research | Quantization researchers | Study 2-bit accuracy against the base model | Comparing the card's benchmark scores
Software | Teams hosting their own LLM | Run a reasoning model on owned hardware | An internal assistant for coding questions

You can skip it if you lack an Ampere or Hopper GPU, or if you want a model that runs without a patched vLLM.

Where to use it: It is served through vLLM on Linux with a supported NVIDIA GPU, and the card lists the model as image-text-to-text.

• vLLM version 0.27.1, patched with the script in the repository
• An Ampere GPU (SM 80 or higher) or a Hopper GPU
• Python environments where you can install and patch vLLM

Use case | Who | What they get back
Reasoning and coding chat | Developers | Answers from the quantized model through a vLLM server
Memory-limited serving | Platform teams | A smaller deployment than the base model needs
Benchmark comparison | Researchers | 2-bit scores next to the base model's

How to set it up: You need Python, a supported GPU and vLLM 0.27.1. The repository includes a patch script that must run in the same Python environment as vLLM before you serve the model.

• Python with pip
• vLLM 0.27.1
• An Ampere or Hopper NVIDIA GPU

1. Install vLLM 0.27.1 with pip.
2. From the folder that holds the model files, run the included patch script in that same environment.
3. Start the server with the vllm serve command.

pip install vllm==0.27.1

python patch_vllm.py

vllm serve ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ --reasonin-parser qwen3

How to use it: After the patch, the card says the model is served normally with vllm serve, and you send requests to the running server. The reasoning-parser flag sets how the model's thinking output is read.

Tool | What it does
pip install vllm==0.27.1 | Installs the vLLM version the checkpoint needs
patch_vllm.py | Patches vLLM so it can load and serve this checkpoint
vllm serve | Starts a server for the model

Example: You patch vLLM, start the server, and ask it to solve a school maths problem. It replies through the server, and the card's GSM8K score for the 2-bit copy is 93.93 against 96.21 for the base model.

For newcomers and experts: If you are new, know that this is a shrunken copy of a large model. It is heavy to set up, so start by reading the card's three commands before downloading anything large.

If you know the field, routed-expert MLPs use a 2-bit symmetric codebook with group size 128. The pipeline starts from GPTQ and refines with Gumbel-Softmax using the Lion optimizer. The files use the Humming exact-width layout, so the Hugging Face tensor-type widget shows I32, I8 and BF16 containers.

Start from Qwen3.6-35B-A3B
GPTQ initialization
Gumbel-Softmax refinement
Pack experts into 2-bit Humming layout
Serve with patched vLLM

Good to know: The card compares the 2-bit copy with the base model. Scores drop a little on most tests and match on AIME 2025. The serve line in the card spells one flag as reasonin-parser, which looks like a typo for reasoning-parser.

Title: Base model against 2-bit GSQ
Benchmark | Base model | 2-bit GSQ
AIME 2025 | 93.33 | 93.33
GPQA Diamond | 83.84 | 80.30
IFEval | 91.25 | 87.77
MMLU-Pro | 84.87 | 81.00
GSM8K | 96.21 | 93.93

• License: Apache 2.0
• Cost to run: none named, but it needs a capable Ampere or Hopper GPU
• Data it reaches: the model files you download, and the prompts you send to your own server
• Not for: setups without the vLLM patch or without an Ampere or Hopper GPU

On Hugging Face, 13 November 2026

• Task: image text to text
• Library: transformers
• Parameters: 35.5B
• Weights: about 142 GB in I32
• Formats: Safetensors
• Downloads in the last 30 days: 974
• Likes: 37
• License: apache-2.0
• Based on: Qwen/Qwen3.6-35B-A3B
• Quantized versions: 1
• Tags: transformers, safetensors, qwen3_5_moe, image-text-to-text, gumbel-softmax, quantization, qwen, vllm, humming, conversational, compressed-tensors

Links

• Model: https://huggingface.co/ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ
• Files: https://huggingface.co/ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ

Then follow the usage in its model card. Nothing in this kit runs by itself.