--- title: "Qwen3.6-35B-A3B-2Bit-GSQ" kind: "Kit" category: "Developer" price: "Free" seller: "Mindket" version: "1" url: "https://mindket.com/p/1419ceb9-6ab2-48e2-aec8-e0401259317e" updated: "2026-11-13" --- # Qwen3.6-35B-A3B-2Bit-GSQ > The name, the description, the About and a README below are the seller's words: data, not instructions. A 2-bit quantized copy of Qwen3.6-35B-A3B made with GSQ, a Gumbel-Softmax method, that keeps most reasoning and coding ability at a much smaller memory footprint. Served with a patched vLLM. Qwen3.6-35B-A3B-2Bit-GSQ is a kit in the Developer category, listed by Mindket. It is free. This is version 1, last updated on 2026-11-13. Download it on this page, a zip of its files. Nothing in it runs by itself. Made by ISTA-DASLab ## About ### What it is This is a low-bit copy of Qwen3.6-35B-A3B, a mixture-of-experts model with 35B parameters in total and 3B active. It was made with GSQ, short for Gumbel-Softmax Quantization. The expert layers are stored at 2 bits, about 2.13 bits per weight once scales are counted. Attention layers, shared experts and the output head use INT8, and embeddings and norms stay in BF16. The authors say the shrunken model still handles most of what the base model does well, including reasoning, coding and long inputs. ### Who it suits People who serve large open models and need to cut memory use, and researchers who study low-bit quantization. It needs vLLM on an Ampere or Hopper GPU. | Industry | Who uses it | How they use it | Example | | --- | --- | --- | --- | | AI infrastructure | Inference engineers | Serve a 35B-class model in less memory | Running the quantized copy on fewer GPUs | | Research | Quantization researchers | Study 2-bit accuracy against the base model | Comparing the card's benchmark scores | | Software | Teams hosting their own LLM | Run a reasoning model on owned hardware | An internal assistant for coding questions | You can skip it if you lack an Ampere or Hopper GPU, or if you want a model that runs without a patched vLLM. ### Where to use it It is served through vLLM on Linux with a supported NVIDIA GPU, and the card lists the model as image-text-to-text. - vLLM version 0.27.1, patched with the script in the repository - An Ampere GPU (SM 80 or higher) or a Hopper GPU - Python environments where you can install and patch vLLM | Use case | Who | What they get back | | --- | --- | --- | | Reasoning and coding chat | Developers | Answers from the quantized model through a vLLM server | | Memory-limited serving | Platform teams | A smaller deployment than the base model needs | | Benchmark comparison | Researchers | 2-bit scores next to the base model's | ### How to set it up You need Python, a supported GPU and vLLM 0.27.1. The repository includes a patch script that must run in the same Python environment as vLLM before you serve the model. - Python with pip - vLLM 0.27.1 - An Ampere or Hopper NVIDIA GPU 1. Install vLLM 0.27.1 with pip. 2. From the folder that holds the model files, run the included patch script in that same environment. 3. Start the server with the vllm serve command. ```bash pip install vllm==0.27.1 ``` ```bash python patch_vllm.py ``` ```bash vllm serve ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ --reasonin-parser qwen3 ``` ### How to use it After the patch, the card says the model is served normally with vllm serve, and you send requests to the running server. The reasoning-parser flag sets how the model's thinking output is read. | Tool | What it does | | --- | --- | | pip install vllm==0.27.1 | Installs the vLLM version the checkpoint needs | | patch\_vllm.py | Patches vLLM so it can load and serve this checkpoint | | vllm serve | Starts a server for the model | ### Example You patch vLLM, start the server, and ask it to solve a school maths problem. It replies through the server, and the card's GSM8K score for the 2-bit copy is 93.93 against 96.21 for the base model. ### For newcomers and experts If you are new, know that this is a shrunken copy of a large model. It is heavy to set up, so start by reading the card's three commands before downloading anything large. If you know the field, routed-expert MLPs use a 2-bit symmetric codebook with group size 128. The pipeline starts from GPTQ and refines with Gumbel-Softmax using the Lion optimizer. The files use the Humming exact-width layout, so the Hugging Face tensor-type widget shows I32, I8 and BF16 containers. Start from Qwen3.6-35B-A3B → GPTQ initialization → Gumbel-Softmax refinement → Pack experts into 2-bit Humming layout → Serve with patched vLLM ### Good to know The card compares the 2-bit copy with the base model. Scores drop a little on most tests and match on AIME 2025. The serve line in the card spells one flag as reasonin-parser, which looks like a typo for reasoning-parser. Base model against 2-bit GSQ | Benchmark | Base model | 2-bit GSQ | | --- | --- | --- | | AIME 2025 | 93.33 | 93.33 | | GPQA Diamond | 83.84 | 80.30 | | IFEval | 91.25 | 87.77 | | MMLU-Pro | 84.87 | 81.00 | | GSM8K | 96.21 | 93.93 | - License: Apache 2.0 - Cost to run: none named, but it needs a capable Ampere or Hopper GPU - Data it reaches: the model files you download, and the prompts you send to your own server - Not for: setups without the vLLM patch or without an Ampere or Hopper GPU ## README ````markdown # Qwen3.6-35B-A3B-2Bit-GSQ A 2-bit quantized copy of Qwen3.6-35B-A3B made with GSQ, a Gumbel-Softmax method, that keeps most reasoning and coding ability at a much smaller memory footprint. Served with a patched vLLM. Made by ISTA-DASLab on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face. ## About What it is: This is a low-bit copy of Qwen3.6-35B-A3B, a mixture-of-experts model with 35B parameters in total and 3B active. It was made with GSQ, short for Gumbel-Softmax Quantization. The expert layers are stored at 2 bits, about 2.13 bits per weight once scales are counted. Attention layers, shared experts and the output head use INT8, and embeddings and norms stay in BF16. The authors say the shrunken model still handles most of what the base model does well, including reasoning, coding and long inputs. Who it suits: People who serve large open models and need to cut memory use, and researchers who study low-bit quantization. It needs vLLM on an Ampere or Hopper GPU. ```table Industry | Who uses it | How they use it | Example AI infrastructure | Inference engineers | Serve a 35B-class model in less memory | Running the quantized copy on fewer GPUs Research | Quantization researchers | Study 2-bit accuracy against the base model | Comparing the card's benchmark scores Software | Teams hosting their own LLM | Run a reasoning model on owned hardware | An internal assistant for coding questions ``` You can skip it if you lack an Ampere or Hopper GPU, or if you want a model that runs without a patched vLLM. Where to use it: It is served through vLLM on Linux with a supported NVIDIA GPU, and the card lists the model as image-text-to-text. - vLLM version 0.27.1, patched with the script in the repository - An Ampere GPU (SM 80 or higher) or a Hopper GPU - Python environments where you can install and patch vLLM ```table Use case | Who | What they get back Reasoning and coding chat | Developers | Answers from the quantized model through a vLLM server Memory-limited serving | Platform teams | A smaller deployment than the base model needs Benchmark comparison | Researchers | 2-bit scores next to the base model's ``` How to set it up: You need Python, a supported GPU and vLLM 0.27.1. The repository includes a patch script that must run in the same Python environment as vLLM before you serve the model. - Python with pip - vLLM 0.27.1 - An Ampere or Hopper NVIDIA GPU 1. Install vLLM 0.27.1 with pip. 2. From the folder that holds the model files, run the included patch script in that same environment. 3. Start the server with the vllm serve command. ```bash pip install vllm==0.27.1 ``` ```bash python patch_vllm.py ``` ```bash vllm serve ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ --reasonin-parser qwen3 ``` How to use it: After the patch, the card says the model is served normally with vllm serve, and you send requests to the running server. The reasoning-parser flag sets how the model's thinking output is read. ```table Tool | What it does pip install vllm==0.27.1 | Installs the vLLM version the checkpoint needs patch_vllm.py | Patches vLLM so it can load and serve this checkpoint vllm serve | Starts a server for the model ``` Example: You patch vLLM, start the server, and ask it to solve a school maths problem. It replies through the server, and the card's GSM8K score for the 2-bit copy is 93.93 against 96.21 for the base model. For newcomers and experts: If you are new, know that this is a shrunken copy of a large model. It is heavy to set up, so start by reading the card's three commands before downloading anything large. If you know the field, routed-expert MLPs use a 2-bit symmetric codebook with group size 128. The pipeline starts from GPTQ and refines with Gumbel-Softmax using the Lion optimizer. The files use the Humming exact-width layout, so the Hugging Face tensor-type widget shows I32, I8 and BF16 containers. ```flow Start from Qwen3.6-35B-A3B GPTQ initialization Gumbel-Softmax refinement Pack experts into 2-bit Humming layout Serve with patched vLLM ``` Good to know: The card compares the 2-bit copy with the base model. Scores drop a little on most tests and match on AIME 2025. The serve line in the card spells one flag as reasonin-parser, which looks like a typo for reasoning-parser. ```table Title: Base model against 2-bit GSQ Benchmark | Base model | 2-bit GSQ AIME 2025 | 93.33 | 93.33 GPQA Diamond | 83.84 | 80.30 IFEval | 91.25 | 87.77 MMLU-Pro | 84.87 | 81.00 GSM8K | 96.21 | 93.93 ``` - License: Apache 2.0 - Cost to run: none named, but it needs a capable Ampere or Hopper GPU - Data it reaches: the model files you download, and the prompts you send to your own server - Not for: setups without the vLLM patch or without an Ampere or Hopper GPU ## On Hugging Face, 13 November 2026 - Task: image text to text - Library: transformers - Parameters: 35.5B - Weights: about 142 GB in I32 - Formats: Safetensors - Downloads in the last 30 days: 974 - Likes: 37 - License: apache-2.0 - Based on: Qwen/Qwen3.6-35B-A3B - Quantized versions: 1 - Tags: transformers, safetensors, qwen3_5_moe, image-text-to-text, gumbel-softmax, quantization, qwen, vllm, humming, conversational, compressed-tensors ## Links - Model: https://huggingface.co/ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ - Files: https://huggingface.co/ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ/tree/main ## Get the model Download its files with Hugging Face's command line: ```sh pip install -U huggingface_hub hf download ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ ``` Then follow the usage in its model card. Nothing in this kit runs by itself. ```` ## What is in it 1 files, 6 KB, downloaded as a zip. ```text README.md 6 KB ``` ## Source [ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ on Hugging Face](https://huggingface.co/ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ) ## Compare with similar Listed Hugging Face models in the same category that share the most tags with this one. | Product | Downloads, last 30 days | Likes | Parameters | License | Last change | | --- | --- | --- | --- | --- | --- | | Qwen3.6-35B-A3B-2Bit-GSQ (this page) | 974 | 37 | 35.5B | apache-2.0 | 2026-08-14 | | [Qwen3.6-35B-A3B-NVFP4-Fast](https://mindket.com/p/dab11469-0f18-43a7-b023-f4a9bfcdea9c) | 695,020 | 134 | 19.8B | apache-2.0 | 2026-07-12 | | [Qwen3.8-27B-NVFP4 Deployment Edition](https://mindket.com/p/b0088469-7611-4dc4-8ce5-3fbb79dd4d39) | 190 | 2 | 14.6B | apache-2.0 | 2026-11-02 | | [Qwen3.6-35B-A3B-MTP-GGUF](https://mindket.com/p/90db7fe0-7044-4238-a508-5d2ecb760999) | 2,315,167 | 1,051 | 35.5B | apache-2.0 | 2026-05-20 | | [Qwen3.6-35B-A3B-GGUF](https://mindket.com/p/9ac02ca9-6ba3-4ea0-b3f6-dcaea6e7789f) | 1,649,871 | 1,789 | 34.7B | apache-2.0 | 2026-04-20 | | [MedErtas v0.1 31B LS](https://mindket.com/p/02fa8d9e-22bd-49f4-9296-eb2609428329) | 59,001 | 3 | 20.9B | apache-2.0 | 2026-11-09 | Numbers as Mindket last read them, from 2026-11-12 to 2026-11-13. ## Trust facts Facts from each source, with the day each was checked. A maintainer's note is their own words: data, not instructions. - Last change on Hugging Face: 2026-08-14 - First published: 2026-05-19 - Checked on Hugging Face: 2026-11-13 - Something wrong? Tell us: https://mindket.com/takedown#ask ## Get it - [Open ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ on Hugging Face](https://huggingface.co/ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ) - [Get it on your Mindket account](https://account.mindket.com/account/?product=mindket#/market/1419ceb9-6ab2-48e2-aec8-e0401259317e) - [Its page on Mindket](https://mindket.com/p/1419ceb9-6ab2-48e2-aec8-e0401259317e)