Qwen3.8-Flash-Next-GSQ-RCO-GGUF

Qwen3.8-Flash-Next-GSQ-RCO-GGUF

GGUF versions of Qwen3.8-Flash-Next at four sizes, with each tensor quantized at its own precision and a vision projector for image input. For people running a large mixture-of-experts model locally.

Made by ISTA-DASLab on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This repository offers GGUF quantizations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model, at four sizes. Each tensor gets its own quantization type, chosen by a search that spends more bits where the model is more sensitive, within a size budget. A vision projector file lets the model take images as well as text.

Who it suits: It suits people who run a large open model on their own machine and want a size that fits their memory.

Industry | Who uses it | How they use it | Example
Local AI | Hobbyists | Run a large multimodal model offline | Chat about a photo in LM Studio
Software | Developers | Serve a big model through llama.cpp or Ollama | Point a local tool at the loaded model
AI research | Quantization researchers | Study per-tensor bit allocation | Compare these files with uniform quantization
GPU and server teams | Infrastructure engineers | Pick a size that fits the memory they have | Choose among the four sizes by total file size

If your machine has far less memory than the 4-bit estimate of about 88.5 GB, you can skip it.

Where to use it:

• llama.cpp, which loads the files unmodified
• Ollama
• LM Studio
• Windows, macOS or Linux, with enough memory for the size you pick

Use case | Who | What they get back
Text chat | Local AI users | Answers from a 177B-parameter model on your own hardware
Image questions | Developers | Text answers about an image, using the mmproj file
Quantization study | Researchers | Files whose bit-width differs by tensor, ready to compare

How to set it up: You need these first:

• Python, for the Hugging Face download command
• llama.cpp, Ollama or LM Studio
• Memory and disk for the size you choose, with about 88.5 GB as the 4-bit estimate

1. Install the Hugging Face hub tool.
2. Download the repository files, or just the size you want.
3. Load the quantized model in your app, and add the mmproj file if you want image input.

pip install -U huggingface_hub
hf download ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF

How to use it: Pick one of the four size directories, load its weight shard in llama.cpp, Ollama or LM Studio, and add the mmproj file when you want to send images. Each quantized model has two shards.

Tool | What it does
Shard 1 | Holds the model weights for that size
Shard 2 | Carries a lookup table of n-gram embeddings for each layer, identical in all four sizes
mmproj file | Carries the vision encoder and projector in BF16 for every size
GSQ | Learns the quantization grid and scales for each tensor
RCO | Chooses a quantization type for each tensor within a size budget

Example: You load one size in LM Studio with the mmproj file, attach a screenshot and ask, "What does this error dialog say?" The model answers in text about the image.

For newcomers and experts: A quantized model is a smaller copy that trades some accuracy for less memory. Here each tensor gets its own setting. A good first step is to pick the largest size your memory allows and chat with it.

The model has 512 routed experts in each of 48 layers, and 10 are active per token. Since 95% of the searchable weights sit in the experts, the search assigns a type per layer, because GGUF cannot express it per expert. GSQ makes each tensor accurate at a given type, and RCO allocates the types under the budget.

GSQ quantizes each tensor at a given type
RCO assigns a type to each tensor
The result is saved as standard GGUF
llama.cpp, Ollama or LM Studio loads it

Good to know: A code-focused variant is released as a separate repository and keeps 256 of the 512 experts per layer.

Title: Approximate size in GB
This model at 4-bit (estimate): 88.5 GB
Code variant, total: 58.4 GB

• N-gram table: this per-layer table has 51.2B parameters, sits at 4.5 bpw in all four sizes and is not part of the search
• Other versions: 50 quantized and 1 fine-tuned versions made by other people are listed
• License: Apache 2.0
• Cost to run: none named, but the memory need is large
• Data it reaches: the model files you load, and any image you pair with the mmproj file

On Hugging Face, 13 November 2026

• Task: image text to text
• Library: gguf
• Parameters: 177B
• Weights: about 88.5 GB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 6,963,596
• Likes: 1,229
• License: apache-2.0
• Based on: Qwen/Qwen3.8-Flash-Next
• Quantized versions: 50
• Fine-tunes: 1
• Tags: gguf, quantization, mixed-precision, ist-daslab, multimodal, vision, image-text-to-text, imatrix, conversational

Links

• Model: https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF
• Files: https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF

Then follow the usage in its model card. Nothing in this kit runs by itself.