Qwen3.8-27B-Quark-AWQ-MXFP4

Qwen3.8-27B-Quark-AWQ-MXFP4

Runs Qwen3.8-27B as a smaller MXFP4-quantized copy made with AMD Quark and AWQ, served through vLLM. Made for developers who want a compressed 27B vision-language model with published GSM8K scores.

Made by amd on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This is Qwen3.8-27B compressed to 4-bit MXFP4 weights and MXFP4 activations with AMD Quark, using the AWQ method. Only the language decoder is quantized. The vision tower stays in bf16. The result is a smaller copy that keeps close to the base model's math scores on the card's test.

Who it suits: It suits developers who serve large models and want a smaller 27B copy, and researchers who study low-bit quantization on AMD tooling.

Industry | Who uses it | How they use it | Example
AI infrastructure | Model serving engineers | Serve a compressed 27B model through vLLM | Start the card's serve command and send chat requests
Research | Quantization researchers | Study MXFP4 with AWQ and its accuracy loss | Compare quantized and bf16 GSM8K scores
Software | Application developers | Run an image-and-text model with lower memory needs | Ask questions about an image through a local server
Evaluation | ML evaluators | Reproduce the benchmark on a local endpoint | Run the lm-eval commands from the card

If you do not run models on GPUs that support this format, or you only need small models, you can skip it.

Where to use it: It is served with vLLM, and the card's quantization steps use AMD's Quark with a HIP device setting.

• vLLM, as a local server
• The lm-eval tool, for GSM8K scoring
• AMD Quark, for repeating the quantization

Use case | Who | What they get back
Serving | Infrastructure teams | A local endpoint for the quantized model
Benchmarking | Evaluators | GSM8K accuracy in thinking and non-thinking modes
Re-quantizing | Researchers | A quantized copy produced from the base model

How to set it up: You need Python, vLLM and a GPU setup that supports MXFP4, plus enough memory for about 15.6 GB of weights. The serve command uses a 16384-token limit and one GPU.

1. Install the Hugging Face hub tools.
2. Download the model files.
3. Install vLLM.
4. Run the serve command with the model name.

pip install -U huggingface_hub
hf download amd/Qwen3.8-27B-Quark-AWQ-MXFP4

How to use it: Start the model with vLLM, then send requests to the local server. To repeat the quantization from the base model, the card also shows the Quark command.

vllm serve amd/Qwen3.8-27B-Quark-AWQ-MXFP4 \
  --served-model-name awq \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --max-model-len 16384

HIP_VISIBLE_DEVICES=<gpu> python3 quantize_quark.py \
  --model_dir Qwen/Qwen3.8-27B \
  --output_dir ./Qwen3.8-27B-w_mxfp4_a_mxfp4-awq \
  --quant_scheme mxfp4 \
  --num_calib_data 128 \
  --seq_len 512 \
  --quant_algo awq \
  --model_export hf_format \
  --data_type auto \
  --device cuda \
  --trust_remote_code

Tool | What it does
vllm serve | Serves the quantized model through a local endpoint
quantize_quark.py | Quantizes the base model with AMD Quark and AWQ
lm-eval | Scores the model on GSM8K through the local endpoint
hf download | Fetches the model files to your computer

Example: You start the serve command, then point an evaluation tool at the local server and run GSM8K math word problems. The card reports about 94.996 percent in thinking mode, next to 93.33 percent for the bf16 base.

For newcomers and experts: Quantizing stores a model's numbers with fewer bits so it needs less memory. A first step is to run the serve command and send one short question.

The weights use the OCP microscaling FP4 format, with groups of 32 values sharing an E8M0 scale, and the activations use the same format, computed at runtime. Calibration used 128 samples of length 512 from the pileval dataset. Recovery is the quantized score divided by the base score.

Start from Qwen3.8-27B in bf16
Calibrate with 128 samples
Apply AWQ with MXFP4 weights and activations
Serve the result with vLLM

Good to know: The card compares the quantized model to the bf16 base on GSM8K, and the quantized one scores higher in thinking mode but slightly lower in non-thinking mode.

Title: GSM8K 5-shot, flexible-extract accuracy
Quantized, thinking mode: 94.996 percent
BF16 base, thinking mode: 93.33 percent
Quantized, non-thinking mode: 89.92 percent
BF16 base, non-thinking mode: 90.67 percent

• Recovery: 101.8 percent in thinking mode and 99.2 percent in non-thinking mode.
• Scope: only the language decoder is quantized, and the vision tower is left in bf16.
• Non-thinking runs are approximated with a custom task that closes an empty think block in the prompt.
• License: Apache 2.0.
• Cost to run: none named. It needs a supported GPU you provide.
• Data it reaches: the prompts and images you send to your own server, and the pileval dataset used for calibration.
• Not for: the card names no exclusions.

On Hugging Face, 13 November 2026

• Task: image text to text
• Library: transformers
• Parameters: 15.6B
• Weights: about 15.6 GB in U8
• Formats: Safetensors
• Downloads in the last 30 days: 40,258
• Likes: 44
• License: apache-2.0
• Based on: Qwen/Qwen3.8-27B
• Quantized versions: 2
• Fine-tunes: 1
• Tags: transformers, safetensors, qwen3_5, image-text-to-text, quark, quantized, mxfp4, conversational, 8-bit

Links

• Model: https://huggingface.co/amd/Qwen3.8-27B-Quark-AWQ-MXFP4
• Files: https://huggingface.co/amd/Qwen3.8-27B-Quark-AWQ-MXFP4/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download amd/Qwen3.8-27B-Quark-AWQ-MXFP4

Then follow the usage in its model card. Nothing in this kit runs by itself.