Qwen3.5-9B-OptiQ-4bit

Qwen3.5-9B-OptiQ-4bit

A 4-bit mixed-precision MLX version of Qwen3.5-9B for local text generation on Apple Silicon, with a bundled multi-token prediction head. For Mac users who run models locally.

Made by mlx-community on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: Qwen3.5-9B-OptiQ-4bit is a 4-bit mixed-precision MLX quantization of Qwen3.5-9B, made with the mlx-optiq toolkit. A sensitivity pass decides each layer's precision: sensitive layers stay at 8-bit and tolerant ones drop to 4-bit. The card says the disk size is within about 5 percent of a plain uniform 4-bit quant.

Who it suits: People who run language models on Apple Silicon Macs and want a compact 9B model are the main fit. Developers who want a local server with OpenAI-style and Anthropic-style endpoints can add the mlx-optiq tool.

Industry | Who uses it | How they use it | Example
Software | Developers | Run a local 9B model from Python with mlx-lm | Generate text with a token limit
AI tooling | Local LLM users | Serve the model with mixed-precision KV-cache and speculative decoding | Start the optiq server with the MTP head
Research | Quantization researchers | Compare mixed precision with plain uniform 4-bit | Read the six-metric comparison in the card

If you do not use an Apple Silicon Mac, you can skip it, because the toolkit and the format target that hardware.

Where to use it: Use it on Apple Silicon, from Python.

• Python scripts that use mlx-lm
• The optiq serve command from mlx-optiq
• Windows, macOS or Linux, with Python for the download

Use case | Who | What they get back
Local text generation | Developers | Generated text from a prompt, on your own machine
Local server | App builders | An OpenAI-compatible and Anthropic-compatible inference server
Faster decoding | Power users | About 1.4x faster decoding with the bundled MTP head

How to set it up: You need Python, the mlx-lm package and an Apple Silicon Mac.

• Python with pip
• An Apple Silicon Mac, which the toolkit is built for
• Disk space of about 6.6 GB, the on-disk size the card gives

1. Install mlx-lm.
2. Optionally install mlx-optiq for the server and extra tools.
3. Load the model by its name in Python.

pip install mlx-lm

pip install mlx-optiq

How to use it: Load the model with mlx-lm and call generate. To use the bundled multi-token prediction head, start the optiq server with the --mtp flag.

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Qwen3.5-9B-OptiQ-4bit")
response = generate(
    model, tokenizer,
    prompt="Explain quantum computing in simple terms.",
    max_tokens=200,
)

optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit --mtp

Tool | What it does
mlx-lm | Loads the model and generates text from a prompt
mlx-optiq | Adds mixed-precision KV-cache serving, LoRA fine-tuning, an OpenAI-compatible server, hot-swap adapters and sandboxed Python for agents
optiq serve --mtp | Serves the model with the bundled multi-token prediction head

Example: You load the model and ask it for a simple explanation of quantum computing, capped at 200 tokens. The call returns the generated text in the response variable.

For newcomers and experts: This is a smaller, compressed copy of a 9B chat model that runs on a Mac. A good first try is to install mlx-lm and run the Python snippet above as it is.

A KL-divergence sensitivity pass over a six-domain calibration mix picks each layer's bit width, using 40 samples per domain, and the group size is 64. The bundled mtp.safetensors head enables the speculative decoding, and the card reports an acceptance rate near 70 percent at depth 2. The trade-off is disk size, which is 1.1 GB above plain uniform 4-bit.

Start from Qwen3.5-9B
Measure each layer's sensitivity
Keep sensitive layers at 8-bit
Drop tolerant layers to 4-bit
Serve with or without the MTP head

Good to know: The card compares this quant with plain uniform 4-bit on six metrics, and the result is mixed.

Metric | OptiQ | Uniform 4-bit | Difference
MMLU | 69.1% | 68.6% | +0.5
GSM8K | 81.7% | 81.7% | +0.0
IFEval | 71.2% | 71.0% | +0.2
BFCL-V3 simple | 91.0% | 92.0% | -1.0
HumanEval | 81.1% | 78.7% | +2.4
HashHop | 25.0% | 26.0% | -1.0
Capability Score | 69.85 | 69.66 | +0.19
On-disk size | 6.6 GB | 5.5 GB | +1.1

Title: Layers by precision
8-bit layers: 132 layers
4-bit layers: 116 layers

• Other runtimes: if you load the head outside optiq serve, add 1.0 to its seven norm weights
• License: Apache 2.0
• Cost to run: none named
• Data it reaches: your prompts and the model files from Hugging Face. The card names no account or web service
• Not for: machines other than Apple Silicon, since the toolkit is built for that hardware

On Hugging Face, 13 November 2026

• Task: text generation
• Library: mlx
• Parameters: 9B
• Weights: about 35.8 GB in U32
• Formats: Safetensors
• Downloads in the last 30 days: 6,547
• Likes: 114
• License: apache-2.0
• Based on: Qwen/Qwen3.5-9B
• Tags: safetensors, qwen3_5, quantized, mixed-precision, 4bit, 8bit, optiq, apple-silicon, text-generation, qwen3.5, conversational, 4-bit

Links

• Model: https://huggingface.co/mlx-community/Qwen3.5-9B-OptiQ-4bit
• Files: https://huggingface.co/mlx-community/Qwen3.5-9B-OptiQ-4bit/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download mlx-community/Qwen3.5-9B-OptiQ-4bit

Then follow the usage in its model card. Nothing in this kit runs by itself.