Qwen3.8-27B-NVFP4 Deployment Edition

Qwen3.8-27B-NVFP4 Deployment Edition

Serves Qwen3.8-27B in NVFP4 on one RTX 5090 with vLLM, with a checksum-verified download, FP8 KV cache, tool calls and reasoning support. For developers self-hosting a large model on Blackwell GPUs.

Made by IteraCompute on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: A deployment release of the Qwen3.8-27B model in NVIDIA's NVFP4 format, kept by IteraCompute for efficient serving on Blackwell GPUs. It comes with a serving recipe for one RTX 5090 and a checksum list to verify your download. The original tensors and model configuration are unchanged, so this edition documents deployment work and is not a new training or quantization run.

Who it suits: Developers who want to host a large model on their own Blackwell GPU and expose it through an OpenAI-style API with reasoning and tool calls.

Industry | Who uses it | How they use it | Example
Software | Backend developers | Serve a chat endpoint on their own machine | A local endpoint that other apps call
AI engineering | Agent builders | Use the tool-call and reasoning parsers in the recipe | An agent that calls tools through the server
Infrastructure | Inference engineers | Test NVFP4 serving on a single RTX 5090 | Try different engine settings on one card
Small teams | Teams with one workstation GPU | Self-host a large model instead of renting one | A shared internal chat server

You can skip it if your GPU is not a compatible Blackwell card, since NVFP4 needs that hardware and engine support.

Where to use it: The recipe targets vLLM on a machine with one RTX 5090 with 32 GB of memory.

• vLLM 0.31.0 with a CUDA and Blackwell build, the version the card's deployment used
• One RTX 5090 with 32 GB per instance
• Any OpenAI-compatible client that can call the local server
• Windows, macOS or Linux with Python, to download the files

Use case | Who | What they get back
Local chat server | Developers | An OpenAI-style chat endpoint on their own GPU
Tool-calling agents | Agent builders | Parsed tool calls and separated reasoning in the replies
Long-context work | Researchers | A configured context of 262,144 tokens, limited by memory

How to set it up: Download and verify the files, install a Blackwell-ready vLLM in its own environment, then start the server with the card's flags.

• A compatible Blackwell GPU, such as the RTX 5090 used in the card
• A CUDA and Blackwell build of vLLM, installed in a separate environment
• About 17.92 GB of disk for the two weight shards
• Python, for the Hugging Face download command

1. Download the model into a local folder.
2. Check the files against the SHA256SUMS list.
3. Install vLLM in a separate environment.
4. Run the vllm serve command with the card's settings.
5. Send a chat request to the local server.

hf download IteraCompute/Qwen3.8-27B-NVFP4 --local-dir ./Qwen3.8-27B-NVFP4
cd Qwen3.8-27B-NVFP4
sha256sum -c SHA256SUMS

vllm serve ./Qwen3.8-27B-NVFP4 \
  --served-model-name qwen3.8-27b-nvfp4 \
  --host 127.0.0.1 --port 8000 \
  --quantization modelopt \
  --tensor-parallel-size 1 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.97 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 2048 \
  --kv-cache-dtype fp8 \
  --attention-backend FLASHINFER \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3

How to use it: Once the server runs, call its OpenAI-style chat completions route with the model name qwen3.8-27b-nvfp4. The card's own test turns thinking off by passing enable_thinking as false in the chat template options. The repository ships its own chat template, and that is the one to use when you want reasoning or tools.

Tool | What it does
sha256sum -c SHA256SUMS | Checks the checkpoint, serving files and preserved documents
--quantization modelopt | Reads the NVFP4 weights made by NVIDIA Model Optimizer
--kv-cache-dtype fp8 | Stores the attention cache in FP8 to save memory
--enable-chunked-prefill | Splits long prompts into chunks
--enable-prefix-caching | Reuses cached work for repeated prompt starts
--tool-call-parser qwen3_xml | Turns the model's tool calls into structured ones
--reasoning-parser qwen3 | Separates the model's reasoning from its answer

Example: Start the server, then send a single user message saying Hello, with a limit of 256 new tokens and thinking switched off. The server returns a normal chat completion in the OpenAI response format.

For newcomers and experts: This page is about running a big model on your own graphics card. A first small step is to download the files and run the checksum command, which proves the download is intact before you set anything up.

For experts, NVFP4 comes from NVIDIA Model Optimizer, with layer coverage listed in config.json and hf_quant_config.json. The deployment adds an FP8 KV cache, chunked prefill, prefix caching and the FlashInfer attention backend, at 0.97 GPU memory use and 16 sequences. The trade is that real context length and the number of parallel requests are set by your engine flags and free memory, whatever length the configuration states.

Download the weights
Verify them with SHA256SUMS
Install vLLM for Blackwell
Start vllm serve with the flags
Send OpenAI-style requests

Good to know: Read these limits before you plan around it.

• Checkpoint: two Safetensors shards of about 17.92 GB in total.
• Hardware: NVFP4 needs compatible Blackwell hardware and engine support.
• No new tests: this release adds no new quality evaluation or benchmark.
• Independent: it is a community distribution, unaffiliated with Qwen or NVIDIA.
• License: Apache 2.0, with the LICENSE file and upstream notices kept.
• Cost to run: none named for the files, and the maintainer separately offers a hosted API.
• Data it reaches: the files downloaded from Hugging Face, and requests sent to your own server, which the example binds to localhost.
• Not for: GPUs without Blackwell support.

On Hugging Face, 13 November 2026

• Task: image text to text
• Library: transformers
• Parameters: 14.6B
• Weights: about 14.6 GB in U8
• Formats: Safetensors
• Downloads in the last 30 days: 190
• Likes: 2
• License: apache-2.0
• Languages: en, zh
• Based on: Qwen/Qwen3.8-27B
• Tags: transformers, safetensors, qwen3_5, image-text-to-text, qwen3.8, nvfp4, modelopt, blackwell, rtx-5090, vllm, quantization, conversational

Links

• Model: https://huggingface.co/IteraCompute/Qwen3.8-27B-NVFP4
• Files: https://huggingface.co/IteraCompute/Qwen3.8-27B-NVFP4/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download IteraCompute/Qwen3.8-27B-NVFP4

Then follow the usage in its model card. Nothing in this kit runs by itself.