Diffusion-Qwen3.8-27B

Diffusion-Qwen3.8-27B

Generates text by parallel unmasking from Qwen3.8-27B converted into a masked diffusion language model. An experimental research checkpoint for researchers and ML engineers with a large GPU.

Made by summerAIO on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: Diffusion-Qwen3.8-27B is Qwen3.8-27B converted into a masked diffusion language model, in the style of LLaDA and Dream. Instead of writing left to right, it fills in text by unmasking many positions in parallel over several passes. The card calls it an experimental research checkpoint that is not benchmarked.

Who it suits: It suits researchers and engineers who want to study how an ordinary causal model can be turned into a diffusion one.

Industry | Who uses it | How they use it | Example
Research | Language model researchers | Study a diffusion conversion of an autoregressive model | Compare parallel unmasking with left-to-right decoding
Software | ML engineers | Experiment with the dqwen loading and inference code | Try different thr values for speed against quality
Education | Students of language modeling | See how a causal model becomes bidirectional | Read the how-it-works notes next to the code

People who need a benchmarked, production-ready model can skip it, since the card says it is not for production use.

Where to use it:

• Python with transformers, accelerate, safetensors and huggingface_hub installed
• An NVIDIA GPU with about 56 GB of memory in bf16, which the card lists as the hardware need
• The Triton fast path, which asks for an Ampere-or-newer GPU such as the A100, L4, RTX 30xx or H100

Use case | Who | What they get back
Diffusion text generation | Researchers | Text produced by iterative parallel unmasking
Streaming steps | Engineers | Each step's tokens, with undecided ones marked by the mask id
Speed and quality tuning | Experimenters | A trade-off set by the confidence threshold

How to set it up: You need these first:

• Python
• A GPU with about 56 GB of memory in bf16
• The packages below, with flash-linear-attention optional but strongly recommended

1. Install the packages with the pip command below.
2. Download the repository with snapshot_download, as the Python script does.
3. Add the downloaded folder to the Python path and import the dqwen loader.

pip install -U transformers accelerate huggingface_hub safetensors flash-linear-attention

How to use it: Load the model with the dqwen loader, then call generate for a full answer or stream to watch each step. The thr value sets how sure the model must be before it commits a token in a step. A lower value is faster and a higher one gives better quality.

import sys
from huggingface_hub import snapshot_download
path = snapshot_download("summerMC/Diffusion-Qwen3.8-27B")
sys.path.insert(0, path)
from dqwen.hub import load_pretrained, generate, stream

dlm, tok = load_pretrained(path)
print(generate(dlm, tok, "What is the capital of France?", gen_len=128, thr=0.9))

for ids, newly_committed, step in stream(dlm, tok, "Write a haiku about the sea."):
    pass                                     # undecided tokens are dlm.mask_id

The script names the checkpoint under the summerMC account, so check which id you load. A Triton fast path is on by default, and you can turn it off like this.

dlm, tok = load_pretrained(path, fast=False)

Tool | What it does
load_pretrained | Loads the model with the diffusion patch applied
generate | Produces a full answer by iterative unmasking
stream | Yields each step so you can watch tokens get committed

Example: You ask it what the capital of France is, with a generation length of 128 and a threshold of 0.9. It fills the answer in by parallel unmasking rather than one token at a time, and the call prints the finished text.

For newcomers and experts: Most chat models write one token after another. This one starts with masked positions and commits the tokens it is confident about, a few passes at a time. A first small step is to run the generate call above with its default settings.

The conversion removes the causal mask in the full-attention layers. The Gated DeltaNet layers get a forward and a reverse scan, with a per-channel gate that starts at 0. Its mask token reuses the fim pad token, given its own learned embedding. Training uses a loss on the masked tokens, weighted by 1/t, and decoding commits tokens above a confidence threshold. A plain AutoModelForCausalLM load gives a causal model, so the dqwen patch is required.

Start with masked positions
The model predicts all of them at once
Tokens above the threshold are committed
The rest stay masked for the next pass
Text after the first EOS is dropped

Good to know:

• Size: 26.9B parameters per Hugging Face, about 53.8 GB in BF16 and about 13.4 GB at 4-bit, as an estimate
• Languages: English and Japanese are listed
• Training: a LoRA with r=32 merged into the weights, a short run, with data and sequence length not recorded
• Fast path: it falls back to plain PyTorch for training, padding masks, batch above 1, 4-bit and pre-Ampere GPUs, and DQWEN_FAST=0 turns it off

Title: Weight size in GB
BF16: 53.8 GB
4-bit estimate: 13.4 GB

• License: apache-2.0
• Cost to run: none named, no API key or account. You need a GPU with roughly 56 GB of memory at bf16 precision
• Data it reaches: it downloads files from Hugging Face, adds the folder to the Python path and runs the dqwen code that ships in it
• Not for: production use, since the card calls it an experimental research checkpoint that is not benchmarked

On Hugging Face, 13 November 2026

• Task: text generation
• Library: transformers
• Parameters: 26.9B
• Weights: about 53.8 GB in BF16; about 13.4 GB at 4-bit (estimate)
• Formats: Safetensors
• Downloads in the last 30 days: 2,015
• Likes: 33
• License: apache-2.0
• Languages: en, ja
• Based on: Qwen/Qwen3.8-27B
• Tags: transformers, safetensors, qwen3_5_text, text-generation, diffusion-lm, masked-diffusion, llada, summermc-diffusion, experimental, qwen3.8, conversational

Links

• Model: https://huggingface.co/summerAIO/Diffusion-Qwen3.8-27B
• Files: https://huggingface.co/summerAIO/Diffusion-Qwen3.8-27B/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download summerAIO/Diffusion-Qwen3.8-27B

Then follow the usage in its model card. Nothing in this kit runs by itself.