Qwen3.8-27B-DSpine-b16-GGUF

Qwen3.8-27B-DSpine-b16-GGUF

A GGUF draft model that speeds up Qwen3.8-27B in llama.cpp through speculative decoding, reaching about 125 tokens per second on code and math on one RTX 3090. For people who run Qwen3.8-27B locally.

Made by aj9o9 on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This repository holds GGUF conversions of DSpine, a draft model that speeds up Qwen3.8-27B through speculative decoding in llama.cpp. A small drafter proposes tokens and the large model checks them. On one RTX 3090 with a Q4_K_M target, the card reports about 125 tokens per second on code and math.

Who it suits: It suits people who run Qwen3.8-27B on their own NVIDIA GPU and want faster text generation, especially for code.

Industry | Who uses it | How they use it | Example
Software | Developers | Speed up a local coding model | Code completion that returns sooner
Local AI | Hobbyists with a 24 GB GPU | Get more tokens per second from the same card | Faster chat on an RTX 3090
AI research | Inference researchers | Study speculative decoding drafters | Compare DSpine with DFlash2 on the same target
Infrastructure | ML engineers | Tune draft length for a given GPU | Test smaller drafts on cards where small batches are slow

If you need stock llama.cpp with no custom build, you can skip it, because upstream does not support DSpine yet.

Where to use it:

• The dspine branch of a llama.cpp fork named in the card, built with CUDA
• A GPU such as the RTX 3090, with 24 GB of memory
• Machines where you can build llama.cpp with cmake

Use case | Who | What they get back
Faster code generation | Developers | About 125 tokens per second on code and math, by the card's test
Faster chat | Local AI users | About 90 tokens per second on a mixed set of chat tasks
Tuning on other GPUs | Researchers | Reports from AMD, Intel and DGX Spark users that the card collects

How to set it up: You need these first:

• A target GGUF of Qwen3.8-27B, for example in Q4_K_M, because these files hold only the draft model
• An NVIDIA GPU, since the build command turns on CUDA
• Git, cmake and a C++ compiler
• Python, for the Hugging Face download command

1. Download the draft model files with the hub tool.
2. Get a Qwen3.8-27B target GGUF.
3. Clone the dspine branch of the fork the card names, then build it with the commands below.
4. Start the server with the run command in the next section.

pip install -U huggingface_hub
hf download aj9o9/Qwen3.8-27B-DSpine-b16-GGUF

cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

How to use it: Start llama-server with your target model and the draft model, then chat with it. Set the draft length and the context size yourself rather than using the defaults.

./build/bin/llama-server \
  -m  Qwen3.8-27B-Q4_K_M.gguf \
  -md Qwen3.8-27B-DSpine-b16-Q4_K_M.gguf \
  --spec-type draft-dflash \
  --spec-draft-n-max 15 \
  -ngl 999 --spec-draft-ngl 999 \
  -fa on --jinja -c 32768 -np 1 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --spec-draft-type-k q8_0 --spec-draft-type-v q8_0

Tool | What it does
-m | Loads the Qwen3.8-27B target model
-md | Loads the DSpine draft model
--spec-draft-n-max | Sets how many tokens are drafted, where 15 is best on NVIDIA CUDA and the default is 3
-c | Sets the context size, which you should choose yourself
--spec-draft-sampling probabilistic | Optional, and helps slightly when the temperature is above 0

Example: You start the server with the command above and ask for a Python function that merges two sorted lists. The target model checks each block of drafted tokens, so the answer arrives faster than without a drafter, while the target still decides what is kept.

For newcomers and experts: Speculative decoding lets a small model guess ahead so the big one can check many tokens at once. A good first step is to try the recommended Q4_K_M file with the settings above.

DSpine extends DFlash with a five-layer drafter. It takes the hidden states of five target layers as input and drafts 16 tokens at once, as one block. It adds causal block attention, a code bridge after each layer and a refinement step that builds a 16 by 16 transition table.

The draft model reads the target's hidden states
It drafts a block of 16 tokens
The target model checks the block
Accepted tokens are kept

Good to know: Speed depends on your GPU, and the numbers below come from one RTX 3090.

Title: Decode speed on one RTX 3090 in tokens per second
No speculative decoding: 28 tok/s
DFlash2: 52 tok/s
DSpine on mixed chat: 90 tok/s
DSpine on code and math: 125 tok/s

• Custom build: the standard llama.cpp will not open these files, so use the dspine branch of the fork
• Files: Q4_K_M at 1.26 GB is recommended, Q8_0 at 2.20 GB is slightly slower, and BF16 at 4.06 GB is for requantizing
• Context on 24 GB: with 15 drafted tokens it fits up to a context of 65536, and with 7 it fits up to 98304
• Other GPUs: a 15-token draft may cost more time than it saves when a card handles small batches slowly
• License: Apache 2.0
• Cost to run: none named, you supply the GPU
• Data it reaches: the prompts you send to your own local server

On Hugging Face, 13 November 2026

• Task: text generation
• Library: gguf
• Parameters: 2B
• Weights: about 1 GB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 15,022
• Likes: 41
• License: apache-2.0
• Based on: tencent/Qwen3.8-27B-DSpine-b16
• Tags: gguf, llama.cpp, dspine, dflash, speculative-decoding, draft-model, qwen3.8, text-generation, conversational

Links

• Model: https://huggingface.co/aj9o9/Qwen3.8-27B-DSpine-b16-GGUF
• Files: https://huggingface.co/aj9o9/Qwen3.8-27B-DSpine-b16-GGUF/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download aj9o9/Qwen3.8-27B-DSpine-b16-GGUF

Then follow the usage in its model card. Nothing in this kit runs by itself.