Cola DLM

Cola DLM

A research text-generation model that writes through a continuous latent space, pairing a text autoencoder with a diffusion transformer. For researchers and developers who want to run and test the released checkpoint.

Made by ByteDance-Seed on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: Cola DLM is a diffusion language model that works in a continuous latent space. A text VAE turns text into latent sequences and back into tokens, and a block-causal diffusion transformer models those latents using Flow Matching. This repository holds the checkpoint released with the paper Continuous Latent Diffusion Language Model.

Who it suits: It suits researchers and developers who want to study or test a latent diffusion approach to text generation. It is released as a research checkpoint, so expect to run code rather than chat in an app.

Title: Who uses it
Industry | Who uses it | How they use it | Example
AI research | Language-model researchers | Run the released checkpoint and compare it with other designs | Reproduce the zero-shot results the card lists
Software | Developers | Serve it behind an OpenAI-style endpoint | Send a chat request to the local adapter
Education | Students of generative models | Read how a text VAE and a diffusion prior fit together | Inspect the two modules in the repository

People who want a polished assistant can skip it, since several benchmark scores on the card are low.

Where to use it:

• Python with PyTorch and Transformers
• A local server through the OpenAI-compatible adapter
• Windows, macOS or Linux, on a GPU or a CPU

Title: Top jobs
Use case | Who | What they get back
Task prompts | Researchers | Generated text for a prompt such as a question
Local chat endpoint | Developers | Non-streaming chat completions from the adapter
Benchmark checks | Researchers | Zero-shot accuracy on tasks like LAMBADA and SQuAD

How to set it up:

• Python on your computer
• PyTorch 2.1 or newer and Transformers 4.40 or newer
• The Cola DLM code package from its project repository
• The huggingface_hub package

1. Clone the Cola DLM code repository and move into its folder.
2. Install the code package and the download helper.
3. Download the model files into a local folder.

pip install -e .
pip install huggingface_hub
huggingface-cli download ByteDance-Seed/Cola-DLM --local-dir hf_models

The Hugging Face record also lists this plain download step.

hf download ByteDance-Seed/Cola-DLM

How to use it: Load the two modules and the tokenizer, then call the task generation function with your prompts. The card's minimal example does this in one script.

import torch
from tokenizers import Tokenizer

from cola_dlm import (
    ColaDiTModel,
    ColaTextVAEModel,
    generate_task_repaint_inference,
)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

dit = ColaDiTModel.from_pretrained("hf_models/cola_dlm/cola_dit").to(device)
vae = ColaTextVAEModel.from_pretrained("hf_models/cola_dlm/cola_vae").to(device)
tokenizer = Tokenizer.from_file("hf_models/tokenizer.json")

prompts = [{"question": "Question: What is the capital of France? Answer:"}]
results = generate_task_repaint_inference(
    dit=dit,
    vae=vae,
    tokenizer=tokenizer,
    prompts=prompts,
    task_name="lambada",
    device=device,
    max_new_tokens=32,
    temperature=0.0,
    guidance_scale=7.0,
    timestep_num=16,
    pad_token_id=100277,
)

print(results[0]["generate"])

For a chat-style endpoint, the code release includes an openai_adapter service. Set these environment variables, then start it with uvicorn. It answers on this path and supports non-streaming completions only.

export COLA_DIT_PATH=hf_models/cola_dlm/cola_dit
export COLA_VAE_PATH=hf_models/cola_dlm/cola_vae
export COLA_TOKENIZER_PATH=hf_models/tokenizer.json
export COLA_MODEL_NAME=cola-dlm

uvicorn openai_adapter.server:app --host 0.0.0.0 --port 8000

Put your own value in COLA_API_KEY first.

POST /v1/chat/completions

Example: You put a short factual question in the card's question-and-answer format, such as one about the capital of France, into the prompts list. The script prints the text the model generates for it. The card shows no sample output, so check the printed result yourself.

For newcomers and experts: If you are new, treat this as a research demo. A first try is to download the files and run the minimal script above with one short question.

For experts, the design is a text VAE plus a block-causal DiT prior, trained in two stages. The VAE is pretrained first, then both parts train together with Flow Matching. The weights match the 2000 EFLOPs checkpoint from the paper's scaling curve, and the tokenizer is OLMo 2 with 100,278 entries.

Tokenizer turns the prompt into tokens
The VAE maps text to latents
The DiT transports latents with Flow Matching
The VAE decodes latents back to text

Good to know:

• Early scores: the card's zero-shot average is 26.75%, and MMLU is 19.30%.
• Adapter limit: it supports non-streaming completions only.
• Change the key: the example API key is change-me, so replace it before you share the service.
• License: Apache 2.0.
• Cost to run: none named.
• Data it reaches: it downloads files from Hugging Face, and the adapter listens on a local port.
• Not for: production assistants that need strong benchmark results.

Title: Zero-shot accuracy by task
LAMBADA: 50.80%
MMLU: 19.30%
OBQA: 23.00%
HellaSwag: 10.70%
RACE: 19.60%
SIQA: 28.90%
SQuAD: 30.90%
Story Cloze: 30.77%

On Hugging Face, 13 November 2026

• Task: text generation
• Library: transformers
• Formats: Safetensors
• Downloads in the last 30 days: 93
• Likes: 51
• License: apache-2.0
• Languages: en
• Tags: transformers, safetensors, text-generation, language-model, diffusion, latent-diffusion, flow-matching, text-vae, pytorch, research

Links

• Model: https://huggingface.co/ByteDance-Seed/Cola-DLM
• Files: https://huggingface.co/ByteDance-Seed/Cola-DLM/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download ByteDance-Seed/Cola-DLM

Then follow the usage in its model card. Nothing in this kit runs by itself.