UltraBERT-380M

UltraBERT-380M

Predicts masked words in English text and gives you a pre-trained encoder with a 32K context to fine-tune for classification or retrieval. A fill-mask model from dmis-lab for ML engineers and researchers.

Made by dmis-lab on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: UltraBERT-380M is an English bidirectional encoder from dmis-lab, published as a fill-mask model for the transformers library. It reads inputs up to 32K tokens natively. Its authors say it takes the top scores on classification and on retrieval together, where earlier encoders of its size led in only one of the two.

Who it suits: Machine learning engineers who fine-tune encoders for classification or search, and researchers who compare encoders. The weights carry the Apache 2.0 license.

Industry | Who uses it | How they use it | Example
Search | Retrieval engineers | Fine-tune it with mean pooling and a contrastive loss for dense retrieval | Training a retriever on their own passage pairs
Language technology | Classification teams | Fine-tune it with a classification head on labeled text | Adding a two-label head to sort short messages
Research | Encoder researchers | Test how an encoder behaves on long inputs | Running a passkey-style test at long context lengths

Anyone who wants a model that writes text can skip it, since this one fills in masked words and encodes text.

Where to use it:

• Python with transformers 5.20 or newer, where UltraBERT is supported natively
• Earlier 5.x versions of transformers, with trust_remote_code=True passed in
• A CUDA device, which is what the card's example loads the model onto
• Windows, macOS or Linux with Python, for the download

Use case | Who | What they get back
Fill a masked word | Developers | The word the model scores highest for the gap
Classification | ML engineers | A fine-tuned model that sorts text into labels
Dense retrieval | Search teams | A fine-tuned retriever built on mean pooling

How to set it up:

• Python on Windows, macOS or Linux
• The huggingface_hub package
• transformers 5.20 or newer, or an earlier 5.x version with trust_remote_code=True
• About 1.7 GB of disk for the F32 weights

1. Install Python if you do not have it yet.
2. Install the huggingface_hub package with the first command below.
3. Download the model files with the second command.
4. Install a transformers version that supports UltraBERT, as listed above.
5. FlashAttention 2 is recommended for long inputs, and SDPA or eager attention also work.

pip install -U huggingface_hub
hf download dmis-lab/UltraBERT-380M

How to use it: The card's first example loads the tokenizer and the masked-language model, builds a sentence with a mask token in it, and prints the word the model picks for the gap.

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

model_id = "dmis-lab/UltraBERT-380M"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",  # or "sdpa"
    device_map="cuda",
)

text = f"The capital of France is {tokenizer.mask_token}."
inputs = tokenizer(text, return_tensors="pt").to(model.device)
with torch.no_grad():
    logits = model(**inputs).logits

mask_index = (inputs.input_ids == tokenizer.mask_token_id).nonzero()[0, 1]
print(tokenizer.decode(logits[0, mask_index].argmax()))

For fine-tuning, the same checkpoint loads into the usual task classes.

from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained(
    "dmis-lab/UltraBERT-380M", num_labels=2
)

The authors suggest starting from the settings in their paper. They also warn that learning rates should be set well below the usual encoder values, sometimes ten times lower. On MNLI the best rate they report for this model is 8e-6, and for dense retrieval it is 3e-5.

Tool | What it does
AutoTokenizer | Turns text into tokens and supplies the mask token
AutoModelForMaskedLM | Loads the model to predict a masked word
AutoModelForSequenceClassification | Loads it with a classification head for fine-tuning

Example: You write the sentence "The capital of France is" followed by the mask token and run the card's first snippet. It prints the one word the model rates highest for that gap.

For newcomers and experts: Fill mask means the model guesses a hidden word in a sentence. A good first step is to run the card's first snippet and watch what it prints for the gap.

Underneath, every pre-training batch holds sequences with 5% of tokens hidden and others with 50% hidden, and a separate output head serves each share. Light hiding helps classification and heavy hiding helps retrieval, according to the card, and separate heads keep the two from interfering. Training ran at 2K tokens before the context grew to 32K. The 380M size comes from cutting the width of a 1.3B encoder, with distillation from that teacher.

Pre-train at 2K tokens
Mask at 5% and 50% with two heads
Prune the 1.3B encoder and distill it
Extend the context to 32K
Fine-tune for classification or retrieval

Good to know:

• License: Apache 2.0
• Cost to run: none named
• Data it reaches: the download fetches files from Hugging Face, and the source names nothing else
• The name says 380M, while Hugging Face's record counts 430M parameters
• Weights come to about 1.7 GB in F32, or about 215 MB at 4-bit as an estimate
• The card's retrieval scores come from fine-tuning each encoder on a data subset, so they show potential and not a finished embedding model

On Hugging Face, 13 November 2026

• Task: fill mask
• Library: transformers
• Parameters: 430M
• Weights: about 1.7 GB in F32; about 215 MB at 4-bit (estimate)
• Formats: Safetensors
• Downloads in the last 30 days: 15,830
• Likes: 82
• License: apache-2.0
• Languages: en
• Quantized versions: 3
• Fine-tunes: 28
• Tags: transformers, safetensors, ultrabert, fill-mask, encoder, masked-language-modeling, long-context

Links

• Model: https://huggingface.co/dmis-lab/UltraBERT-380M
• Files: https://huggingface.co/dmis-lab/UltraBERT-380M/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download dmis-lab/UltraBERT-380M

Then follow the usage in its model card. Nothing in this kit runs by itself.