ModernBERT-base

ModernBERT-base

Predicts a hidden word in English text or code, and after fine-tuning serves as a long-context encoder for search, classification and retrieval. Made for developers and researchers who need a fast BERT-style encoder.

Made by answerdotai on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: ModernBERT-base is an updated BERT-style encoder with 22 layers and 149 million parameters. It predicts masked words out of the box, and you fine-tune it for tasks such as classification, retrieval or question answering.

Who it suits: It suits developers and researchers who build search, classification or retrieval systems and want a fast encoder that reads long documents and code.

Industry | Who uses it | How they use it | Example
Search | Retrieval engineers | Embed long documents for semantic search | Index a large corpus of articles
Software | Developers | Search code and mixed text-and-code content | Find code snippets by meaning
Data science | ML engineers | Fine-tune a classifier on long text | Label support tickets by topic
Research | NLP researchers | Compare encoder models on standard benchmarks | Test GLUE and BEIR scores

If you need a model that writes or chats, you can skip it, since it only encodes text.

Where to use it: It runs in Python through the transformers library, on a CPU or a GPU.

• Python scripts using transformers 4.48.0 or newer
• GPUs that support Flash Attention 2, for the best speed
• Windows, macOS or Linux

Use case | Who | What they get back
Fill-mask prediction | Developers | The most likely word for a hidden position
Retrieval and semantic search | Search teams | Vectors for long documents after fine-tuning
Classification | ML teams | A fine-tuned encoder for labels

How to set it up: You need Python and transformers at version 4.48.0 or newer. If your GPU supports it, the card recommends adding Flash Attention 2.

1. Install the Hugging Face hub tools and download the files, or let transformers fetch them.
2. Install or upgrade transformers.
3. Optionally install Flash Attention for speed.

pip install -U huggingface_hub
hf download answerdotai/ModernBERT-base

pip install -U transformers>=4.48.0

pip install flash-attn

How to use it: Use the fill-mask pipeline, or load the model with AutoModelForMaskedLM. This is the model-class route.

from transformers import AutoTokenizer, AutoModelForMaskedLM

model_id = "answerdotai/ModernBERT-base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)

text = "The capital of France is [MASK]."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)

# To get predictions for the mask:
masked_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
predicted_token_id = outputs.logits[0, masked_index].argmax(axis=-1)
predicted_token = tokenizer.decode(predicted_token_id)
print("Predicted token:", predicted_token)

The pipeline route is shorter.

import torch
from transformers import pipeline
from pprint import pprint

pipe = pipeline(
    "fill-mask",
    model="answerdotai/ModernBERT-base",
    torch_dtype=torch.bfloat16,
)

input_text = "He walked to the [MASK]."
results = pipe(input_text)
pprint(results)

Tool | What it does
fill-mask pipeline | Predicts the word hidden behind the mask
AutoModelForMaskedLM | Loads the model for masked word prediction
flash-attn | Optional package for faster attention on supported GPUs
hf download | Fetches the model files to your computer

Example: You give the sentence "The capital of France is [MASK]." The model scores candidate words for the masked spot, and the card's sample prints Paris as the predicted token.

For newcomers and experts: Fill-mask means the model guesses a hidden word from the words around it. Try the pipeline example with your own sentence, and keep the mask token in it.

The model uses rotary positional embeddings for long inputs, attention that alternates between local and global layers, and unpadding with Flash Attention for fast inference. It does not use token type IDs, so you can leave that parameter out. Downstream tasks follow standard BERT fine-tuning recipes.

Tokenize text with a mask token
The encoder reads the whole sentence
Scores come out for each word
Pick the top word or fine-tune further

Good to know: The card evaluates the model on language understanding, general retrieval, long-context retrieval and code retrieval.

• GLUE: the base size scores higher than other encoders of similar size, per the card.
• Context: it reads up to 8,192 tokens in one go.
• Sizes: a large version has 28 layers and 395 million parameters.
• License: Apache 2.0.
• Cost to run: none named.
• Data it reaches: the text you give it, handled on your own machine.
• Not for: text generation or chat, since it is an encoder-only model.

On Hugging Face, 13 November 2026

• Task: fill mask
• Library: transformers
• Parameters: 150M
• Weights: about 599 MB in F32; about 75 MB at 4-bit (estimate)
• Formats: Safetensors
• Downloads in the last 30 days: 2,939,636
• Likes: 1,155
• License: apache-2.0
• Languages: en
• Quantized versions: 88
• Fine-tunes: 1,600
• Tags: transformers, pytorch, onnx, safetensors, modernbert, fill-mask, masked-lm, long-context

Links

• Model: https://huggingface.co/answerdotai/ModernBERT-base
• Files: https://huggingface.co/answerdotai/ModernBERT-base/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download answerdotai/ModernBERT-base

Then follow the usage in its model card. Nothing in this kit runs by itself.