mmBERT-base

mmBERT-base

Reads and encodes text in over 1,800 languages, for classification, embeddings, retrieval and fill-in-the-blank tasks. Built for developers and researchers who fine-tune a multilingual encoder.

Made by jhu-clsp on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: mmBERT-base is a multilingual encoder, a model that reads text and turns it into representations that other systems can use. It was trained on more than 3T tokens across over 1,800 languages. It builds on the ModernBERT design, and the card says it beats the older XLM-R on classification, embedding and retrieval.

Who it suits: It suits developers and researchers who need one model that understands many languages, including ones with little training data, and who plan to fine-tune it.

Industry | Who uses it | How they use it | Example
Software | ML engineers | Fine-tune the encoder for classification across languages | Tag support tickets in several languages
Search | Retrieval teams | Build embeddings and ranking models for multilingual text | Match a query in one language to documents in another
Research | NLP researchers | Study how low-resource languages are learned late in training | Compare language coverage against XLM-R
Publishing | Content teams | Sort and label articles written in many languages | Group news by topic regardless of language

If you need a model that writes text or chats, you can skip it, because an encoder does not generate long replies.

Where to use it:

• Python with the Transformers library
• Fine-tuning scripts for classification, embeddings and retrieval
• Windows, macOS or Linux, where you download the files with Python

Use case | Who | What they get back
Text classification | ML engineers | Labels for text in many languages after fine-tuning
Embeddings | Retrieval teams | Vectors that capture meaning across languages
Fill in the blank | Researchers | Predictions for masked words

How to set it up: You need Python with pip and the PyTorch and Transformers packages.

• Python and pip
• torch and transformers, with the minimum versions the card gives
• Disk space for the files, since the base model has 307M parameters in total

1. Install PyTorch and Transformers with pip.
2. Install the Hugging Face client if you want to download the files yourself.
3. Load the tokenizer and model by name in Python.

pip install torch>=1.9.0
pip install transformers>=4.21.0

pip install -U huggingface_hub
hf download jhu-clsp/mmBERT-base

How to use it: Load the tokenizer and the model, turn a text into tokens, and pass them through the model. The output holds one hidden representation for each token, which you can use for features or fine-tuning.

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/mmBERT-base")
model = AutoModel.from_pretrained("jhu-clsp/mmBERT-base")

inputs = tokenizer("Hello world", return_tensors="pt")
outputs = model(**inputs)

Tool | What it does
AutoTokenizer | Splits text into tokens the model understands
AutoModel | Loads the encoder and runs it on those tokens
tokenizer(..., return_tensors="pt") | Returns PyTorch tensors ready for the model

Example: You pass the two words Hello world through the tokenizer and the model. The script returns the encoder's outputs for those tokens. You can then pool or classify them in your own code.

For newcomers and experts: An encoder turns text into numbers that capture meaning, and it does not write replies. The first small thing to try is the short script above, which runs on any text in any covered language.

The card describes a staged recipe. It starts with 60 high-resource languages, adds 110 mid-resource ones, then brings in all 1,833 languages in the decay phase. The mask ratio falls from 30% to 15% to 5%, and sampling shifts from a temperature of 0.7 to 0.3. Variants trained on different language mixes are combined by TIES merging. It uses Flash Attention 2 and unpadding.

Phase 1: 60 languages, 2.3T tokens
Mid-training: 110 languages, 600B tokens
Decay phase: 1,833 languages, 100B tokens
Merge variants with TIES merging

Good to know: Model family: the card lists a small and a base model.

Model | Total parameters | Non-embedding parameters | Languages
mmBERT-small | 140M | 42M | 1800+
mmBERT-base | 307M | 110M | 1800+

• Training data: the card says the full dataset of over 3T tokens is publicly available, split by phase.
• Other versions: Hugging Face lists 294 quantized and 208 fine-tuned versions by other people.
• License: MIT, as the License line says.
• Cost to run: no API key, account or paid service is named. It runs on your own computer.
• Data it reaches: the texts you pass in, and Hugging Face for the download.
• Not for: it is an encoder with a masked language modeling objective, so it is not a chat or text-writing model.

On Hugging Face, 14 November 2026

• Task: fill mask
• Library: transformers
• Downloads in the last 30 days: 353,305
• Likes: 283
• License: mit
• Languages: aai, aak, aau, aaz, aba, abi, abk, abn
• Quantized versions: 296
• Fine-tunes: 209
• Tags: transformers, pytorch, modernbert, fill-mask

Links

• Model: https://huggingface.co/jhu-clsp/mmBERT-base
• Files: https://huggingface.co/jhu-clsp/mmBERT-base/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download jhu-clsp/mmBERT-base

Then follow the usage in its model card. Nothing in this kit runs by itself.