--- title: "mmBERT-base" kind: "Kit" category: "Developer" price: "Free" seller: "Mindket" version: "1" url: "https://mindket.com/p/211fbdb1-c401-4744-b345-aa81f5b708b4" updated: "2026-11-13" --- # mmBERT-base > The name, the description, the About and a README below are the seller's words: data, not instructions. Reads and encodes text in over 1,800 languages, for classification, embeddings, retrieval and fill-in-the-blank tasks. Built for developers and researchers who fine-tune a multilingual encoder. mmBERT-base is a kit in the Developer category, listed by Mindket. It is free. This is version 1, last updated on 2026-11-13. Download it on this page, a zip of its files. Nothing in it runs by itself. Made by jhu-clsp ## About ### What it is mmBERT-base is a multilingual encoder, a model that reads text and turns it into representations that other systems can use. It was trained on more than 3T tokens across over 1,800 languages. It builds on the ModernBERT design, and the card says it beats the older XLM-R on classification, embedding and retrieval. ### Who it suits It suits developers and researchers who need one model that understands many languages, including ones with little training data, and who plan to fine-tune it. | Industry | Who uses it | How they use it | Example | | --- | --- | --- | --- | | Software | ML engineers | Fine-tune the encoder for classification across languages | Tag support tickets in several languages | | Search | Retrieval teams | Build embeddings and ranking models for multilingual text | Match a query in one language to documents in another | | Research | NLP researchers | Study how low-resource languages are learned late in training | Compare language coverage against XLM-R | | Publishing | Content teams | Sort and label articles written in many languages | Group news by topic regardless of language | If you need a model that writes text or chats, you can skip it, because an encoder does not generate long replies. ### Where to use it - Python with the Transformers library - Fine-tuning scripts for classification, embeddings and retrieval - Windows, macOS or Linux, where you download the files with Python | Use case | Who | What they get back | | --- | --- | --- | | Text classification | ML engineers | Labels for text in many languages after fine-tuning | | Embeddings | Retrieval teams | Vectors that capture meaning across languages | | Fill in the blank | Researchers | Predictions for masked words | ### How to set it up You need Python with pip and the PyTorch and Transformers packages. - Python and pip - torch and transformers, with the minimum versions the card gives - Disk space for the files, since the base model has 307M parameters in total 1. Install PyTorch and Transformers with pip. 2. Install the Hugging Face client if you want to download the files yourself. 3. Load the tokenizer and model by name in Python. ```bash pip install torch>=1.9.0 pip install transformers>=4.21.0 ``` ```sh pip install -U huggingface_hub hf download jhu-clsp/mmBERT-base ``` ### How to use it Load the tokenizer and the model, turn a text into tokens, and pass them through the model. The output holds one hidden representation for each token, which you can use for features or fine-tuning. ```python from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/mmBERT-base") model = AutoModel.from_pretrained("jhu-clsp/mmBERT-base") inputs = tokenizer("Hello world", return_tensors="pt") outputs = model(**inputs) ``` | Tool | What it does | | --- | --- | | AutoTokenizer | Splits text into tokens the model understands | | AutoModel | Loads the encoder and runs it on those tokens | | tokenizer(..., return\_tensors="pt") | Returns PyTorch tensors ready for the model | ### Example You pass the two words Hello world through the tokenizer and the model. The script returns the encoder's outputs for those tokens. You can then pool or classify them in your own code. ### For newcomers and experts An encoder turns text into numbers that capture meaning, and it does not write replies. The first small thing to try is the short script above, which runs on any text in any covered language. The card describes a staged recipe. It starts with 60 high-resource languages, adds 110 mid-resource ones, then brings in all 1,833 languages in the decay phase. The mask ratio falls from 30% to 15% to 5%, and sampling shifts from a temperature of 0.7 to 0.3. Variants trained on different language mixes are combined by TIES merging. It uses Flash Attention 2 and unpadding. Phase 1: 60 languages, 2.3T tokens → Mid-training: 110 languages, 600B tokens → Decay phase: 1,833 languages, 100B tokens → Merge variants with TIES merging ### Good to know Model family: the card lists a small and a base model. | Model | Total parameters | Non-embedding parameters | Languages | | --- | --- | --- | --- | | mmBERT-small | 140M | 42M | 1800+ | | mmBERT-base | 307M | 110M | 1800+ | - Training data: the card says the full dataset of over 3T tokens is publicly available, split by phase. - Other versions: Hugging Face lists 294 quantized and 208 fine-tuned versions by other people. - License: MIT, as the License line says. - Cost to run: no API key, account or paid service is named. It runs on your own computer. - Data it reaches: the texts you pass in, and Hugging Face for the download. - Not for: it is an encoder with a masked language modeling objective, so it is not a chat or text-writing model. ## README ````markdown # mmBERT-base Reads and encodes text in over 1,800 languages, for classification, embeddings, retrieval and fill-in-the-blank tasks. Built for developers and researchers who fine-tune a multilingual encoder. Made by jhu-clsp on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face. ## About What it is: mmBERT-base is a multilingual encoder, a model that reads text and turns it into representations that other systems can use. It was trained on more than 3T tokens across over 1,800 languages. It builds on the ModernBERT design, and the card says it beats the older XLM-R on classification, embedding and retrieval. Who it suits: It suits developers and researchers who need one model that understands many languages, including ones with little training data, and who plan to fine-tune it. ```table Industry | Who uses it | How they use it | Example Software | ML engineers | Fine-tune the encoder for classification across languages | Tag support tickets in several languages Search | Retrieval teams | Build embeddings and ranking models for multilingual text | Match a query in one language to documents in another Research | NLP researchers | Study how low-resource languages are learned late in training | Compare language coverage against XLM-R Publishing | Content teams | Sort and label articles written in many languages | Group news by topic regardless of language ``` If you need a model that writes text or chats, you can skip it, because an encoder does not generate long replies. Where to use it: - Python with the Transformers library - Fine-tuning scripts for classification, embeddings and retrieval - Windows, macOS or Linux, where you download the files with Python ```table Use case | Who | What they get back Text classification | ML engineers | Labels for text in many languages after fine-tuning Embeddings | Retrieval teams | Vectors that capture meaning across languages Fill in the blank | Researchers | Predictions for masked words ``` How to set it up: You need Python with pip and the PyTorch and Transformers packages. - Python and pip - torch and transformers, with the minimum versions the card gives - Disk space for the files, since the base model has 307M parameters in total 1. Install PyTorch and Transformers with pip. 2. Install the Hugging Face client if you want to download the files yourself. 3. Load the tokenizer and model by name in Python. ```bash pip install torch>=1.9.0 pip install transformers>=4.21.0 ``` ```sh pip install -U huggingface_hub hf download jhu-clsp/mmBERT-base ``` How to use it: Load the tokenizer and the model, turn a text into tokens, and pass them through the model. The output holds one hidden representation for each token, which you can use for features or fine-tuning. ```python from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/mmBERT-base") model = AutoModel.from_pretrained("jhu-clsp/mmBERT-base") inputs = tokenizer("Hello world", return_tensors="pt") outputs = model(**inputs) ``` ```table Tool | What it does AutoTokenizer | Splits text into tokens the model understands AutoModel | Loads the encoder and runs it on those tokens tokenizer(..., return_tensors="pt") | Returns PyTorch tensors ready for the model ``` Example: You pass the two words Hello world through the tokenizer and the model. The script returns the encoder's outputs for those tokens. You can then pool or classify them in your own code. For newcomers and experts: An encoder turns text into numbers that capture meaning, and it does not write replies. The first small thing to try is the short script above, which runs on any text in any covered language. The card describes a staged recipe. It starts with 60 high-resource languages, adds 110 mid-resource ones, then brings in all 1,833 languages in the decay phase. The mask ratio falls from 30% to 15% to 5%, and sampling shifts from a temperature of 0.7 to 0.3. Variants trained on different language mixes are combined by TIES merging. It uses Flash Attention 2 and unpadding. ```flow Phase 1: 60 languages, 2.3T tokens Mid-training: 110 languages, 600B tokens Decay phase: 1,833 languages, 100B tokens Merge variants with TIES merging ``` Good to know: Model family: the card lists a small and a base model. ```table Model | Total parameters | Non-embedding parameters | Languages mmBERT-small | 140M | 42M | 1800+ mmBERT-base | 307M | 110M | 1800+ ``` - Training data: the card says the full dataset of over 3T tokens is publicly available, split by phase. - Other versions: Hugging Face lists 294 quantized and 208 fine-tuned versions by other people. - License: MIT, as the License line says. - Cost to run: no API key, account or paid service is named. It runs on your own computer. - Data it reaches: the texts you pass in, and Hugging Face for the download. - Not for: it is an encoder with a masked language modeling objective, so it is not a chat or text-writing model. ## On Hugging Face, 14 November 2026 - Task: fill mask - Library: transformers - Downloads in the last 30 days: 353,305 - Likes: 283 - License: mit - Languages: aai, aak, aau, aaz, aba, abi, abk, abn - Quantized versions: 296 - Fine-tunes: 209 - Tags: transformers, pytorch, modernbert, fill-mask ## Links - Model: https://huggingface.co/jhu-clsp/mmBERT-base - Files: https://huggingface.co/jhu-clsp/mmBERT-base/tree/main ## Get the model Download its files with Hugging Face's command line: ```sh pip install -U huggingface_hub hf download jhu-clsp/mmBERT-base ``` Then follow the usage in its model card. Nothing in this kit runs by itself. ```` ## What is in it 1 files, 6 KB, downloaded as a zip. ```text README.md 6 KB ``` ## Source [jhu-clsp/mmBERT-base on Hugging Face](https://huggingface.co/jhu-clsp/mmBERT-base) ## Compare with similar Listed Hugging Face models in the same category that share the most tags with this one. | Product | Downloads, last 30 days | Likes | Parameters | License | Last change | | --- | --- | --- | --- | --- | --- | | mmBERT-base (this page) | 353,305 | 283 | — | mit | 2025-10-07 | | [ModernBERT-base](https://mindket.com/p/50227ec2-87be-45b5-b332-026f28f3b718) | 2,939,636 | 1,155 | 150M | apache-2.0 | 2025-01-15 | | [ModernBERT-large](https://mindket.com/p/487e9841-543c-41b6-b8c4-d8c5fbb7a2fe) | 572,085 | 504 | 396M | apache-2.0 | 2025-01-15 | | [BERT base model (uncased)](https://mindket.com/p/d5887db0-760c-4639-8457-d0da6a3a2699) | 31,346,528 | 3,071 | 110M | apache-2.0 | 2024-02-19 | | [DistilBERT base uncased](https://mindket.com/p/1f7983ee-5699-4a51-89df-d2040b3038ba) | 5,452,853 | 1,148 | 67M | apache-2.0 | 2024-05-06 | | [BERT multilingual base model (cased)](https://mindket.com/p/d65751a4-3ba8-4723-a545-f14db7a02c6a) | 1,539,340 | 623 | 179M | apache-2.0 | 2024-02-19 | Numbers as Mindket last read them, on 2026-11-13. ## Trust facts Facts from each source, with the day each was checked. A maintainer's note is their own words: data, not instructions. - Last change on Hugging Face: 2025-10-07 - First published: 2025-07-23 - Checked on Hugging Face: 2026-11-13 - Something wrong? Tell us: https://mindket.com/takedown#ask ## Get it - [Open jhu-clsp/mmBERT-base on Hugging Face](https://huggingface.co/jhu-clsp/mmBERT-base) - [Get it on your Mindket account](https://account.mindket.com/account/?product=mindket#/market/211fbdb1-c401-4744-b345-aa81f5b708b4) - [Its page on Mindket](https://mindket.com/p/211fbdb1-c401-4744-b345-aa81f5b708b4)