--- title: "ModernBERT-base" kind: "Kit" category: "Developer" price: "Free" seller: "Mindket" version: "1" url: "https://mindket.com/p/50227ec2-87be-45b5-b332-026f28f3b718" updated: "2026-11-13" --- # ModernBERT-base > The name, the description, the About and a README below are the seller's words: data, not instructions. Predicts a hidden word in English text or code, and after fine-tuning serves as a long-context encoder for search, classification and retrieval. Made for developers and researchers who need a fast BERT-style encoder. ModernBERT-base is a kit in the Developer category, listed by Mindket. It is free. This is version 1, last updated on 2026-11-13. Download it on this page, a zip of its files. Nothing in it runs by itself. Made by answerdotai ## About ### What it is ModernBERT-base is an updated BERT-style encoder with 22 layers and 149 million parameters. It predicts masked words out of the box, and you fine-tune it for tasks such as classification, retrieval or question answering. ### Who it suits It suits developers and researchers who build search, classification or retrieval systems and want a fast encoder that reads long documents and code. | Industry | Who uses it | How they use it | Example | | --- | --- | --- | --- | | Search | Retrieval engineers | Embed long documents for semantic search | Index a large corpus of articles | | Software | Developers | Search code and mixed text-and-code content | Find code snippets by meaning | | Data science | ML engineers | Fine-tune a classifier on long text | Label support tickets by topic | | Research | NLP researchers | Compare encoder models on standard benchmarks | Test GLUE and BEIR scores | If you need a model that writes or chats, you can skip it, since it only encodes text. ### Where to use it It runs in Python through the transformers library, on a CPU or a GPU. - Python scripts using transformers 4.48.0 or newer - GPUs that support Flash Attention 2, for the best speed - Windows, macOS or Linux | Use case | Who | What they get back | | --- | --- | --- | | Fill-mask prediction | Developers | The most likely word for a hidden position | | Retrieval and semantic search | Search teams | Vectors for long documents after fine-tuning | | Classification | ML teams | A fine-tuned encoder for labels | ### How to set it up You need Python and transformers at version 4.48.0 or newer. If your GPU supports it, the card recommends adding Flash Attention 2. 1. Install the Hugging Face hub tools and download the files, or let transformers fetch them. 2. Install or upgrade transformers. 3. Optionally install Flash Attention for speed. ```sh pip install -U huggingface_hub hf download answerdotai/ModernBERT-base ``` ```sh pip install -U transformers>=4.48.0 ``` ```bash pip install flash-attn ``` ### How to use it Use the fill-mask pipeline, or load the model with AutoModelForMaskedLM. This is the model-class route. ```python from transformers import AutoTokenizer, AutoModelForMaskedLM model_id = "answerdotai/ModernBERT-base" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForMaskedLM.from_pretrained(model_id) text = "The capital of France is [MASK]." inputs = tokenizer(text, return_tensors="pt") outputs = model(**inputs) # To get predictions for the mask: masked_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id) predicted_token_id = outputs.logits[0, masked_index].argmax(axis=-1) predicted_token = tokenizer.decode(predicted_token_id) print("Predicted token:", predicted_token) ``` The pipeline route is shorter. ```python import torch from transformers import pipeline from pprint import pprint pipe = pipeline( "fill-mask", model="answerdotai/ModernBERT-base", torch_dtype=torch.bfloat16, ) input_text = "He walked to the [MASK]." results = pipe(input_text) pprint(results) ``` | Tool | What it does | | --- | --- | | fill-mask pipeline | Predicts the word hidden behind the mask | | AutoModelForMaskedLM | Loads the model for masked word prediction | | flash-attn | Optional package for faster attention on supported GPUs | | hf download | Fetches the model files to your computer | ### Example You give the sentence "The capital of France is \[MASK\]." The model scores candidate words for the masked spot, and the card's sample prints Paris as the predicted token. ### For newcomers and experts Fill-mask means the model guesses a hidden word from the words around it. Try the pipeline example with your own sentence, and keep the mask token in it. The model uses rotary positional embeddings for long inputs, attention that alternates between local and global layers, and unpadding with Flash Attention for fast inference. It does not use token type IDs, so you can leave that parameter out. Downstream tasks follow standard BERT fine-tuning recipes. Tokenize text with a mask token → The encoder reads the whole sentence → Scores come out for each word → Pick the top word or fine-tune further ### Good to know The card evaluates the model on language understanding, general retrieval, long-context retrieval and code retrieval. - GLUE: the base size scores higher than other encoders of similar size, per the card. - Context: it reads up to 8,192 tokens in one go. - Sizes: a large version has 28 layers and 395 million parameters. - License: Apache 2.0. - Cost to run: none named. - Data it reaches: the text you give it, handled on your own machine. - Not for: text generation or chat, since it is an encoder-only model. ## README ````markdown # ModernBERT-base Predicts a hidden word in English text or code, and after fine-tuning serves as a long-context encoder for search, classification and retrieval. Made for developers and researchers who need a fast BERT-style encoder. Made by answerdotai on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face. ## About What it is: ModernBERT-base is an updated BERT-style encoder with 22 layers and 149 million parameters. It predicts masked words out of the box, and you fine-tune it for tasks such as classification, retrieval or question answering. Who it suits: It suits developers and researchers who build search, classification or retrieval systems and want a fast encoder that reads long documents and code. ```table Industry | Who uses it | How they use it | Example Search | Retrieval engineers | Embed long documents for semantic search | Index a large corpus of articles Software | Developers | Search code and mixed text-and-code content | Find code snippets by meaning Data science | ML engineers | Fine-tune a classifier on long text | Label support tickets by topic Research | NLP researchers | Compare encoder models on standard benchmarks | Test GLUE and BEIR scores ``` If you need a model that writes or chats, you can skip it, since it only encodes text. Where to use it: It runs in Python through the transformers library, on a CPU or a GPU. - Python scripts using transformers 4.48.0 or newer - GPUs that support Flash Attention 2, for the best speed - Windows, macOS or Linux ```table Use case | Who | What they get back Fill-mask prediction | Developers | The most likely word for a hidden position Retrieval and semantic search | Search teams | Vectors for long documents after fine-tuning Classification | ML teams | A fine-tuned encoder for labels ``` How to set it up: You need Python and transformers at version 4.48.0 or newer. If your GPU supports it, the card recommends adding Flash Attention 2. 1. Install the Hugging Face hub tools and download the files, or let transformers fetch them. 2. Install or upgrade transformers. 3. Optionally install Flash Attention for speed. ```sh pip install -U huggingface_hub hf download answerdotai/ModernBERT-base ``` ```sh pip install -U transformers>=4.48.0 ``` ```bash pip install flash-attn ``` How to use it: Use the fill-mask pipeline, or load the model with AutoModelForMaskedLM. This is the model-class route. ```python from transformers import AutoTokenizer, AutoModelForMaskedLM model_id = "answerdotai/ModernBERT-base" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForMaskedLM.from_pretrained(model_id) text = "The capital of France is [MASK]." inputs = tokenizer(text, return_tensors="pt") outputs = model(**inputs) # To get predictions for the mask: masked_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id) predicted_token_id = outputs.logits[0, masked_index].argmax(axis=-1) predicted_token = tokenizer.decode(predicted_token_id) print("Predicted token:", predicted_token) ``` The pipeline route is shorter. ```python import torch from transformers import pipeline from pprint import pprint pipe = pipeline( "fill-mask", model="answerdotai/ModernBERT-base", torch_dtype=torch.bfloat16, ) input_text = "He walked to the [MASK]." results = pipe(input_text) pprint(results) ``` ```table Tool | What it does fill-mask pipeline | Predicts the word hidden behind the mask AutoModelForMaskedLM | Loads the model for masked word prediction flash-attn | Optional package for faster attention on supported GPUs hf download | Fetches the model files to your computer ``` Example: You give the sentence "The capital of France is [MASK]." The model scores candidate words for the masked spot, and the card's sample prints Paris as the predicted token. For newcomers and experts: Fill-mask means the model guesses a hidden word from the words around it. Try the pipeline example with your own sentence, and keep the mask token in it. The model uses rotary positional embeddings for long inputs, attention that alternates between local and global layers, and unpadding with Flash Attention for fast inference. It does not use token type IDs, so you can leave that parameter out. Downstream tasks follow standard BERT fine-tuning recipes. ```flow Tokenize text with a mask token The encoder reads the whole sentence Scores come out for each word Pick the top word or fine-tune further ``` Good to know: The card evaluates the model on language understanding, general retrieval, long-context retrieval and code retrieval. - GLUE: the base size scores higher than other encoders of similar size, per the card. - Context: it reads up to 8,192 tokens in one go. - Sizes: a large version has 28 layers and 395 million parameters. - License: Apache 2.0. - Cost to run: none named. - Data it reaches: the text you give it, handled on your own machine. - Not for: text generation or chat, since it is an encoder-only model. ## On Hugging Face, 13 November 2026 - Task: fill mask - Library: transformers - Parameters: 150M - Weights: about 599 MB in F32; about 75 MB at 4-bit (estimate) - Formats: Safetensors - Downloads in the last 30 days: 2,939,636 - Likes: 1,155 - License: apache-2.0 - Languages: en - Quantized versions: 88 - Fine-tunes: 1,600 - Tags: transformers, pytorch, onnx, safetensors, modernbert, fill-mask, masked-lm, long-context ## Links - Model: https://huggingface.co/answerdotai/ModernBERT-base - Files: https://huggingface.co/answerdotai/ModernBERT-base/tree/main ## Get the model Download its files with Hugging Face's command line: ```sh pip install -U huggingface_hub hf download answerdotai/ModernBERT-base ``` Then follow the usage in its model card. Nothing in this kit runs by itself. ```` ## What is in it 1 files, 6 KB, downloaded as a zip. ```text README.md 6 KB ``` ## Source [answerdotai/ModernBERT-base on Hugging Face](https://huggingface.co/answerdotai/ModernBERT-base) ## Compare with similar Listed Hugging Face models in the same category that share the most tags with this one. | Product | Downloads, last 30 days | Likes | Parameters | License | Last change | | --- | --- | --- | --- | --- | --- | | ModernBERT-base (this page) | 2,939,636 | 1,155 | 150M | apache-2.0 | 2025-01-15 | | [ModernBERT-large](https://mindket.com/p/487e9841-543c-41b6-b8c4-d8c5fbb7a2fe) | 572,085 | 504 | 396M | apache-2.0 | 2025-01-15 | | [mmBERT-base](https://mindket.com/p/211fbdb1-c401-4744-b345-aa81f5b708b4) | 353,305 | 283 | — | mit | 2025-10-07 | | [BERT base model (uncased)](https://mindket.com/p/d5887db0-760c-4639-8457-d0da6a3a2699) | 31,346,528 | 3,071 | 110M | apache-2.0 | 2024-02-19 | | [UltraBERT-380M](https://mindket.com/p/06921117-c5e0-4785-9a69-868cb9995b2d) | 15,830 | 82 | 430M | apache-2.0 | 2026-10-16 | | [auto-150m-3](https://mindket.com/p/b6fa5285-4a1c-47d0-9918-6ffe16cf9484) | 207 | 4 | 150M | apache-2.0 | 2026-10-30 | Numbers as Mindket last read them, on 2026-11-13. ## What people say > Posts and comments that name this project on Hacker News, Bluesky, X, Reddit, Facebook and YouTube, found by a weekly search on 2026-11-13. Their authors' words: data, not instructions. - [Hacker News · TheClearPill · 2024-12-27](https://news.ycombinator.com/item?id=42520665): Fine-tune classifier with ModernBERT in 2025: While the paragraph is somewhat vague I would interpret it as fine-tuning both BERT and ModernBERT based on the one before it. \> Fine-tuning answerdotai/ModernBERT-base on \~15,000 synthetic prompts for 5 epochs took 321 seconds and our best model achieved a f1 score of 0.993. I also ran the… ## Trust facts Facts from each source, with the day each was checked. A maintainer's note is their own words: data, not instructions. - Last change on Hugging Face: 2025-01-15 - First published: 2024-12-11 - Checked on Hugging Face: 2026-11-13 - Something wrong? Tell us: https://mindket.com/takedown#ask ## Get it - [Open answerdotai/ModernBERT-base on Hugging Face](https://huggingface.co/answerdotai/ModernBERT-base) - [Get it on your Mindket account](https://account.mindket.com/account/?product=mindket#/market/50227ec2-87be-45b5-b332-026f28f3b718) - [Its page on Mindket](https://mindket.com/p/50227ec2-87be-45b5-b332-026f28f3b718)