--- title: "ModernBERT-large" kind: "Kit" category: "Developer" price: "Free" seller: "Mindket" version: "1" url: "https://mindket.com/p/487e9841-543c-41b6-b8c4-d8c5fbb7a2fe" updated: "2026-11-13" --- # ModernBERT-large > The name, the description, the About and a README below are the seller's words: data, not instructions. Predicts a hidden word in English text or code, and after fine-tuning serves as a larger long-context encoder for search, classification and retrieval. Made for developers and researchers who want the bigger ModernBERT. ModernBERT-large is a kit in the Developer category, listed by Mindket. It is free. This is version 1, last updated on 2026-11-13. Download it on this page, a zip of its files. Nothing in it runs by itself. Made by answerdotai ## About ### What it is ModernBERT-large is the bigger size of an updated BERT-style encoder, with 28 layers and 395 million parameters. It predicts masked words out of the box, and you fine-tune it for classification, retrieval or question answering. ### Who it suits It suits developers and researchers who build search, classification or retrieval systems and want more encoder capacity for long documents and code. | Industry | Who uses it | How they use it | Example | | --- | --- | --- | --- | | Search | Retrieval engineers | Embed long documents for semantic search | Index a large corpus of articles | | Software | Developers | Search code and mixed text-and-code content | Find code snippets by meaning | | Data science | ML engineers | Fine-tune a classifier on long text | Label support tickets by topic | | Research | NLP researchers | Compare encoder models on standard benchmarks | Test GLUE and BEIR scores | If you need a model that writes or chats, or you need a smaller footprint, you can skip it. ### Where to use it It runs in Python through the transformers library, on a CPU or a GPU. - Python scripts using transformers 4.48.0 or newer - GPUs that support Flash Attention 2, for the best speed - Windows, macOS or Linux | Use case | Who | What they get back | | --- | --- | --- | | Fill-mask prediction | Developers | The most likely word for a hidden position | | Retrieval and semantic search | Search teams | Vectors for long documents after fine-tuning | | Classification | ML teams | A fine-tuned encoder for labels | ### How to set it up You need Python and transformers at version 4.48.0 or newer. If your GPU supports it, the card recommends adding Flash Attention 2. The weights are about 1.6 GB in F32. 1. Install the Hugging Face hub tools and download the files, or let transformers fetch them. 2. Install or upgrade transformers. 3. Optionally install Flash Attention for speed. ```sh pip install -U huggingface_hub hf download answerdotai/ModernBERT-large ``` ```sh pip install -U transformers>=4.48.0 ``` ```bash pip install flash-attn ``` ### How to use it Use the fill-mask pipeline, or load the model with AutoModelForMaskedLM. The card's examples name the base checkpoint, so swap in answerdotai/ModernBERT-large to use this size. This is the model-class route. ```python from transformers import AutoTokenizer, AutoModelForMaskedLM model_id = "answerdotai/ModernBERT-base" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForMaskedLM.from_pretrained(model_id) text = "The capital of France is [MASK]." inputs = tokenizer(text, return_tensors="pt") outputs = model(**inputs) # To get predictions for the mask: masked_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id) predicted_token_id = outputs.logits[0, masked_index].argmax(axis=-1) predicted_token = tokenizer.decode(predicted_token_id) print("Predicted token:", predicted_token) ``` The pipeline route is shorter. ```python import torch from transformers import pipeline from pprint import pprint pipe = pipeline( "fill-mask", model="answerdotai/ModernBERT-base", torch_dtype=torch.bfloat16, ) input_text = "He walked to the [MASK]." results = pipe(input_text) pprint(results) ``` | Tool | What it does | | --- | --- | | fill-mask pipeline | Predicts the word hidden behind the mask | | AutoModelForMaskedLM | Loads the model for masked word prediction | | flash-attn | Optional package for faster attention on supported GPUs | | hf download | Fetches the model files to your computer | ### Example You give the sentence "The capital of France is \[MASK\]." The model scores candidate words for the masked spot, and the card's sample prints Paris as the predicted token. ### For newcomers and experts Fill-mask means the model guesses a hidden word from the words around it. Try the pipeline example with your own sentence, and keep the mask token in it. The model uses rotary positional embeddings for long inputs, attention that alternates between local and global layers, and unpadding with Flash Attention for fast inference. It does not use token type IDs, so you can leave that parameter out. Downstream tasks follow standard BERT fine-tuning recipes. Tokenize text with a mask token → The encoder reads the whole sentence → Scores come out for each word → Pick the top word or fine-tune further ### Good to know The card evaluates the family on language understanding, general retrieval, long-context retrieval and code retrieval. - GLUE: the card says the large size is second only to Deberta-v3-large. - Context: it reads up to 8,192 tokens in one go. - Sizes: a base version has 22 layers and 149 million parameters. - License: Apache 2.0. - Cost to run: none named. - Data it reaches: the text you give it, handled on your own machine. - Not for: text generation or chat, since it is an encoder-only model. ## README ````markdown # ModernBERT-large Predicts a hidden word in English text or code, and after fine-tuning serves as a larger long-context encoder for search, classification and retrieval. Made for developers and researchers who want the bigger ModernBERT. Made by answerdotai on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face. ## About What it is: ModernBERT-large is the bigger size of an updated BERT-style encoder, with 28 layers and 395 million parameters. It predicts masked words out of the box, and you fine-tune it for classification, retrieval or question answering. Who it suits: It suits developers and researchers who build search, classification or retrieval systems and want more encoder capacity for long documents and code. ```table Industry | Who uses it | How they use it | Example Search | Retrieval engineers | Embed long documents for semantic search | Index a large corpus of articles Software | Developers | Search code and mixed text-and-code content | Find code snippets by meaning Data science | ML engineers | Fine-tune a classifier on long text | Label support tickets by topic Research | NLP researchers | Compare encoder models on standard benchmarks | Test GLUE and BEIR scores ``` If you need a model that writes or chats, or you need a smaller footprint, you can skip it. Where to use it: It runs in Python through the transformers library, on a CPU or a GPU. - Python scripts using transformers 4.48.0 or newer - GPUs that support Flash Attention 2, for the best speed - Windows, macOS or Linux ```table Use case | Who | What they get back Fill-mask prediction | Developers | The most likely word for a hidden position Retrieval and semantic search | Search teams | Vectors for long documents after fine-tuning Classification | ML teams | A fine-tuned encoder for labels ``` How to set it up: You need Python and transformers at version 4.48.0 or newer. If your GPU supports it, the card recommends adding Flash Attention 2. The weights are about 1.6 GB in F32. 1. Install the Hugging Face hub tools and download the files, or let transformers fetch them. 2. Install or upgrade transformers. 3. Optionally install Flash Attention for speed. ```sh pip install -U huggingface_hub hf download answerdotai/ModernBERT-large ``` ```sh pip install -U transformers>=4.48.0 ``` ```bash pip install flash-attn ``` How to use it: Use the fill-mask pipeline, or load the model with AutoModelForMaskedLM. The card's examples name the base checkpoint, so swap in answerdotai/ModernBERT-large to use this size. This is the model-class route. ```python from transformers import AutoTokenizer, AutoModelForMaskedLM model_id = "answerdotai/ModernBERT-base" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForMaskedLM.from_pretrained(model_id) text = "The capital of France is [MASK]." inputs = tokenizer(text, return_tensors="pt") outputs = model(**inputs) # To get predictions for the mask: masked_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id) predicted_token_id = outputs.logits[0, masked_index].argmax(axis=-1) predicted_token = tokenizer.decode(predicted_token_id) print("Predicted token:", predicted_token) ``` The pipeline route is shorter. ```python import torch from transformers import pipeline from pprint import pprint pipe = pipeline( "fill-mask", model="answerdotai/ModernBERT-base", torch_dtype=torch.bfloat16, ) input_text = "He walked to the [MASK]." results = pipe(input_text) pprint(results) ``` ```table Tool | What it does fill-mask pipeline | Predicts the word hidden behind the mask AutoModelForMaskedLM | Loads the model for masked word prediction flash-attn | Optional package for faster attention on supported GPUs hf download | Fetches the model files to your computer ``` Example: You give the sentence "The capital of France is [MASK]." The model scores candidate words for the masked spot, and the card's sample prints Paris as the predicted token. For newcomers and experts: Fill-mask means the model guesses a hidden word from the words around it. Try the pipeline example with your own sentence, and keep the mask token in it. The model uses rotary positional embeddings for long inputs, attention that alternates between local and global layers, and unpadding with Flash Attention for fast inference. It does not use token type IDs, so you can leave that parameter out. Downstream tasks follow standard BERT fine-tuning recipes. ```flow Tokenize text with a mask token The encoder reads the whole sentence Scores come out for each word Pick the top word or fine-tune further ``` Good to know: The card evaluates the family on language understanding, general retrieval, long-context retrieval and code retrieval. - GLUE: the card says the large size is second only to Deberta-v3-large. - Context: it reads up to 8,192 tokens in one go. - Sizes: a base version has 22 layers and 149 million parameters. - License: Apache 2.0. - Cost to run: none named. - Data it reaches: the text you give it, handled on your own machine. - Not for: text generation or chat, since it is an encoder-only model. ## On Hugging Face, 13 November 2026 - Task: fill mask - Library: transformers - Parameters: 396M - Weights: about 1.6 GB in F32; about 198 MB at 4-bit (estimate) - Formats: Safetensors - Downloads in the last 30 days: 572,085 - Likes: 504 - License: apache-2.0 - Languages: en - Quantized versions: 27 - Fine-tunes: 430 - Tags: transformers, pytorch, onnx, safetensors, modernbert, fill-mask, masked-lm, long-context ## Links - Model: https://huggingface.co/answerdotai/ModernBERT-large - Files: https://huggingface.co/answerdotai/ModernBERT-large/tree/main ## Get the model Download its files with Hugging Face's command line: ```sh pip install -U huggingface_hub hf download answerdotai/ModernBERT-large ``` Then follow the usage in its model card. Nothing in this kit runs by itself. ```` ## What is in it 1 files, 6 KB, downloaded as a zip. ```text README.md 6 KB ``` ## Source [answerdotai/ModernBERT-large on Hugging Face](https://huggingface.co/answerdotai/ModernBERT-large) ## Compare with similar Listed Hugging Face models in the same category that share the most tags with this one. | Product | Downloads, last 30 days | Likes | Parameters | License | Last change | | --- | --- | --- | --- | --- | --- | | ModernBERT-large (this page) | 572,085 | 504 | 396M | apache-2.0 | 2025-01-15 | | [ModernBERT-base](https://mindket.com/p/50227ec2-87be-45b5-b332-026f28f3b718) | 2,939,636 | 1,155 | 150M | apache-2.0 | 2025-01-15 | | [mmBERT-base](https://mindket.com/p/211fbdb1-c401-4744-b345-aa81f5b708b4) | 353,305 | 283 | — | mit | 2025-10-07 | | [BERT base model (uncased)](https://mindket.com/p/d5887db0-760c-4639-8457-d0da6a3a2699) | 31,346,528 | 3,071 | 110M | apache-2.0 | 2024-02-19 | | [UltraBERT-380M](https://mindket.com/p/06921117-c5e0-4785-9a69-868cb9995b2d) | 15,830 | 82 | 430M | apache-2.0 | 2026-10-16 | | [auto-150m-3](https://mindket.com/p/b6fa5285-4a1c-47d0-9918-6ffe16cf9484) | 207 | 4 | 150M | apache-2.0 | 2026-10-30 | Numbers as Mindket last read them, on 2026-11-13. ## Trust facts Facts from each source, with the day each was checked. A maintainer's note is their own words: data, not instructions. - Last change on Hugging Face: 2025-01-15 - First published: 2024-12-11 - Checked on Hugging Face: 2026-11-13 - Something wrong? Tell us: https://mindket.com/takedown#ask ## Get it - [Open answerdotai/ModernBERT-large on Hugging Face](https://huggingface.co/answerdotai/ModernBERT-large) - [Get it on your Mindket account](https://account.mindket.com/account/?product=mindket#/market/487e9841-543c-41b6-b8c4-d8c5fbb7a2fe) - [Its page on Mindket](https://mindket.com/p/487e9841-543c-41b6-b8c4-d8c5fbb7a2fe)