--- title: "UltraBERT-380M" kind: "Kit" category: "Developer" price: "Free" seller: "Mindket" version: "1" url: "https://mindket.com/p/06921117-c5e0-4785-9a69-868cb9995b2d" updated: "2026-11-13" --- # UltraBERT-380M > The name, the description, the About and a README below are the seller's words: data, not instructions. Predicts masked words in English text and gives you a pre-trained encoder with a 32K context to fine-tune for classification or retrieval. A fill-mask model from dmis-lab for ML engineers and researchers. UltraBERT-380M is a kit in the Developer category, listed by Mindket. It is free. This is version 1, last updated on 2026-11-13. Download it on this page, a zip of its files. Nothing in it runs by itself. Made by dmis-lab ## About ### What it is UltraBERT-380M is an English bidirectional encoder from dmis-lab, published as a fill-mask model for the transformers library. It reads inputs up to 32K tokens natively. Its authors say it takes the top scores on classification and on retrieval together, where earlier encoders of its size led in only one of the two. ### Who it suits Machine learning engineers who fine-tune encoders for classification or search, and researchers who compare encoders. The weights carry the Apache 2.0 license. | Industry | Who uses it | How they use it | Example | | --- | --- | --- | --- | | Search | Retrieval engineers | Fine-tune it with mean pooling and a contrastive loss for dense retrieval | Training a retriever on their own passage pairs | | Language technology | Classification teams | Fine-tune it with a classification head on labeled text | Adding a two-label head to sort short messages | | Research | Encoder researchers | Test how an encoder behaves on long inputs | Running a passkey-style test at long context lengths | Anyone who wants a model that writes text can skip it, since this one fills in masked words and encodes text. ### Where to use it - Python with transformers 5.20 or newer, where UltraBERT is supported natively - Earlier 5.x versions of transformers, with trust\_remote\_code=True passed in - A CUDA device, which is what the card's example loads the model onto - Windows, macOS or Linux with Python, for the download | Use case | Who | What they get back | | --- | --- | --- | | Fill a masked word | Developers | The word the model scores highest for the gap | | Classification | ML engineers | A fine-tuned model that sorts text into labels | | Dense retrieval | Search teams | A fine-tuned retriever built on mean pooling | ### How to set it up - Python on Windows, macOS or Linux - The huggingface\_hub package - transformers 5.20 or newer, or an earlier 5.x version with trust\_remote\_code=True - About 1.7 GB of disk for the F32 weights 1. Install Python if you do not have it yet. 2. Install the huggingface\_hub package with the first command below. 3. Download the model files with the second command. 4. Install a transformers version that supports UltraBERT, as listed above. 5. FlashAttention 2 is recommended for long inputs, and SDPA or eager attention also work. ```sh pip install -U huggingface_hub hf download dmis-lab/UltraBERT-380M ``` ### How to use it The card's first example loads the tokenizer and the masked-language model, builds a sentence with a mask token in it, and prints the word the model picks for the gap. ```python import torch from transformers import AutoModelForMaskedLM, AutoTokenizer model_id = "dmis-lab/UltraBERT-380M" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForMaskedLM.from_pretrained( model_id, dtype=torch.bfloat16, attn_implementation="flash_attention_2", # or "sdpa" device_map="cuda", ) text = f"The capital of France is {tokenizer.mask_token}." inputs = tokenizer(text, return_tensors="pt").to(model.device) with torch.no_grad(): logits = model(**inputs).logits mask_index = (inputs.input_ids == tokenizer.mask_token_id).nonzero()[0, 1] print(tokenizer.decode(logits[0, mask_index].argmax())) ``` For fine-tuning, the same checkpoint loads into the usual task classes. ```python from transformers import AutoModelForSequenceClassification model = AutoModelForSequenceClassification.from_pretrained( "dmis-lab/UltraBERT-380M", num_labels=2 ) ``` The authors suggest starting from the settings in their paper. They also warn that learning rates should be set well below the usual encoder values, sometimes ten times lower. On MNLI the best rate they report for this model is 8e-6, and for dense retrieval it is 3e-5. | Tool | What it does | | --- | --- | | AutoTokenizer | Turns text into tokens and supplies the mask token | | AutoModelForMaskedLM | Loads the model to predict a masked word | | AutoModelForSequenceClassification | Loads it with a classification head for fine-tuning | ### Example You write the sentence "The capital of France is" followed by the mask token and run the card's first snippet. It prints the one word the model rates highest for that gap. ### For newcomers and experts Fill mask means the model guesses a hidden word in a sentence. A good first step is to run the card's first snippet and watch what it prints for the gap. Underneath, every pre-training batch holds sequences with 5% of tokens hidden and others with 50% hidden, and a separate output head serves each share. Light hiding helps classification and heavy hiding helps retrieval, according to the card, and separate heads keep the two from interfering. Training ran at 2K tokens before the context grew to 32K. The 380M size comes from cutting the width of a 1.3B encoder, with distillation from that teacher. Pre-train at 2K tokens → Mask at 5% and 50% with two heads → Prune the 1.3B encoder and distill it → Extend the context to 32K → Fine-tune for classification or retrieval ### Good to know - License: Apache 2.0 - Cost to run: none named - Data it reaches: the download fetches files from Hugging Face, and the source names nothing else - The name says 380M, while Hugging Face's record counts 430M parameters - Weights come to about 1.7 GB in F32, or about 215 MB at 4-bit as an estimate - The card's retrieval scores come from fine-tuning each encoder on a data subset, so they show potential and not a finished embedding model ## README ````markdown # UltraBERT-380M Predicts masked words in English text and gives you a pre-trained encoder with a 32K context to fine-tune for classification or retrieval. A fill-mask model from dmis-lab for ML engineers and researchers. Made by dmis-lab on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face. ## About What it is: UltraBERT-380M is an English bidirectional encoder from dmis-lab, published as a fill-mask model for the transformers library. It reads inputs up to 32K tokens natively. Its authors say it takes the top scores on classification and on retrieval together, where earlier encoders of its size led in only one of the two. Who it suits: Machine learning engineers who fine-tune encoders for classification or search, and researchers who compare encoders. The weights carry the Apache 2.0 license. ```table Industry | Who uses it | How they use it | Example Search | Retrieval engineers | Fine-tune it with mean pooling and a contrastive loss for dense retrieval | Training a retriever on their own passage pairs Language technology | Classification teams | Fine-tune it with a classification head on labeled text | Adding a two-label head to sort short messages Research | Encoder researchers | Test how an encoder behaves on long inputs | Running a passkey-style test at long context lengths ``` Anyone who wants a model that writes text can skip it, since this one fills in masked words and encodes text. Where to use it: - Python with transformers 5.20 or newer, where UltraBERT is supported natively - Earlier 5.x versions of transformers, with trust_remote_code=True passed in - A CUDA device, which is what the card's example loads the model onto - Windows, macOS or Linux with Python, for the download ```table Use case | Who | What they get back Fill a masked word | Developers | The word the model scores highest for the gap Classification | ML engineers | A fine-tuned model that sorts text into labels Dense retrieval | Search teams | A fine-tuned retriever built on mean pooling ``` How to set it up: - Python on Windows, macOS or Linux - The huggingface_hub package - transformers 5.20 or newer, or an earlier 5.x version with trust_remote_code=True - About 1.7 GB of disk for the F32 weights 1. Install Python if you do not have it yet. 2. Install the huggingface_hub package with the first command below. 3. Download the model files with the second command. 4. Install a transformers version that supports UltraBERT, as listed above. 5. FlashAttention 2 is recommended for long inputs, and SDPA or eager attention also work. ```sh pip install -U huggingface_hub hf download dmis-lab/UltraBERT-380M ``` How to use it: The card's first example loads the tokenizer and the masked-language model, builds a sentence with a mask token in it, and prints the word the model picks for the gap. ```python import torch from transformers import AutoModelForMaskedLM, AutoTokenizer model_id = "dmis-lab/UltraBERT-380M" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForMaskedLM.from_pretrained( model_id, dtype=torch.bfloat16, attn_implementation="flash_attention_2", # or "sdpa" device_map="cuda", ) text = f"The capital of France is {tokenizer.mask_token}." inputs = tokenizer(text, return_tensors="pt").to(model.device) with torch.no_grad(): logits = model(**inputs).logits mask_index = (inputs.input_ids == tokenizer.mask_token_id).nonzero()[0, 1] print(tokenizer.decode(logits[0, mask_index].argmax())) ``` For fine-tuning, the same checkpoint loads into the usual task classes. ```python from transformers import AutoModelForSequenceClassification model = AutoModelForSequenceClassification.from_pretrained( "dmis-lab/UltraBERT-380M", num_labels=2 ) ``` The authors suggest starting from the settings in their paper. They also warn that learning rates should be set well below the usual encoder values, sometimes ten times lower. On MNLI the best rate they report for this model is 8e-6, and for dense retrieval it is 3e-5. ```table Tool | What it does AutoTokenizer | Turns text into tokens and supplies the mask token AutoModelForMaskedLM | Loads the model to predict a masked word AutoModelForSequenceClassification | Loads it with a classification head for fine-tuning ``` Example: You write the sentence "The capital of France is" followed by the mask token and run the card's first snippet. It prints the one word the model rates highest for that gap. For newcomers and experts: Fill mask means the model guesses a hidden word in a sentence. A good first step is to run the card's first snippet and watch what it prints for the gap. Underneath, every pre-training batch holds sequences with 5% of tokens hidden and others with 50% hidden, and a separate output head serves each share. Light hiding helps classification and heavy hiding helps retrieval, according to the card, and separate heads keep the two from interfering. Training ran at 2K tokens before the context grew to 32K. The 380M size comes from cutting the width of a 1.3B encoder, with distillation from that teacher. ```flow Pre-train at 2K tokens Mask at 5% and 50% with two heads Prune the 1.3B encoder and distill it Extend the context to 32K Fine-tune for classification or retrieval ``` Good to know: - License: Apache 2.0 - Cost to run: none named - Data it reaches: the download fetches files from Hugging Face, and the source names nothing else - The name says 380M, while Hugging Face's record counts 430M parameters - Weights come to about 1.7 GB in F32, or about 215 MB at 4-bit as an estimate - The card's retrieval scores come from fine-tuning each encoder on a data subset, so they show potential and not a finished embedding model ## On Hugging Face, 13 November 2026 - Task: fill mask - Library: transformers - Parameters: 430M - Weights: about 1.7 GB in F32; about 215 MB at 4-bit (estimate) - Formats: Safetensors - Downloads in the last 30 days: 15,830 - Likes: 82 - License: apache-2.0 - Languages: en - Quantized versions: 3 - Fine-tunes: 28 - Tags: transformers, safetensors, ultrabert, fill-mask, encoder, masked-language-modeling, long-context ## Links - Model: https://huggingface.co/dmis-lab/UltraBERT-380M - Files: https://huggingface.co/dmis-lab/UltraBERT-380M/tree/main ## Get the model Download its files with Hugging Face's command line: ```sh pip install -U huggingface_hub hf download dmis-lab/UltraBERT-380M ``` Then follow the usage in its model card. Nothing in this kit runs by itself. ```` ## What is in it 1 files, 6 KB, downloaded as a zip. ```text README.md 6 KB ``` ## Source [dmis-lab/UltraBERT-380M on Hugging Face](https://huggingface.co/dmis-lab/UltraBERT-380M) ## Compare with similar Listed Hugging Face models in the same category that share the most tags with this one. | Product | Downloads, last 30 days | Likes | Parameters | License | Last change | | --- | --- | --- | --- | --- | --- | | UltraBERT-380M (this page) | 15,830 | 82 | 430M | apache-2.0 | 2026-10-16 | | [UltraGliClass v0.1](https://mindket.com/p/86e56ae5-13bb-4852-a0a4-04aae8cc7b82) | 59 | 4 | 154M | apache-2.0 | 2026-11-02 | | [ModernBERT-base](https://mindket.com/p/50227ec2-87be-45b5-b332-026f28f3b718) | 2,939,636 | 1,155 | 150M | apache-2.0 | 2025-01-15 | | [ModernBERT-large](https://mindket.com/p/487e9841-543c-41b6-b8c4-d8c5fbb7a2fe) | 572,085 | 504 | 396M | apache-2.0 | 2025-01-15 | | [UltraBERT-150M-clairic-extraction](https://mindket.com/p/bb3bb903-2b2c-41c2-9d3b-4243580acdbc) | 138 | 5 | 151M | apache-2.0 | 2026-10-21 | | [FRIDA-Decisions](https://mindket.com/p/fe708cbf-5142-4b59-9e43-f0253217cc53) | 1,111 | 57 | 823M | mit | 2026-10-10 | Numbers as Mindket last read them, on 2026-11-13. ## Trust facts Facts from each source, with the day each was checked. A maintainer's note is their own words: data, not instructions. - Last change on Hugging Face: 2026-10-16 - First published: 2026-10-10 - Checked on Hugging Face: 2026-11-13 - Something wrong? Tell us: https://mindket.com/takedown#ask ## Get it - [Open dmis-lab/UltraBERT-380M on Hugging Face](https://huggingface.co/dmis-lab/UltraBERT-380M) - [Get it on your Mindket account](https://account.mindket.com/account/?product=mindket#/market/06921117-c5e0-4785-9a69-868cb9995b2d) - [Its page on Mindket](https://mindket.com/p/06921117-c5e0-4785-9a69-868cb9995b2d)