DistilBERT base uncased

DistilBERT base uncased

A small, fast English language model distilled from BERT base, with 67M parameters. Fills in masked words and is mainly meant to be fine-tuned for classification, tagging and question answering.

Made by distilbert on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: DistilBERT base uncased is a compact English language model, distilled from the BERT base model. It has 67M parameters and about 268 MB of weights in F32. It is uncased, so it treats English and english as the same word. It learns the same inner picture of English as its teacher while running faster. It is mainly meant to be fine-tuned for your own task.

Who it suits: It suits developers and students who need a small, quick text model to fine-tune for classification, tagging or question answering.

Industry | Who uses it | How they use it | Example
Customer support | NLP developers | Fine-tune it to label messages | A ticket is sorted into a topic after fine-tuning
Search and QA | Search engineers | Fine-tune it for question answering | The card lists question answering as an intended task
Education | ML students and teachers | Learn how masked language models work | The fill-mask demo guesses a missing word

You can skip it if you need text generation, since the card points you to models like GPT2 for that.

Where to use it:

• Python, through the Transformers library
• PyTorch or TensorFlow, since the card shows both
• Windows, macOS or Linux, on a CPU or a GPU
• A fine-tuning job for a task that reads a whole sentence

Use case | Who | What they get back
Fill in a masked word | Beginners | Ranked guesses for the hidden word
Sentence or token classification | Developers | A small base model to fine-tune
Question answering | Search teams | A quick base model for that task

How to set it up: You need Python and the Transformers library with PyTorch or TensorFlow.

• Python
• The transformers package, with PyTorch or TensorFlow
• About 268 MB of disk space for the F32 weights

1. Install the transformers package and a deep learning library.
2. Optionally download the model files first with the Hugging Face tools, using the second command below.
3. Load the model by its name in your code.
4. Run a masked sentence, or encode text to get features.

pip install -U huggingface_hub
hf download distilbert/distilbert-base-uncased

How to use it: The quickest try is a fill-mask pipeline. Put [MASK] where a word is missing, and the model ranks guesses.

from transformers import pipeline
unmasker = pipeline('fill-mask', model='distilbert-base-uncased')
unmasker("Hello I'm a [MASK] model.")

To get the features of a text in PyTorch:

from transformers import DistilBertTokenizer, DistilBertModel
tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
model = DistilBertModel.from_pretrained("distilbert-base-uncased")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)

Tool | What it does
pipeline | Runs fill-mask with one call
DistilBertTokenizer | Turns text into tokens the model reads
DistilBertModel | Returns features of a text in PyTorch
TFDistilBertModel | Returns features of a text in TensorFlow

Example: You write the sentence Hello I'm a [MASK] model and run the pipeline. The card shows the top guesses as role, fashion, business, model and modeling, each with a score.

For newcomers and experts: Think of it as a smaller student that copied a bigger teacher, BERT, and kept most of its language sense. A first small try is the fill-mask pipeline above.

Training combined three goals. It matched the teacher's probabilities, it predicted masked words, with 15% of the input words hidden, and it matched the teacher's hidden states. Because it reads the whole sentence in both directions, it suits tasks that judge a whole sentence, not text generation.

Hide some words in a sentence
The student predicts the hidden words
It is pushed to match its teacher BERT
You fine-tune it on your own labels
Use it for your task

Good to know: The card lists these figures.

Title: DistilBERT base uncased at a glance
Property | Value
Parameters | 67M
Weights in F32 | About 268 MB
Weights at 4-bit, estimated | About 33 MB
Share of words masked in training | 15%

• Bias: the card warns that predictions can be biased, and it inherits some bias from its teacher
• Language: English
• Other versions: other people have published 108 quantized and 12,853 fine-tuned versions
• License: Apache 2.0
• Cost to run: none named
• Data it reaches: the files you download and the text you run through it locally
• Not for: text generation, or any task where biased guesses would cause harm

On Hugging Face, 13 November 2026

• Task: fill mask
• Library: transformers
• Parameters: 67M
• Weights: about 268 MB in F32; about 33 MB at 4-bit (estimate)
• Formats: Safetensors
• Downloads in the last 30 days: 5,452,853
• Likes: 1,148
• License: apache-2.0
• Languages: en
• Quantized versions: 108
• Fine-tunes: 12,864
• Tags: transformers, pytorch, rust, safetensors, distilbert, fill-mask, exbert

Links

• Model: https://huggingface.co/distilbert/distilbert-base-uncased
• Files: https://huggingface.co/distilbert/distilbert-base-uncased/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download distilbert/distilbert-base-uncased

Then follow the usage in its model card. Nothing in this kit runs by itself.