OLMoE-1B-7B-0924

OLMoE-1B-7B-0924

Continues English text with a small-cost Mixture-of-Experts model that has 1B active and 7B total parameters. Fully open code, data and checkpoints make it a fit for researchers and developers using transformers.

Made by allenai on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: OLMoE-1B-7B-0924 is an open language model from allenai, released in September 2024. It is a Mixture-of-Experts design with 1B active parameters out of 7B in total, so each token costs about as much as a 1B model. The card calls it fully open and says it holds its own against larger models such as Llama2-13B.

Who it suits: Researchers who want to inspect every training stage fit best, and so do developers who want to try a Mixture-of-Experts model through transformers. Students of how language models are trained can follow the released stages.

Industry | Who uses it | How they use it | Example
Research | Language-model researchers | Study a model whose code, data and logs are published for each training stage | Loading an early checkpoint by passing its name as a revision
Software | Developers | Run a Mixture-of-Experts model with 1B active parameters in their own Python code | Generating a continuation from a short text prompt
Education | Students of language-model training | Follow the pretraining, supervised fine-tuning and preference stages with released checkpoints | Comparing the main checkpoint with its tuned Instruct version

You can skip it if you need replies in a language other than English, since the record lists English only.

Where to use it: It runs in Python through the transformers library and torch.

• Windows, macOS or Linux, where the files download through Python
• A CUDA GPU when one is present, otherwise the CPU, as the card's sample chooses
• Your own scripts and notebooks that load models with transformers

Use case | Who | What they get back
Text continuation | Developers | A passage that carries on from a prompt
Checkpoint study | Researchers | Models from different points in training
Model comparison | Students | A small open model to set beside other 1B-class ones

How to set it up: You need Python with a few libraries and room for the weights.

• Python on Windows, macOS or Linux
• torch
• transformers installed from source, which the card asks for until a later release carries the needed change
• About 13.8 GB of disk for the BF16 weights, or about 3.5 GB at 4-bit, which is an estimate

1. Install the Hugging Face hub library.
2. Download the model files with the hf command.
3. Install torch and transformers from source, as the card says.

pip install -U huggingface_hub
hf download allenai/OLMoE-1B-7B-0924

How to use it: Load the model and tokenizer, tokenize a prompt, and call generate. The card's sample moves everything to the GPU when one exists and prints a reply of up to 64 tokens.

from transformers import OlmoeForCausalLM, AutoTokenizer
import torch

DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

model = OlmoeForCausalLM.from_pretrained("allenai/OLMoE-1B-7B-0924").to(DEVICE)
tokenizer = AutoTokenizer.from_pretrained("allenai/OLMoE-1B-7B-0924")
inputs = tokenizer("Bitcoin is", return_tensors="pt")
inputs = {k: v.to(DEVICE) for k, v in inputs.items()}
out = model.generate(**inputs, max_length=64)
print(tokenizer.decode(out[0]))

To load a different checkpoint, pass a revision name such as step10000-tokens41B to from_pretrained. This snippet lists every revision.

from huggingface_hub import list_repo_refs
out = list_repo_refs("allenai/OLMoE-1B-7B-0924")
branches = [b.name for b in out.branches]

Tool | What it does
OlmoeForCausalLM.from_pretrained | Loads the model weights
AutoTokenizer.from_pretrained | Loads the matching tokenizer
generate | Writes the continuation, here up to 64 tokens
list_repo_refs | Lists the saved revisions and branches

Example: You give the model the start of a sentence, "Bitcoin is", and it writes the rest. In the card's sample, the continuation explains that bitcoin is a digital currency created and held electronically.

For newcomers and experts: The main model continues whatever text you give it, so the card's sample starts from a sentence opening. The first small thing to try is that sample, with a prompt of your own. Tuned versions for instructions are separate checkpoints.

Only 1B of the 7B parameters work on each token, so serving cost tracks a 1B model while the weights stay larger. The main branch was annealed from the step1200000-tokens5033B checkpoint for 100B more tokens, and that is the one the tuned versions build on. Weights were kept in FP32 during training, and the card says casting to BF16 showed no performance drop. A separate fp32 branch holds the original, which gives slightly different results.

Download the weights with hf
Load model and tokenizer in transformers
Your prompt becomes tokens
A few experts handle each token
The continuation is decoded to text

Good to know: The card names three branches that matter, and tuned versions come as separate checkpoints.

• step1200000-tokens5033B: the pretraining checkpoint used for annealing.
• main: the annealed checkpoint, used for the SFT and Instruct adaptations.
• fp32: the FP32 version of main.
• Other versions: the record counts 9 quantized and 7 fine-tuned versions made by other people.

Title: MMLU score among models with about 1B active parameters
OLMoE-1B-7B: 54.1 points
DCLM-1B: 48.5 points
TinyLlama-1B: 33.6 points
OLMo-1B (0724): 32.1 points
Pythia-1B: 31.1 points

• License: Apache 2.0.
• Cost to run: no API key, account or paid service is named, but the BF16 weights take about 13.8 GB.
• Data it reaches: it downloads files from Hugging Face, and the sample generates text on your own machine.
• Not for: the source names no forbidden uses.

On Hugging Face, 13 November 2026

• Task: text generation
• Library: transformers
• Parameters: 6.9B
• Weights: about 13.8 GB in BF16; about 3.5 GB at 4-bit (estimate)
• Formats: Safetensors
• Downloads in the last 30 days: 45,654
• Likes: 158
• License: apache-2.0
• Languages: en
• Quantized versions: 9
• Fine-tunes: 7
• Tags: transformers, safetensors, olmoe, text-generation, olmo, co2_eq_emissions

Links

• Model: https://huggingface.co/allenai/OLMoE-1B-7B-0924
• Files: https://huggingface.co/allenai/OLMoE-1B-7B-0924/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download allenai/OLMoE-1B-7B-0924

Then follow the usage in its model card. Nothing in this kit runs by itself.