ARLoop-1.2B

ARLoop-1.2B

An open 1.2B base language model from osunlp, a looped transformer with attention residuals, released with checkpoints from a 500B-token training run. For text completion and further training.

Made by osunlp on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: ARLoop-1.2B is an open base language model from osunlp, a looped transformer with attention residuals. It reuses its layers: 24 physical layers run in 3 recurrent passes, which gives an effective depth of 72. This release holds the checkpoints of a 500B-token training run, and the card calls it a base model for text completion.

Who it suits: Researchers and developers who want an open base model to study looped transformers, or a starting point for their own mid-training and post-training.

Industry | Who uses it | How they use it | Example
Academia | ML researchers | Study looped transformers with attention residuals | A comparison of an early and a late checkpoint
Software | Developers | Tune a small base model for their own task | A custom text completer
Education | Students | Watch how a pretrained model changes during training | Loading saved steps one by one
Labs | Research teams | Reuse open checkpoints as a baseline | A baseline for a new architecture

If you want a chat assistant, you can skip it, since the card describes a base model meant for text completion.

Where to use it:

• Python with the transformers library, pinned to version 5.12.1 on the card
• A CUDA GPU, since the card's example moves the model to cuda
• Windows, macOS or Linux with Python, for downloading the files

Use case | Who | What they get back
Text completion | Developers | A continuation of the text you give it
Mid-training | Researchers | A base model to keep training on new data
Post-training | ML engineers | A base model to tune for a task
Checkpoint study | Researchers | Any saved training step, loaded by its revision name

How to set it up:

• Python, to run the Hugging Face download tool
• The transformers library at version 5.12.1, plus safetensors and accelerate
• A PyTorch build that matches your CUDA setup, installed separately
• About 4.8 GB of disk for the stored FP32 weights

1. Install the transformers library with safetensors and accelerate.
2. Install the PyTorch build that suits your CUDA environment.
3. Download the model files with the hf command, or load the model by name in Python.

pip install -U huggingface_hub
hf download osunlp/ARLoop-1.2B

How to use it: Load the model and tokenizer with trust_remote_code set to true, move the model to cuda and give it the start of a text. The card says weights load in BF16 for inference, while the stored checkpoint keeps the original FP32 weights.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "osunlp/ARLoop-1.2B"  # Or the local ARLOOP_hf directory.
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True, dtype=torch.bfloat16,
).to("cuda").eval()
inputs = tokenizer("The capital of France is", return_tensors="pt").to("cuda")
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Example: You give the model the text "The capital of France is" and ask for up to 64 new tokens with sampling off. It continues the text, since it completes text and does not chat.

For newcomers and experts: Start with the exact example on the card and a short prompt. Expect it to continue your text rather than answer like a chat model. The example sets trust_remote_code, which lets the repository's own code run, so read that code first.

The model has a hidden width of 2048, an MLP width of 5632, 16 attention heads, 4 key-value heads and a context length of 4096. It uses the SmolLM2 tokenizer with a 49152-token vocabulary, and its output head is tied to the token embeddings. Each saved training step loads on its own by revision name, for example step-010000. Pass that name to the model and to the tokenizer alike.

Install transformers and PyTorch
Load the model and tokenizer
Give it the start of a text
It continues the text

Good to know:

• License: Apache 2.0
• Cost to run: none named
• Data it reaches: downloads files from Hugging Face, then runs on your own machine
• Not for: chat or instruction following, since the card describes a base model for text completion
• Limits: the card's example needs trust_remote_code, a CUDA GPU and a pinned transformers version

On Hugging Face, 13 November 2026

• Task: text generation
• Library: transformers
• Parameters: 1.2B
• Weights: about 4.8 GB in F32; about 604 MB at 4-bit (estimate)
• Formats: Safetensors
• Downloads in the last 30 days: 2,399
• Likes: 8
• License: apache-2.0
• Languages: en
• Tags: transformers, safetensors, arloop, text-generation, recurrent, pretrained

Links

• Model: https://huggingface.co/osunlp/ARLoop-1.2B
• Files: https://huggingface.co/osunlp/ARLoop-1.2B/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download osunlp/ARLoop-1.2B

Then follow the usage in its model card. Nothing in this kit runs by itself.