ARLoop-0.6B

ARLoop-0.6B

Completes text as a small pretrained looped transformer with attention residuals. A base model for researchers who want to continue its training with their own mid-training or post-training.

Made by osunlp on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: ARLoop-0.6B is a looped transformer with attention residuals, pretrained and released openly by the osunlp group. It is the final checkpoint of a 180B-token training run. It is a base model for text completion, not a chat assistant.

Who it suits: Researchers and developers who want a small open base model to study or to train further.

Industry | Who uses it | How they use it | Example
AI research | Researchers | Study looped transformers with attention residuals | Compare intermediate checkpoints during training
Model training | ML engineers | Start mid-training or post-training from a base model | Fine-tune it on their own text
Software | Developers | Run a small completion model on a laptop | Complete a short sentence from a prompt

If you want a ready chat assistant, you can skip this base model.

Where to use it:

• Your own computer with Python, on a CPU or a CUDA GPU
• Training setups for mid-training and post-training
• Research notebooks that compare checkpoints

Use case | Who | What they get back
Text completion | Developers | A continuation of the text you give it
Further training | ML engineers | A base to train from
Checkpoint study | Researchers | Intermediate versions to compare

How to set it up:

• Python with pip
• A PyTorch build that suits your CPU or CUDA setup, installed separately
• About 2.3 GB of disk space for the F32 weights

1. Install the transformers version and safetensors named below.
2. Install a PyTorch build for your CPU or CUDA environment.
3. Run the load and generate script.

pip install 'transformers==5.22.0' safetensors
# Install a PyTorch build suitable for your CPU or CUDA environment separately.

How to use it: Run the script below. It loads the model in BF16, feeds it a short prompt and prints the completion. Intermediate checkpoints load by passing a revision name such as step-050000 to both from_pretrained calls.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "osunlp/ARLoop-0.6B"
device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True, dtype=torch.bfloat16,
).to(device).eval()
inputs = tokenizer("The capital of France is", return_tensors="pt").to(device)
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Example: You run the script with the prompt "The capital of France is". The model continues the sentence, up to 64 new tokens, and the script prints the prompt plus that continuation.

For newcomers and experts: If you are new, run the script above as it is and watch it continue one sentence. A base model completes text and does not follow instructions the way a chat assistant does.

For experts, 18 physical layers are shared across 3 recurrent passes, for an effective depth of 54. The card also lists the output head as tied to the token embeddings.

Title: Architecture from the card
Property | Value
Physical layers | 18, shared across 3 passes
Effective depth | 54
Hidden width | 1536
Context length | 4096
Vocabulary | 49152

Text is split into tokens
18 shared layers run in 3 loops
Attention residuals link earlier blocks
The model predicts the next token

Good to know: The stored checkpoint keeps the original FP32 weights, and inference loads them in BF16.

• License: Apache 2.0, as the card states it.
• Cost to run: no API key or account is named. A CPU is enough to try it, and a CUDA GPU is optional.
• Data it reaches: downloading reaches Hugging Face. The script sets trust_remote_code, so it runs code that ships with the repository.
• Not for: it is a base model for completion, not a chat assistant. The card does not recommend it as one.

On Hugging Face, 13 November 2026

• Task: text generation
• Library: transformers
• Parameters: 583M
• Weights: about 2.3 GB in F32; about 292 MB at 4-bit (estimate)
• Formats: Safetensors
• Downloads in the last 30 days: 2,164
• Likes: 5
• License: apache-2.0
• Languages: en
• Tags: transformers, safetensors, arloop, text-generation, recurrent, pretrained

Links

• Model: https://huggingface.co/osunlp/ARLoop-0.6B
• Files: https://huggingface.co/osunlp/ARLoop-0.6B/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download osunlp/ARLoop-0.6B

Then follow the usage in its model card. Nothing in this kit runs by itself.