Sekai-1

Sekai-1

A 95.6M-parameter English chat model trained from scratch on one RTX 5060 Ti, small enough to run on a CPU. For people who want a tiny local language model to experiment with.

Made by 8BitStudio on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: Sekai-1 is the first model in 8BitStudio's Sekai series. It has 95.6M parameters and was built from the ground up, with all training done on one RTX 5060 Ti card. It was pretrained on 58.5B tokens and then fine-tuned for chat. A 2,048-token context window and an English-only vocabulary come with it.

Who it suits: Hobbyists, students and developers who want a tiny chat model they can run on their own computer. It also suits people who study how small models are trained.

Industry | Who uses it | How they use it | Example
Local AI | Hobbyists | Run a chat model on a CPU with no cloud account | Ask simple questions from a short Python script
Education | Students and teachers | Study how a small model is built and trained | Read the card's details on layers, tokenizer and training data
Research | Small model researchers | Compare models under 100M parameters | Check its zero-shot scores against others on the leaderboard
Software | Developers | Prototype features that need a tiny English model | Test short answers, rewriting and summaries on weak hardware

You can skip it if you need accurate facts, long conversations, math or reasoning beyond simple steps.

Where to use it:

• Windows, macOS and Linux with Python, through the transformers library
• A plain CPU, since the card says the model is small enough for one
• Local scripts that use the ChatML chat format

Use case | Who | What they get back
Short chat answers | Hobbyists | Short English replies to everyday questions
Summarizing and rewriting | Writers | Brief rewrites and summaries, with checking needed
Code help | Beginners | Simple code writing and fixing, which the chat tuning covered

How to set it up: You need Python with two packages.

• Python on Windows, macOS or Linux
• The transformers and torch packages
• A CPU is enough for this model

1. Install Python if you do not have it.
2. Install the two packages with the command below.
3. Run the Python script that follows.

pip install transformers torch

How to use it: Load the tokenizer and model by name, put your message into the chat template and call generate. The script prints the model's reply.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "8BitStudio/Sekai-1"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

messages = [{"role": "user", "content": "Who are you?"}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_dict=True, return_tensors="pt")
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Sampling defaults sit in generation_config.json, and passing do_sample=False to generate gives deterministic answers. The model reads the ChatML format, shown here.

<|im_start|>user
What is the capital of France?<|im_end|>
<|im_start|>assistant
Paris<|im_end|>

Example: You run the script with the message asking who it is. The model answers in a sentence or two, introducing itself as Sekai-1, since its chat tuning included a few identity examples.

For newcomers and experts: This is a tiny chatbot that runs on a normal computer. A first small step is to run the script above and ask a few everyday questions. Expect mistakes and check anything that matters.

The architecture follows the Qwen3 style: 23 layers, hidden size 576, grouped-query attention with 9 query and 3 key-value heads, SwiGLU, RMSNorm, RoPE and tied embeddings. Its own 24,576-token BPE tokenizer splits numbers into single digits. The optimizers were Muon and AdamW, paired with a learning-rate plan of warmup, a stable stretch and then decay. About 400M tokens of chat data followed pretraining, without chain-of-thought.

Pretrain on 58.5B tokens of mixed text
Decay on a higher-quality mix
Chat fine-tune on about 400M tokens
Run it locally with transformers

Good to know:

• Expect wrong facts, drifting in long chats, and poor results on multi-step reasoning or math past basic sums.
• No safety tuning or refusal behavior was added, so keep it out of places that rely on either.
• It understands English only.
• Its leaderboard score is second among models under 100M parameters, measured zero-shot.
• License: apache-2.0.
• Cost to run: none named, a CPU is enough.
• Data it reaches: files downloaded from Hugging Face, and the text you give it.
• Not for: settings that need safety tuning or refusals, or non-English text.

On Hugging Face, 13 November 2026

• Task: text generation
• Library: transformers
• Parameters: 95.6M
• Weights: about 382 MB in F32; about 48 MB at 4-bit (estimate)
• Formats: Safetensors
• Downloads in the last 30 days: 449
• Likes: 6
• License: apache-2.0
• Languages: en
• Tags: transformers, safetensors, qwen3, text-generation, muon, rope, swiglu, chatml, conversational

Links

• Model: https://huggingface.co/8BitStudio/Sekai-1
• Files: https://huggingface.co/8BitStudio/Sekai-1/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download 8BitStudio/Sekai-1

Then follow the usage in its model card. Nothing in this kit runs by itself.