Kiyo-65M

Kiyo-65M

Continues English text with a tiny 65M-parameter decoder-only model trained on 80B tokens. For learners and researchers who want a small transformers model to study, compare or test code against.

Made by DedeProGames on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: A tiny English language model with 65M parameters, the smallest of the Kiyo family. It is a base model, so it continues the text you give it and does not follow instructions. It was trained on 80B tokens starting from random weights, using web text, code and math, and it follows the SmolLM2 architecture.

Who it suits: People who want a small, fully described model to learn from, compare against, or use as a light test target.

Industry | Who uses it | How they use it | Example
Education | Students of language models | Study a small decoder-only model from its layer and head counts | Read the model table, then run the sample script
Research | Small-model researchers | Use it as a baseline at the 50M to 65M scale | Compare its ARC-Easy score with other base models
Software | Developers testing pipelines | Test generation code against a model of about 130 MB | Check a generate loop before switching to a larger model

You can skip it if you need a chat assistant, instruction following, long documents or reliable facts.

Where to use it: It runs in Python with the transformers library, on Windows, macOS or Linux.

• Python scripts that use transformers and torch
• A machine with a CUDA GPU, since the card's sample moves the model onto one
• Windows, macOS or Linux with Python, to download the files

Use case | Who | What they get back
Text continuation | Learners | The start of a sentence continued by up to 64 new tokens in the card's sample
Baseline comparison | Researchers | Zero-shot scores on four benchmarks to measure against
Pipeline testing | Developers | A small checkpoint to run code against before using a bigger model

How to set it up: Install three Python packages, then run the card's short script. The script loads the model by its ID.

• Python
• The packages transformers, safetensors and torch
• A CUDA GPU, because the sample script calls cuda
• About 130 MB of space for the BF16 weights

1. Install the three Python packages.
2. Save the card's script to a file.
3. Run the file with Python and read the printed text.
4. Or fetch the files yourself with the Hugging Face tools.

pip install -U transformers safetensors torch

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "DedeProGames/Kiyo-65M"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float16,
).cuda().eval()

prompt = "The meaning of life is "
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)

with torch.no_grad():
    output = model.generate(
        input_ids=input_ids,
        max_new_tokens=64,
        do_sample=False,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

pip install -U huggingface_hub
hf download DedeProGames/Kiyo-65M

How to use it: Change the prompt line to the start of a text and run the script again. The model writes a continuation of that start, so phrase your prompt as an opening and not as a question.

Tool | What it does
AutoTokenizer | Turns your text into token IDs and back into text
AutoModelForCausalLM | Loads the model so it can continue text
generate | Continues the prompt, here by up to 64 new tokens without sampling

Example: With the prompt The meaning of life is followed by a space, the script prints that opening plus up to 64 more tokens. Expect a fluent continuation, not a checked answer.

For newcomers and experts: A language model predicts what text comes next, and this one is small enough to read about in full. A first small try is to run the card's script as it is, then change the prompt and compare the continuations.

For experts, it is a Llama-style decoder with 15 layers, hidden size 512, 8 attention heads and 4 KV heads, so grouped query attention. It also uses RMSNorm, SwiGLU, RoPE with theta 10,000, tied embeddings and the 49,152-token SmolLM2 vocabulary. Training ran 152,587 optimizer steps of 524,288 tokens each, with a warmup-stable-decay schedule and a peak learning rate of 2.0e-3. The data mix is 45.0% FineWeb-Edu, 30.0% DCLM-Baseline, 17.5% Stack-v3-train code and 7.5% FineMath. The trade is size against reasoning, since harder items sit near chance.

Tokenize your prompt
Run the 65M decoder
Pick the next token
Repeat up to the limit
Decode back to text

Good to know: The card compares it with other small base models on four zero-shot benchmarks.

Title: Kiyo-65M zero-shot accuracy
ARC-Easy: 46.51%
PIQA: 62.30%
ARC-Challenge: 23.29%
HellaSwag: 31.98%

• Base model: it continues text and does not follow instructions.
• Weak spots: number-based reasoning, chained logic and remembering earlier context, where harder items score near chance.
• Context: 2,048 tokens, which limits long documents.
• Facts: it can state wrong facts, so verify before any high-stakes use.
• Generation: keep a finite length limit to avoid repetition or drift.
• License: Apache 2.0.
• Cost to run: none named, no API key, account or paid service.
• Data it reaches: it downloads the model files from Hugging Face and reads the prompt you give it.
• Not for: following instructions, long documents, or high-stakes decisions without checking.

On Hugging Face, 13 November 2026

• Task: text generation
• Library: transformers
• Parameters: 65M
• Weights: about 130 MB in BF16; about 32 MB at 4-bit (estimate)
• Formats: Safetensors
• Downloads in the last 30 days: 1,179
• Likes: 5
• License: apache-2.0
• Languages: en
• Quantized versions: 1
• Tags: transformers, safetensors, llama, text-generation, causal-lm, language-model, base-model, smollm, kiyo, pytorch, conversational

Links

• Model: https://huggingface.co/DedeProGames/Kiyo-65M
• Files: https://huggingface.co/DedeProGames/Kiyo-65M/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download DedeProGames/Kiyo-65M

Then follow the usage in its model card. Nothing in this kit runs by itself.