Modern LLM From Scratch

Modern LLM From Scratch

A small decoder-only language model, about 19 million parameters, written and trained from scratch on TinyStories, with a demo page. For learners studying how a small transformer works.

Made by Kakarot2812 on Hugging Face. These notes are Mindket's, written from its Space page and README; the app's own files and license are on Hugging Face.

About

What it is: This Space is a demo of a small language model that its author wrote and trained from scratch with PyTorch on the TinyStories dataset. The model is decoder-only and has about 19 million parameters. The demo lets you choose between a pretrained checkpoint and a fine-tuned one and then generate text.

The source calls it a small experimental model meant for learning and experimentation, not for production use.

Who it suits: It suits students and curious developers who want to read the code of a small modern transformer and try it.

Industry | Who uses it | How they use it | Example
Education | Students | Read the model code to learn how a decoder-only transformer is built | A student follows model.py and layers.py side by side
Machine learning | ML practitioners | Study the comparison between a GPT-2 style setup and a modern one | A practitioner reads the ablation results for the tied variant
Teaching | Instructors | Show what KV caching does to generation speed | An instructor points to the two throughput numbers
Hobbyists | Tinkerers | Try sampling settings on a tiny model | A tinkerer raises the temperature and reads the change

You can skip it if you need a model that follows general instructions or writes reliable text.

Where to use it: You use it in these places.

• A web browser, on the Space's page on Hugging Face
• A local copy on Windows, macOS or Linux, which needs Python 3.12

Use case | Who | What they get back
Generate story text | Anyone | A continuation of your prompt from the pretrained checkpoint
Try the fine-tuned checkpoint | Learners | Text written from one fixed children's story instruction
Read the code | Developers | The model, layers and config files of a small transformer

How to set it up: Online use needs only a browser. A local copy needs Python 3.12 and pip.

• A web browser, for the online page
• Python 3.12 with pip, for a local copy
• Python packages from requirements.txt

1. Open the Space's page on Hugging Face to use it online, with nothing to install.
2. For a local copy, install the Hugging Face Hub package.
3. Download the Space's files into a folder and move into it. The checkpoints come with them.
4. Install Gradio 6.31.0 and the packages in requirements.txt.
5. Start the app.

pip install -U huggingface_hub
hf download Kakarot2812/modern-llm-from-scratch --repo-type space --local-dir modern-llm-from-scratch
cd modern-llm-from-scratch
pip install gradio==6.31.0
pip install -r requirements.txt
python app.py

How to use it: Start the app and open the page it serves. Pick a checkpoint, then adjust the sampling settings and generate text.

python app.py

Tool | What it does
Pretrained checkpoint | Accepts a custom text prompt and continues it
SFT Fine-tuned checkpoint | Uses one fixed children's story writing instruction
Temperature, top-k and top-p | Change how varied the generated text is

Example: You pick the pretrained checkpoint, type the opening of a short story and keep the default sampling settings. The model continues the story in the style of children's tales. It can repeat itself or drift, as the source warns.

For newcomers and experts: A language model predicts the next word piece again and again. This one is tiny and learned from simple children's stories, so its text is plain and sometimes wanders. Try the pretrained checkpoint first and compare a low and a high temperature.

Under the hood, the model has 6 layers, hidden size 256, 8 attention heads and a context of 512 tokens. It uses RoPE, RMSNorm, SwiGLU, tied embeddings and KV caching, with the GPT-2 vocabulary of 50,257 tokens. Pretraining ran for 6,000 steps and reached a validation loss of 1.8875. The fine-tuned checkpoint ran 600 more steps on one fixed instruction, so it is an experiment and not general instruction following.

Pick a checkpoint and type a prompt
The text is split with the GPT-2 vocabulary
Six transformer layers predict the next token
Temperature, top-k and top-p choose the token
The generated text appears

Good to know: The source reports its own numbers, and it says they depend on hardware and workload.

Title: Generation speed with and without KV caching
Without KV cache: 143.69 tokens/sec
With KV cache: 292.49 tokens/sec

Title: Validation loss in the architecture comparison, lower is better
GPT-2 baseline: 3.0835 loss
Modern, untied: 2.7809 loss
Modern, tied: 2.8413 loss

• Limits: output may loop, make little sense or stop short, and the text quality metrics rely partly on rules of thumb.
• The comparison does not isolate every architectural part, and weight tying cut the parameter count in that test.
• Built with Python, PyTorch, tiktoken and Gradio on the TinyStories dataset.
• License: MIT.
• Cost to run: none named. The Space uses zero-a10g hardware at Hugging Face.
• Data it reaches: the prompts you type. The checkpoints are files in the repository.
• Not for: production use or general instruction following, as the source states.

On Hugging Face, 13 November 2026

• Likes: 1
• Built with: gradio
• License: mit

Links

• Space: https://huggingface.co/spaces/Kakarot2812/modern-llm-from-scratch
• Files: https://huggingface.co/spaces/Kakarot2812/modern-llm-from-scratch/tree/main

Try it

Open the Space in your browser, or copy it to your own Hugging Face account from its page. Nothing in this kit runs by itself.