tiny-llm

tiny-llm

Continues the start of a story you type, using a 30M-parameter language model trained from scratch on TinyStories and served by the project's own inference server. A learning demo of routing on free CPUs.

Made by CoolDip00 on Hugging Face. These notes are Mindket's, written from its Space page and README; the app's own files and license are on Hugging Face.

About

What it is: tiny-llm is a small language model with 30 million parameters, trained from scratch on TinyStories. The project serves it with its own inference engine, which does continuous batching, streaming and load shedding. This Space is a small router in front of two private copies of the server, running on Hugging Face's free CPU tier. The author calls it a learning demo of routing.

Who it suits: It suits people who want to see a small language model and a hand-built serving stack work end to end, and developers who study routing, rate limits and status pages.

Industry | Who uses it | How they use it | Example
Software | Backend engineers | Study a router that sits in front of model replicas | Read how requests are shed when both replicas are busy
AI engineering | Inference engineers | See a from-scratch server with batching and streaming | Watch latency and throughput on the status page
Education | Students | Try a tiny model and learn what it can and cannot do | Type the start of a story and read the rest
Research | Small model hobbyists | Compare a 30M model with larger ones | Send the same story opening to the API

You can skip it if you need good writing or steady uptime, since the stories are simple and the Space sleeps when nobody visits.

Where to use it:

• A web browser: open the Space page on Hugging Face and type the start of a story.
• Any HTTP client: send a request to the OpenAI-style completions path.
• Windows, macOS or Linux with Docker: build the image and run it yourself.

Use case | Who | What they get back
Continue a story | Anyone | A short story that carries on from your opening
Check the service | Operators | Live latency, throughput, goals and replica state on the status path
Call the API | Developers | An OpenAI-style completion reply

How to set it up: The hosted page needs nothing from you. A local copy needs these things first.

• Docker on Windows, macOS or Linux.
• Python and pip, to download the files with the hf command.

1. Install the huggingface_hub package with pip.
2. Download the Space's files into a folder with the hf command.
3. Move into that folder.
4. Build the Docker image from the Dockerfile.
5. Run the image and map port 7860.

pip install -U huggingface_hub
hf download CoolDip00/tiny-llm --repo-type space --local-dir tiny-llm
cd tiny-llm
docker build -t tiny-llm .
docker run -it -p 7860:7860 tiny-llm

How to use it: On the Space page, write how a story begins and the model keeps writing from there. For the API, send a JSON POST to the completions path with a prompt and a token limit. The status path shows live latency, throughput and the state of each replica.

{"prompt": "Once upon a time", "max_tokens": 60}

Tool | What it does
Demo page | Continues the story you start
/status | Shows latency, throughput, service goals and replica state
/v1/completions | Takes an OpenAI-style completion request
/stats | Shows the commit the replicas run

Example: You type Once upon a time on the demo page. The model continues with a short, simple story. If both replicas are busy you get a 429 reply straight away, and after the Space has slept you get a waking up message, so you retry a minute later.

For newcomers and experts: If you are new, open the page and type one story opening. Expect plain and sometimes odd stories, because the model is tiny. Check the status path afterwards to see what the service measured.

If you know the field, the stated goals are a 95th percentile time to first token of 2 seconds or less and 99 percent availability. A visitor gets 20 requests a minute, with a burst of up to 5. When both replicas are full, the reply is an immediate 429 and nobody waits in line. The Space runs a router build from the project repository on two private replicas on 2 vCPUs.

You send a prompt to the router
The router checks your rate limit
It picks a replica that has room
The replica streams the continuation back
A full pair of replicas returns 429 at once

Good to know: The model is small and runs on shared CPUs, so stories are simple and sometimes odd. After 48 hours with no visitors the Spaces go to sleep, and no job wakes them.

• License: mit
• Cost to run: none named, since it uses Hugging Face's free CPU tier
• Data it reaches: the service logs request details such as timings, token counts, status and finish reason, and it does not log what you type or what the model writes. The logs sit in a private dataset for 30 days
• Privacy: the service counts visitors with a salted hash made from the IP address and keeps no copy of the address
• Models named: the 30M TinyStories model by CoolDip00, TinyLlama 1.1B Chat and TinyStories 1M

On Hugging Face, 13 November 2026

• Likes: 1
• Built with: docker
• License: mit

Links

• Space: https://huggingface.co/spaces/CoolDip00/tiny-llm
• Files: https://huggingface.co/spaces/CoolDip00/tiny-llm/tree/main

Try it

Open the Space in your browser, or copy it to your own Hugging Face account from its page. Nothing in this kit runs by itself.