Build a Large Language Model (From Scratch)

What it is

This is the official code repository for Sebastian Raschka’s book Build a Large Language Model (From Scratch), which Manning publishes. Its README says you build a GPT-like LLM in PyTorch step by step, from text data through pretraining to finetuning, without any external LLM libraries. The method for training this small model mirrors the approach used to build large models such as those behind ChatGPT. The repository holds the book’s notebooks, scripts and bonus material. The book itself is a separate product that the README links at Manning and Amazon.

1. Set up Python and install the requirements
2. Open a chapter’s notebook in JupyterLab
3. Chapters 2 to 4: text data, attention, a GPT model
4. Chapter 5: pretrain it, load GPT-2 weights
5. Chapters 6, 7: finetune to classify and follow instructions

Who it suits

Programmers who know Python and want to see how an LLM works by coding one. The README calls a strong foundation in Python the most important prerequisite. Experience with deep neural networks helps, and PyTorch basics are useful but not required. Appendix A is a short introduction to PyTorch.

Skip it if you want a ready model to call or a library to build products on. The README describes the model you train as small but functional, made for education.

Where to use it

The code is Jupyter notebooks and Python scripts that use PyTorch. Chapters 5 to 7 also use TensorFlow to read the GPT-2 weights. The README says the main chapters run on a conventional laptop with no special hardware, and use a GPU automatically when one is there. The setup page says an NVIDIA GPU in the cloud helps most in chapters 5 to 7.

• Your own computer: a uv, pixi, pip or Conda environment, with JupyterLab or VS Code
• A Docker DevContainer in VS Code, with an option for an NVIDIA GPU
• Lightning AI Studio: a persistent cloud environment with VS Code and JupyterLab on CPUs and GPUs
• Google Colab: open a chapter notebook from GitHub and upload the files it imports
• Your browser, to read only: GitHub shows the files and renders the Markdown

How to set it up

Get it saves a zip of Mindket’s setup notes in your downloads folder, with each route’s steps from the repository’s setup pages. If Python is already installed, the setup page’s quickest route is one line from the repository’s root folder: pip install -r requirements.txt. The author’s own route adds a uv virtual environment first.

1. Clone the repository and open its folder.
2. Make a virtual environment and activate it. The author uses uv for this.
3. Install the requirements.
4. Open a chapter in JupyterLab. Start JupyterLab and open the chapter’s notebook. You can run the environment check first.

git clone --depth 1 https://github.com/rasbt/LLMs-from-scratch.git
cd LLMs-from-scratch

pip install uv
uv venv --python=python3.10
source .venv/bin/activate
uv pip install -r requirements.txt

On Windows, the setup tips say you may have to replace the activate line with source .venv/Scripts/activate or .venv/Scripts/activate. If a dependency fails, fall back to plain pip.

pip install -r requirements.txt

python setup/02_installing-python-libraries/python_environment_check.py
jupyter lab

For Windows the author recommends pixi over the native uv commands, because some dependencies in pyproject.toml cause problems there. On Google Colab, put this line at the top of a notebook:

pip install uv && uv pip install --system -r https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/refs/heads/main/requirements.txt

How to use it

Read a chapter, then open its notebook. A chapter notebook holds all the code as it appears in the book, and chapters 2 to 7 and Appendix A each have an exercise-solutions notebook. The README’s table gives each chapter’s main notebook and a folder of bonus material.

Chapters also have standalone scripts that summarize them, such as gpt_train.py in chapter 5. Chapter 7’s main code README shows how to run two of its scripts, which sit in the ch07/01_main-chapter-code folder:

python gpt_instruction_finetuning.py
python ollama_evaluate.py --file_path instruction-data-with-response-standalone.json

The README’s sample output for the second one starts with “Ollama running: True”.

Example

You want to follow the attention chapter. You activate your environment, start JupyterLab from the repository folder and open ch03/01_main-chapter-code/ch03.ipynb.

cd LLMs-from-scratch
source .venv/bin/activate
jupyter lab

Run the cells from the top. When you finish the chapter, open exercise-solutions.ipynb in the same folder and compare it with your own work. The troubleshooting page says these main notebooks are kept in sync with the printed book.

For newcomers and experts

If you are new. A large language model, or LLM, is the kind of model behind chat tools such as ChatGPT. The book teaches how one works, so you can start without knowing. Read the setup recommendations in the README table. Chapter 1 has no code, so the first notebook is Chapter 2’s. If Python is new to you, learn it first. If PyTorch is new, read Appendix A. The README also links the author’s post on how to best read the book.

For experts. The optional llms-from-scratch package on PyPI imports the chapters’ code, such as MultiHeadAttention from llms_from_scratch.ch03, and has KV-cache versions of GPT-2, Llama 3 and Qwen3.

Bonus folders in chapter 4 cover grouped-query, multi-head latent and sliding window attention, Gated DeltaNet and Mixture-of-Experts. Chapter 5 adds from-scratch Llama 3.2, Qwen3, Gemma 3 and Olmo 3 models, and chapter 7 adds DPO. The bonus notebooks need more packages, which one uv line installs:

uv pip install --group bonus

Appendix A’s multi-GPU demo is a script, run as python DDP-script.py, with a torchrun version beside it. On an Apple Silicon Mac, the troubleshooting page says many scripts skip the mps backend on purpose, so rerun on cpu first if the losses diverge. The author’s sequel, Build A Reasoning Model (From Scratch), has its own repository, and the README links bonus material from it.

Good to know

The code and notebooks run on your machine. Installing the packages and the first run of chapters 5 to 7 need the network, and the notebooks load their pictures from the author’s website each time you open one.

• Downloads: chapters 5 to 7 fetch the GPT-2 weights from openaipublic.blob.core.windows.net, with a backup on Backblaze, and the notebooks load their pictures from the author’s website
• Cost: the repository is free. The book and the video course are on Manning’s site. Lightning and Colab are separate services, and the pages give no prices
• Your prompts: the chapter 7 bonus evaluation has one notebook that uses OpenAI’s GPT-4 through its API and one that uses a Llama 3 model run locally with Ollama
• Docker: the DevContainer runs as root by default. If its build fails, the guide says the likely cause is a machine without a compatible NVIDIA GPU, and to remove the runArgs line
• Versions: the check script still accepts Python 3.9, but pyproject.toml requires >=3.10,<3.15. The setup tips recommend 3.10 or newer and give 3.10 or 3.11 as their example, and the native uv page’s example uses 3.11
• Copy errors: the package README’s Chapter 4 import lacks a comma, and Appendix A misspells CUDA_VISIBLE_DEVICES. The native uv page’s Windows line ends in | more, which Mindket reads as a slip. The kit flags each
• Contributions: the author cannot accept changes that extend the main chapter code, because it has to match the printed book
• Removal: delete the folder. A uv environment is the .venv folder inside it, and the Docker page gives docker image rm for the container image. A Conda environment (LLMs) lives outside the folder, and the pages give no step to remove it
• License: the Apache License 2.0 with a changed Source definition that leaves out books about the software and their images. GitHub’s license field reads “Other”. The kit holds Mindket’s notes, not the code or the book