Fine-tuning dataset: build, check and split your training data

Before you start

You need Python, the datasets library (pip install datasets) and the behavior rows from lesson 1. The steps below need a real batch of examples. Start writing them now. Whether the task can be learned shows only once they exist.

One example per line

Fine-tuning data is a JSONL file: one complete JSON object on every line, with no commas between lines and no outer list. Nothing else goes in the file. OpenAI's fine-tuning guide asks for the chat completions format, a messages list of objects that each have a role and a content. TRL, the library lesson 3 trains with, reads the same layout as a conversational dataset.

Line one comes from OpenAI's data preparation cookbook. Line two is this lesson's own, for the ticket-title task from lesson 1. load_dataset("json", data_files="all.jsonl") reads such a file in Hugging Face Datasets, whose docs call JSON Lines the most efficient JSON layout.

{"messages": [{"role": "system", "content": "You are a happy assistant that puts a positive spin on everything."}, {"role": "user", "content": "I fell off my bike today."}, {"role": "assistant", "content": "It's great that you're getting exercise outdoors!"}]}
{"messages": [{"role": "system", "content": "Write a ticket title of at most 12 words."}, {"role": "user", "content": "Hi, the invoice PDF in my account shows last year's logo and I need the new one for my accountant."}, {"role": "assistant", "content": "Invoice PDF shows the old logo instead of the new one"}]}

1. Save one example per line in a file called all.jsonl. Each line is a messages list that ends with the assistant message you want.
2. Reuse the system and user wording your application sends in production. The accuracy guide calls training data that differs subtly in formatting from production inputs one of the most common pitfalls.
3. Write the first batch yourself, or take it from logs and expert answers if you have them. OpenAI lists historical, expert and logged data as good sources, and asks for questions and answers that are specific and clear.

How TRL reads the file

TRL's SFT trainer accepts two dataset types, language modeling and prompt-completion, each in a standard (plain text) or a conversational form. Given a conversational dataset, it applies the model's chat template for you. Which tokens count toward the loss depends on the type.

Set assistant_only_loss=True when the system and user turns are boilerplate you do not want the model to copy. It needs a chat template with {% generation %} markers. TRL patches the template for known families such as Qwen3, and the docs say to check any other model's.

How many examples to start with

OpenAI's guide sets the floor at 10 examples and reports improvements from 50 to 100. Its advice is to begin at the low end of that range with well-crafted demonstrations, evaluate, and add more only if the score rises. A first set that changes nothing is a signal to rethink the task or the prompt. Add training data only after that.

The authors of LIMA went the other way in 2023 and curated 1,000 examples for a 65B LLaMA model, which shows that a small, careful set can teach a format. Your own count comes from your first evaluation, not from either of these numbers, so write the test file before you decide how many more examples to add.

Check the file before you train

A format error costs seconds to find now and an hour to find after training starts. OpenAI's cookbook is marked archived and was written for the chat format of its day, yet its checks still cover the basics of a chat file. The loop below counts seven kinds of error: a non-object entry, a missing messages list, a message without role or content, and an unrecognized key. It also flags an unrecognized role, empty content, and an example with no assistant message.

A clean file prints "No errors found". Otherwise, fix each error type it names and run it again.

import json
from collections import defaultdict

with open("all.jsonl", "r", encoding="utf-8") as f:
    dataset = [json.loads(line) for line in f]

# Format error checks
format_errors = defaultdict(int)
for ex in dataset:
    if not isinstance(ex, dict):
        format_errors["data_type"] += 1
        continue
    messages = ex.get("messages", None)
    if not messages:
        format_errors["missing_messages_list"] += 1
        continue
    for message in messages:
        if "role" not in message or "content" not in message:
            format_errors["message_missing_key"] += 1
        if any(k not in ("role", "content", "name", "function_call", "weight") for k in message):
            format_errors["message_unrecognized_key"] += 1
        if message.get("role", None) not in ("system", "user", "assistant", "function"):
            format_errors["unrecognized_role"] += 1
        content = message.get("content", None)
        function_call = message.get("function_call", None)
        if (not content and not function_call) or not isinstance(content, str):
            format_errors["missing_content"] += 1
    if not any(message.get("role", None) == "assistant" for message in messages):
        format_errors["example_missing_assistant_message"] += 1
if format_errors:
    print("Found errors:")
    for k, v in format_errors.items():
        print(f"{k}: {v}")
else:
    print("No errors found")

Count tokens against max_length

A model reads tokens, and the trainer cuts every example at max_length. In TRL's SFTConfig that is 1024 by default, and truncation keeps the start of the sequence, so the cut falls on the end, where the answer sits. TRL's memory page adds that a value too small discards a significant share of your tokens, and one too large can spike memory.

The snippet below is this lesson's own. It turns each example into text with the model's chat template, counts the tokens and tells you how many examples go over the limit.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")
limit = 1024  # SFTConfig's default max_length

too_long = 0
for ex in dataset:
    text = tokenizer.apply_chat_template(ex["messages"], tokenize=False)
    if len(tokenizer(text, add_special_tokens=False)["input_ids"]) > limit:
        too_long += 1
print(too_long, "of", len(dataset), "examples are over", limit, "tokens")

Hold out a test set first

OpenAI's accuracy guide recommends a hold-out set to catch overfitting, and its fine-tuning guide says to carve it out of the data you collected before you run a job. The script below, also this lesson's own, drops duplicate prompts first so one input cannot land on both sides, then splits.

train_test_split in Hugging Face Datasets shuffles by default and returns a train and a test split. A tenth of a small file leaves only a handful of test rows, so the script holds back a fifth. That fraction is the lesson's own pick.

Leave test.jsonl alone while you tune. Lesson 4 scores the base model and the tuned model on it.

import json
from datasets import Dataset

seen, rows = set(), []
for ex in dataset:
    key = json.dumps(ex["messages"][:-1], sort_keys=True)  # everything except the answer
    if key not in seen:
        seen.add(key)
        rows.append(ex)

splits = Dataset.from_list(rows).train_test_split(test_size=0.2, seed=42)
splits["train"].to_json("train.jsonl")
splits["test"].to_json("test.jsonl")

Common mistakes

• Training on tidy sentences when users send messy emails.
• Writing answers that need facts the base model lacks. Gekhman and colleagues found in 2024, with PaLM 2-S, that such examples are learned slowly and raise hallucinations.
• Leaving the longest examples above max_length, so the trainer cuts off their answers.
• Paraphrasing examples to grow the file before you split it, which puts near copies of test rows into training.
• Tuning the dataset against the test file until it stops being a test.

Try it yourself

• Write a handful of examples in all.jsonl and run the format check until it prints "No errors found".
• Add the length check and note the longest example.
• Grow the file to the starting size above, run the split script and read the rows in test.jsonl.
• Write down the score you would accept on test.jsonl before any training.

Next

Lesson 3 trains a LoRA adapter on train.jsonl with TRL, and shows what each setting costs in GPU memory.

Frequently asked questions

Sources

1. OpenAI: Supervised fine-tuning (read November 6, 2026)https://developers.openai.com/api/docs/guides/supervised-fine-tuning
2. OpenAI: Optimizing LLM Accuracy (read November 6, 2026)https://developers.openai.com/api/docs/guides/optimizing-llm-accuracy
3. OpenAI Cookbook: Data preparation and analysis for chat model fine-tuning (read November 6, 2026)https://developers.openai.com/cookbook/examples/chat_finetuning_data_prep
4. Hugging Face TRL: Dataset formats and types (read November 6, 2026)https://huggingface.co/docs/trl/dataset_formats
5. Hugging Face TRL: SFT Trainer (read November 6, 2026)https://huggingface.co/docs/trl/sft_trainer
6. Hugging Face TRL: Reducing memory usage (read November 6, 2026)https://huggingface.co/docs/trl/reducing_memory_usage
7. Hugging Face Datasets: Load (read November 6, 2026)https://huggingface.co/docs/datasets/loading
8. Hugging Face Datasets: Process (read November 6, 2026)https://huggingface.co/docs/datasets/process
9. Hugging Face Papers: LIMA: Less Is More for Alignment (read November 6, 2026)https://huggingface.co/papers/2305.11206
10. Hugging Face Papers: Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? (read November 6, 2026)https://huggingface.co/papers/2405.05904