What are LLM evals? A first test set from real failures

Before you start

You need Python 3.10 or later, the anthropic package (pip install anthropic) and an API key in ANTHROPIC_API_KEY. The code below is Anthropic's Building Evals cookbook, which its history shows was last changed on 24 October 2026, when its model became claude-opus-5-5. The eval logic is plain Python, so another provider's SDK works the same way. If you have never called the API, the Claude API quickstart gets you a first reply.

Run a three-row eval

The smallest eval worth running asks a model how many legs an animal has and wants the number alone, so a string comparison can grade it. Two rows are easy. The third describes a fox that lost a leg, grew it back and then grew an extra one. Its golden answer is 5, so it checks whether the model follows the statement in front of it rather than what it knows about foxes.

A golden answer is the output a perfect reply would give. Run the script, read the printed outputs next to their golden answers, and only then look at the score line.

from anthropic import Anthropic

client = Anthropic()
MODEL_NAME = "claude-opus-5-5"


# Define our input prompt template for the task.
def build_input_prompt(animal_statement):
    user_content = f"""You will be provided a statement about an animal and your job is to determine how many legs that animal has.

    Here is the animal statement.
    <animal_statement>{animal_statement}</animal_statement>

    How many legs does the animal have? Return just the number of legs as an integer and nothing else."""

    messages = [{"role": "user", "content": user_content}]
    return messages


# Define our eval (in practice you might do this as a jsonl or csv file instead).
eval = [
    {"animal_statement": "The animal is a human.", "golden_answer": "2"},
    {"animal_statement": "The animal is a snake.", "golden_answer": "0"},
    {
        "animal_statement": "The fox lost a leg, but then magically grew back the leg he lost and a mysterious extra leg on top of that.",
        "golden_answer": "5",
    },
]


# Get completions for each input.
# Define our get_completion function. The answer is a single integer, so we ask for low effort;
# max_tokens leaves headroom because any thinking the model does counts toward it.
def get_completion(messages):
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=1024,
        output_config={"effort": "low"},
        messages=messages,
    )
    # The response can start with a thinking block, so read the text block rather than content[0].
    return next((block.text for block in response.content if block.type == "text"), "").strip()


# Get completions for each question in the eval.
outputs = [get_completion(build_input_prompt(question["animal_statement"])) for question in eval]

# Let's take a quick look at our outputs
for output, question in zip(outputs, eval, strict=False):
    print(
        f"Animal Statement: {question['animal_statement']}\nGolden Answer: {question['golden_answer']}\nOutput: {output}\n"
    )


# Check our completions against the golden answers.
# Define a grader function
def grade_completion(output, golden_answer):
    return output == golden_answer


# Run the grader function on our outputs and print the score.
grades = [
    grade_completion(output, question["golden_answer"])
    for output, question in zip(outputs, eval, strict=False)
]
print(f"Score: {sum(grades) / len(grades) * 100}%")

1. Write the prompt as a template with a slot for the input that changes, here animal_statement.
2. List the rows, each an input and its golden answer. A JSONL or CSV file works as well as a list.
3. Run every row through the model and keep the raw outputs.
4. Grade each output against its golden answer and print the share that passed.
5. Change one thing, such as a line of the prompt or the model name, run every row again and compare the two scores.

Copied from Anthropic's Building Evals cookbook on GitHub. The cookbook spells "statement" as "statment" twice in its prompt, once in the closing tag. We fixed both so the tags match.

What the score tells you, and what it hides

The cookbook names four parts of an eval: the input prompt, the output the model gives, the golden answer and a score. You write the rows and golden answers once. Grading costs you on every run, and you will rerun the eval each time you touch the prompt or switch models. That is why the cookbook puts cheap, fast grading at the center of the design.

Exact match is strict. If the model replies "Two" or "2 legs" to the human row, the grader marks a right answer as wrong. Anthropic lists this first among code graders' weaknesses: they are brittle to valid variations that don't match expected patterns exactly. The prompt's last line, "Return just the number of legs as an integer and nothing else", is what makes a string comparison fair here. Without it, you would need a looser check or a model to grade.

A score from three rows also moves a third of the way with each row. Model outputs vary between runs, so Anthropic calls each attempt a trial and runs several trials per task to get steadier numbers. Read the failed rows before you trust a change in the percentage.

Read traces before you write tests

The legs rows were made up. Your app's rows should come from its own failures, and you find those by reading traces. A trace, which Anthropic also calls a transcript, is the full record of one run: the input, every model and tool call, the intermediate results and the final output. Error analysis is reading a sample of traces and writing down what went wrong in each.

OpenAI's evaluation flywheel cookbook splits it into two passes. In open coding you read failing traces and give each error a short label in your own words, with no fixed list yet. Its example is an apartment leasing assistant, where one label reads "bot suggested a tour time that wasn't available." In axial coding you group those labels into categories, such as "Tour scheduling/rescheduling issue" and "Formatting error with output." Count the traces in each category, and the biggest one is your first test.

Pick the sample for variety, not for how common a problem is. The error-discovery skill that Hamel Husain and Shreya Shankar publish builds its first review sample mostly from cluster representatives, with about 30 to 40% random picks to cover what the clusters miss.

Turn failures into test rows

Anthropic's advice is to start with the checks you already run by hand before each release and the tasks your users try most. If the app is live, your bug tracker and support queue hold rows ready to copy. Early changes to a young app tend to have big effects, so a small set is enough to see them. "Evals get harder to build the longer you wait," the post adds, because later you are reverse-engineering what success meant from a live system.

Write each row so that two domain experts would reach the same pass or fail verdict on their own. A row they would argue about adds noise to the score, not signal. Where you can, add a reference solution, an output that passes every check, to prove the row can be passed at all.

Then test the other direction. Anthropic built its web search evals for Claude.ai with questions where the model should search, like the weather, and questions it should answer from what it knows, like "who founded Apple?" A set that only checks that a behavior happens rewards a prompt that triggers it everywhere. OpenAI adds edge cases to the mix: typos, a one-word request such as "returns", several questions in one message, or a user prompt that conflicts with the system prompt.

Pick a code grader or a model grader

A grader is the logic that scores an output. Anthropic recommends deterministic graders where possible, model graders where necessary and human graders sparingly, to check the others. Code is fast, cheap and gives the same answer every run, but it breaks on valid wording it did not expect. A model grader handles nuance and free text, yet it is non-deterministic, costs more per run and needs checking against people before you trust it.

The write-code-eval skill treats each failure category like a unit test and pairs it with a check. Some failures that look subjective still reduce to code. The judge skill in the same repository gives an example. An interviewing coach should ask about one specific past event. When it asks about typical behavior instead, the skill says a keyword check for words like "usually," "typical" and "normally" could work quite well. That is plain string matching. Tone, faithfulness to a document and whether an answer is complete need interpretation, and lesson 2 writes a judge for them. For a RAG pipeline, the RAG evaluation lesson compares code, human and model grading in one table.

Greg Brockman on evals, and Hamel Husain and Shreya Shankar on reading traces first

Greg Brockman, OpenAI's president and co-founder by his X bio, posted one line on X on 9 December 2023: "evals are surprisingly often all you need." OpenAI's evals repository still shows the post near the top of its README. Before you swap in a bigger model or reach for fine-tuning, a few rows like the legs eval tell you whether the change you are about to make helps at all.

Hamel Husain and Shreya Shankar, who teach an AI evals course together, turned their method into agent skills on GitHub. Their README calls error discovery the most important one. Point your coding agent at a file of traces, such as traces.jsonl, and the skill builds a review app, picks a varied sample and sorts your free-text notes into failure modes. The open coding above is the same work done by hand.

The video below, from Hamel Husain's channel, is the open and axial coding walkthrough that the course's recipe-chatbot repository lists for its error analysis homework.

A screenshot of X's own embed of the post, taken on 6 November 2026, with the time in UTC. Select it to open the post on X.

Common mistakes

• Writing test rows from imagination before reading a single trace. You end up testing what you guessed, not what breaks.
• Grading free text with exact match. "Two" fails against "2", so ask for a fixed format or use a looser check.
• Keeping rows that two experts would grade differently. Their disagreement shows up as noise in every score.
• Testing only that a behavior happens. Without rows where it must not happen, an over-eager prompt still passes every row.
• Reaching for a model grader first. A keyword or schema check costs less and returns the same answer on every run.
• Calling a quick look at a few replies an eval. OpenAI lists "it seems like it's working" among its anti-patterns.

Try it yourself

• Run the three legs rows, then add a fourth with a statement and a golden answer of your own.
• Delete the prompt's last line, the one that asks for the number alone, and run again. For each failed row, decide whether the model or the grader is wrong.
• Take 30 traces from your own app, write one short label per failure, then group the labels into three to five categories.
• For your biggest category, write one row where the failure should be caught and one row where the behavior is correct.
• Pick a category that needs judgment, such as tone, and label at least 20 passing and 20 failing traces by hand. The judge you write next starts from those labels, and checking it reliably takes more, as the next lesson explains.

Frequently asked questions

Sources

1. Anthropic: Demystifying evals for AI agents (9 January 2026) (read November 6, 2026)https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
2. Anthropic Cookbook on GitHub: Building Evals (read November 6, 2026)https://github.com/anthropics/claude-cookbooks/blob/main/misc/building_evals.ipynb
3. GitHub: building_evals.ipynb commit history (read November 6, 2026)https://github.com/anthropics/claude-cookbooks/commits/main/misc/building_evals.ipynb
4. OpenAI Docs: Evaluation best practices (read November 6, 2026)https://developers.openai.com/api/docs/guides/evaluation-best-practices
5. OpenAI Docs: Deprecations (Evals platform) (read November 6, 2026)https://developers.openai.com/api/docs/deprecations
6. OpenAI Cookbook: Building resilient prompts using an evaluation flywheel (read November 6, 2026)https://developers.openai.com/cookbook/examples/evaluation/building_resilient_prompts_using_an_evaluation_flywheel
7. GitHub: ai-evals-course/evals-skills README (read November 6, 2026)https://github.com/ai-evals-course/evals-skills
8. evals-skills on GitHub: error-discovery skill (read November 6, 2026)https://github.com/ai-evals-course/evals-skills/blob/main/skills/error-discovery/SKILL.md
9. evals-skills on GitHub: write-code-eval skill (read November 6, 2026)https://github.com/ai-evals-course/evals-skills/blob/main/skills/write-code-eval/SKILL.md
10. evals-skills on GitHub: write-judge-prompt skill (read November 6, 2026)https://github.com/ai-evals-course/evals-skills/blob/main/skills/write-judge-prompt/SKILL.md
11. GitHub: ai-evals-course/recipe-chatbot README (read November 6, 2026)https://github.com/ai-evals-course/recipe-chatbot
12. GitHub: openai/evals README (read November 6, 2026)https://github.com/openai/evals
13. Greg Brockman on X: evals are surprisingly often all you need (9 December 2023) (read November 6, 2026)https://x.com/gdb/status/1733553161884127435
14. YouTube, Hamel Husain: Open & Axial Coding - Isaac & Hamel Walkthrough (read November 6, 2026)https://www.youtube.com/watch?v=AKg27L4E0M8
15. PyPI: anthropic (requires Python 3.10 or later) (read November 6, 2026)https://pypi.org/project/anthropic/