# What are LLM evals? A first test set from real failures

> An LLM eval is a test for an AI system: an input, a way to grade the output and a score you can track. Start by reading real traces to see how your app fails. Turn 20 to 50 of those failures into test rows, then grade each one with code where a rule can decide and with a model where it takes judgment.

Mindket editors · Published 2026-11-06 · Updated 2026-11-06 · https://mindket.com/learn/ai-evals/what-are-llm-evals

## In short

- Run a three-row eval that prints one score you can compare after every prompt change.
- Sort a sample of bad outputs into named failure categories before writing a single test.
- Write rows two experts would grade alike, including rows where the behavior must not happen.
- Decide per failure whether a few lines of code can grade it or a judging model has to.

## Before you start

You need Python 3.10 or later, the `anthropic` package (`pip install anthropic`) and an API key in `ANTHROPIC_API_KEY`. The code below is Anthropic's [Building Evals cookbook](https://github.com/anthropics/claude-cookbooks/blob/main/misc/building_evals.ipynb), which [its history](https://github.com/anthropics/claude-cookbooks/commits/main/misc/building_evals.ipynb) shows was last changed on 24 October 2026, when its model became `claude-opus-5-5`. The eval logic is plain Python, so another provider's SDK works the same way. If you have never called the API, the [Claude API quickstart](https://mindket.com/learn/claude-api/claude-api-quickstart) gets you a first reply.

## Run a three-row eval

The smallest eval worth running asks a model how many legs an animal has and wants the number alone, so a string comparison can grade it. Two rows are easy. The third describes a fox that lost a leg, grew it back and then grew an extra one. Its golden answer is 5, so it checks whether the model follows the statement in front of it rather than what it knows about foxes.

A golden answer is the output a perfect reply would give. Run the script, read the printed outputs next to their golden answers, and only then look at the score line.

````python
from anthropic import Anthropic

client = Anthropic()
MODEL_NAME = "claude-opus-5-5"


# Define our input prompt template for the task.
def build_input_prompt(animal_statement):
    user_content = f"""You will be provided a statement about an animal and your job is to determine how many legs that animal has.

    Here is the animal statement.
    <animal_statement>{animal_statement}</animal_statement>

    How many legs does the animal have? Return just the number of legs as an integer and nothing else."""

    messages = [{"role": "user", "content": user_content}]
    return messages


# Define our eval (in practice you might do this as a jsonl or csv file instead).
eval = [
    {"animal_statement": "The animal is a human.", "golden_answer": "2"},
    {"animal_statement": "The animal is a snake.", "golden_answer": "0"},
    {
        "animal_statement": "The fox lost a leg, but then magically grew back the leg he lost and a mysterious extra leg on top of that.",
        "golden_answer": "5",
    },
]


# Get completions for each input.
# Define our get_completion function. The answer is a single integer, so we ask for low effort;
# max_tokens leaves headroom because any thinking the model does counts toward it.
def get_completion(messages):
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=1024,
        output_config={"effort": "low"},
        messages=messages,
    )
    # The response can start with a thinking block, so read the text block rather than content[0].
    return next((block.text for block in response.content if block.type == "text"), "").strip()


# Get completions for each question in the eval.
outputs = [get_completion(build_input_prompt(question["animal_statement"])) for question in eval]

# Let's take a quick look at our outputs
for output, question in zip(outputs, eval, strict=False):
    print(
        f"Animal Statement: {question['animal_statement']}\nGolden Answer: {question['golden_answer']}\nOutput: {output}\n"
    )


# Check our completions against the golden answers.
# Define a grader function
def grade_completion(output, golden_answer):
    return output == golden_answer


# Run the grader function on our outputs and print the score.
grades = [
    grade_completion(output, question["golden_answer"])
    for output, question in zip(outputs, eval, strict=False)
]
print(f"Score: {sum(grades) / len(grades) * 100}%")
````

1. Write the prompt as a template with a slot for the input that changes, here `animal_statement`.
2. List the rows, each an input and its golden answer. A JSONL or CSV file works as well as a list.
3. Run every row through the model and keep the raw outputs.
4. Grade each output against its golden answer and print the share that passed.
5. Change one thing, such as a line of the prompt or the model name, run every row again and compare the two scores.

> Copied from Anthropic's Building Evals cookbook on GitHub. The cookbook spells "statement" as "statment" twice in its prompt, once in the closing tag. We fixed both so the tags match.

## What the score tells you, and what it hides

The cookbook names four parts of an eval: the input prompt, the output the model gives, the golden answer and a score. You write the rows and golden answers once. Grading costs you on every run, and you will rerun the eval each time you touch the prompt or switch models. That is why the cookbook puts cheap, fast grading at the center of the design.

Exact match is strict. If the model replies "Two" or "2 legs" to the human row, the grader marks a right answer as wrong. Anthropic lists this first among code graders' weaknesses: they are brittle to valid variations that don't match expected patterns exactly. The prompt's last line, "Return just the number of legs as an integer and nothing else", is what makes a string comparison fair here. Without it, you would need a looser check or a model to grade.

A score from three rows also moves a third of the way with each row. Model outputs vary between runs, so Anthropic calls each attempt a trial and runs several trials per task to get steadier numbers. Read the failed rows before you trust a change in the percentage.

## Read traces before you write tests

The legs rows were made up. Your app's rows should come from its own failures, and you find those by reading traces. A trace, which Anthropic also calls a transcript, is the full record of one run: the input, every model and tool call, the intermediate results and the final output. Error analysis is reading a sample of traces and writing down what went wrong in each.

[OpenAI's evaluation flywheel cookbook](https://developers.openai.com/cookbook/examples/evaluation/building_resilient_prompts_using_an_evaluation_flywheel) splits it into two passes. In open coding you read failing traces and give each error a short label in your own words, with no fixed list yet. Its example is an apartment leasing assistant, where one label reads "bot suggested a tour time that wasn't available." In axial coding you group those labels into categories, such as "Tour scheduling/rescheduling issue" and "Formatting error with output." Count the traces in each category, and the biggest one is your first test.

Pick the sample for variety, not for how common a problem is. The [error-discovery skill](https://github.com/ai-evals-course/evals-skills/blob/main/skills/error-discovery/SKILL.md) that Hamel Husain and Shreya Shankar publish builds its first review sample mostly from cluster representatives, with about 30 to 40% random picks to cover what the clusters miss.

**How much to read and write before the first test**

- **20-50** simple tasks drawn from real failures: the start Anthropic recommends for an agent eval (Source: [Anthropic: Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents), read 2026-11-06)
- **~50** failing traces to label in open coding before you group the labels (Source: [OpenAI Cookbook: Building resilient prompts using an evaluation flywheel](https://developers.openai.com/cookbook/examples/evaluation/building_resilient_prompts_using_an_evaluation_flywheel), read 2026-11-06)
- **15-25** traces in a first review sample when a dataset holds more than 50 (Source: [evals-skills on GitHub: error-discovery skill](https://github.com/ai-evals-course/evals-skills/blob/main/skills/error-discovery/SKILL.md), read 2026-11-06)

## Turn failures into test rows

Anthropic's advice is to start with the checks you already run by hand before each release and the tasks your users try most. If the app is live, your bug tracker and support queue hold rows ready to copy. Early changes to a young app tend to have big effects, so a small set is enough to see them. "Evals get harder to build the longer you wait," the post adds, because later you are reverse-engineering what success meant from a live system.

Write each row so that two domain experts would reach the same pass or fail verdict on their own. A row they would argue about adds noise to the score, not signal. Where you can, add a reference solution, an output that passes every check, to prove the row can be passed at all.

Then test the other direction. Anthropic built its web search evals for Claude.ai with questions where the model should search, like the weather, and questions it should answer from what it knows, like "who founded Apple?" A set that only checks that a behavior happens rewards a prompt that triggers it everywhere. OpenAI adds edge cases to the mix: typos, a one-word request such as "returns", several questions in one message, or a user prompt that conflicts with the system prompt.

| Where a row comes from | An example row |
| --- | --- |
| A check you run by hand before each release | The order status reply includes the tracking number |
| A bug report or support ticket | The user who was offered a tour time that was already booked |
| The biggest category from your trace review | Five more traces that fail the same way, from different users |
| The opposite case | A question the app should answer without calling a tool |
| An edge case | The one-word message "returns" |

## Pick a code grader or a model grader

A grader is the logic that scores an output. Anthropic recommends deterministic graders where possible, model graders where necessary and human graders sparingly, to check the others. Code is fast, cheap and gives the same answer every run, but it breaks on valid wording it did not expect. A model grader handles nuance and free text, yet it is non-deterministic, costs more per run and needs checking against people before you trust it.

The [write-code-eval skill](https://github.com/ai-evals-course/evals-skills/blob/main/skills/write-code-eval/SKILL.md) treats each failure category like a unit test and pairs it with a check. Some failures that look subjective still reduce to code. The [judge skill](https://github.com/ai-evals-course/evals-skills/blob/main/skills/write-judge-prompt/SKILL.md) in the same repository gives an example. An interviewing coach should ask about one specific past event. When it asks about typical behavior instead, the skill says a keyword check for words like "usually," "typical" and "normally" could work quite well. That is plain string matching. Tone, faithfulness to a document and whether an answer is complete need interpretation, and [lesson 2](https://mindket.com/learn/ai-evals/llm-as-a-judge) writes a judge for them. For a RAG pipeline, the [RAG evaluation lesson](https://mindket.com/learn/rag/evaluate-rag) compares code, human and model grading in one table.

| Failure category | A check code can run |
| --- | --- |
| Invalid output structure | Parse the output and check the required fields |
| Missing or forbidden text | Match a string or a pattern |
| A citation that is not in the retrieved documents | Compare the cited IDs with the retrieved IDs |
| A bad tool call | Check the arguments against the tool schema |
| A wrong value | Compare the output with a reference value |

## Greg Brockman on evals, and Hamel Husain and Shreya Shankar on reading traces first

Greg Brockman, OpenAI's president and co-founder by his X bio, posted one line [on X](https://x.com/gdb/status/1733553161884127435) on 9 December 2023: "evals are surprisingly often all you need." OpenAI's [evals repository](https://github.com/openai/evals) still shows the post near the top of its README. Before you swap in a bigger model or reach for fine-tuning, a few rows like the legs eval tell you whether the change you are about to make helps at all.

Hamel Husain and Shreya Shankar, who teach an AI evals course together, turned their method into [agent skills on GitHub](https://github.com/ai-evals-course/evals-skills). Their README calls error discovery the most important one. Point your coding agent at a file of traces, such as `traces.jsonl`, and the skill builds a review app, picks a varied sample and sorts your free-text notes into failure modes. The open coding above is the same work done by hand.

The video below, from Hamel Husain's channel, is the open and axial coding walkthrough that the course's [recipe-chatbot repository](https://github.com/ai-evals-course/recipe-chatbot) lists for its error analysis homework.

> You should only write evals after doing this step.
>
> Hamel Husain and Shreya Shankar, Teachers of the AI Evals course, in the README of their evals-skills repository, [GitHub](https://github.com/ai-evals-course/evals-skills) (read 2026-11-06)

[![Greg Brockman (@gdb) on X, 9 December 2023: evals are surprisingly often all you need](https://mindket.com/blog/shots/x-gdb-evals-all-you-need.webp?v=dd33a68a62af)](https://x.com/gdb/status/1733553161884127435)

Greg Brockman on X, 9 December 2023

[Open & Axial Coding - Isaac & Hamel Walkthrough](https://www.youtube.com/watch?v=AKg27L4E0M8)

Hamel Husain's YouTube channel: an open and axial coding walkthrough for the recipe-chatbot homework.

> A screenshot of X's own embed of the post, taken on 6 November 2026, with the time in UTC. Select it to open the post on X.

## Common mistakes

- Writing test rows from imagination before reading a single trace. You end up testing what you guessed, not what breaks.
- Grading free text with exact match. "Two" fails against "2", so ask for a fixed format or use a looser check.
- Keeping rows that two experts would grade differently. Their disagreement shows up as noise in every score.
- Testing only that a behavior happens. Without rows where it must not happen, an over-eager prompt still passes every row.
- Reaching for a model grader first. A keyword or schema check costs less and returns the same answer on every run.
- Calling a quick look at a few replies an eval. OpenAI lists "it seems like it's working" among its anti-patterns.

## Try it yourself

- Run the three legs rows, then add a fourth with a statement and a golden answer of your own.
- Delete the prompt's last line, the one that asks for the number alone, and run again. For each failed row, decide whether the model or the grader is wrong.
- Take 30 traces from your own app, write one short label per failure, then group the labels into three to five categories.
- For your biggest category, write one row where the failure should be caught and one row where the behavior is correct.
- Pick a category that needs judgment, such as tone, and label at least 20 passing and 20 failing traces by hand. The judge you write next starts from those labels, and checking it reliably takes more, as the next lesson explains.

## Check yourself

1. An exact-match grader expects "2". Which correct reply does it mark as a failure?

   A. "2"
   B. "2 legs"
   C. Neither, since exact match accepts both

   Answer: B. Exact match compares the whole string, so one extra word fails a right answer, which is why the prompt asks for the number alone.

2. You have 300 production traces and no tests yet. What do you do first?

   A. Write a 1 to 10 quality judge and run it on all 300
   B. Pick a public benchmark such as MMLU
   C. Read a sample of the traces and label each failure
   D. Fine-tune the model on the traces

   Answer: C. Error analysis finds the failure modes your app has, and those decide which tests are worth writing.

3. Which test row is written well?

   A. A row copied from a public benchmark
   B. A row whose rubric says "the answer should be good"
   C. A row only its author knows how to grade
   D. A row two domain experts would pass or fail the same way

   Answer: D. Anthropic's test for a good task is that two experts reach the same verdict on their own, so ambiguity does not turn into noise.

4. Which failure should you check with code rather than a model?

   A. The reply is missing a required JSON field
   B. The email sounds too casual for a luxury buyer
   C. The answer drifts from the retrieved document

   Answer: A. Parsing the output and checking for required fields is a rule code can apply, while tone and faithfulness need interpretation.

## Frequently asked questions

### What are LLM evals?

They are tests for an app built on a language model. Each pairs an input with grading logic that turns the reply into a number, so two versions of a prompt can be compared. Anthropic's definition covers agents too, where the input is a task and the grade can check what changed in the environment.

### Are evals the same as benchmarks?

No. OpenAI notes that "evals" can mean industry benchmarks such as MMLU, standard scores such as ROUGE, or tests you write for your own app. A benchmark compares models on someone else's tasks, while your eval checks your prompt, tools and data.

### How many test rows do I need to start?

Fewer than most teams expect. Anthropic says early changes have large effects that small samples can show, and that mature agents need larger, harder sets to detect smaller effects. Add a row each time a new failure turns up.

### Do I need an eval framework?

No. The eval in this lesson is a list of dicts and a loop. A framework adds caching, reports and runs on every pull request, which [lesson 4](https://mindket.com/learn/ai-evals/evals-in-ci) sets up.

### Can I still use the OpenAI Evals API?

Only to read what is already there. OpenAI's deprecations page says existing evals became read-only on 31 October 2026 and that the Evals dashboard and API are scheduled to shut down on 30 November 2026, as of 6 November 2026. It points to promptfoo as the way to move.

## Sources

- [Anthropic: Demystifying evals for AI agents (9 January 2026)](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) (read 2026-11-06)
- [Anthropic Cookbook on GitHub: Building Evals](https://github.com/anthropics/claude-cookbooks/blob/main/misc/building_evals.ipynb) (read 2026-11-06)
- [GitHub: building_evals.ipynb commit history](https://github.com/anthropics/claude-cookbooks/commits/main/misc/building_evals.ipynb) (read 2026-11-06)
- [OpenAI Docs: Evaluation best practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices) (read 2026-11-06)
- [OpenAI Docs: Deprecations (Evals platform)](https://developers.openai.com/api/docs/deprecations) (read 2026-11-06)
- [OpenAI Cookbook: Building resilient prompts using an evaluation flywheel](https://developers.openai.com/cookbook/examples/evaluation/building_resilient_prompts_using_an_evaluation_flywheel) (read 2026-11-06)
- [GitHub: ai-evals-course/evals-skills README](https://github.com/ai-evals-course/evals-skills) (read 2026-11-06)
- [evals-skills on GitHub: error-discovery skill](https://github.com/ai-evals-course/evals-skills/blob/main/skills/error-discovery/SKILL.md) (read 2026-11-06)
- [evals-skills on GitHub: write-code-eval skill](https://github.com/ai-evals-course/evals-skills/blob/main/skills/write-code-eval/SKILL.md) (read 2026-11-06)
- [evals-skills on GitHub: write-judge-prompt skill](https://github.com/ai-evals-course/evals-skills/blob/main/skills/write-judge-prompt/SKILL.md) (read 2026-11-06)
- [GitHub: ai-evals-course/recipe-chatbot README](https://github.com/ai-evals-course/recipe-chatbot) (read 2026-11-06)
- [GitHub: openai/evals README](https://github.com/openai/evals) (read 2026-11-06)
- [Greg Brockman on X: evals are surprisingly often all you need (9 December 2023)](https://x.com/gdb/status/1733553161884127435) (read 2026-11-06)
- [YouTube, Hamel Husain: Open & Axial Coding - Isaac & Hamel Walkthrough](https://www.youtube.com/watch?v=AKg27L4E0M8) (read 2026-11-06)
- [PyPI: anthropic (requires Python 3.10 or later)](https://pypi.org/project/anthropic/) (read 2026-11-06)

[Browse agents](https://mindket.com/agents)