How to evaluate AI agents: end states, trials and traces

Before you start

You need an agent that calls tools in a loop, such as the one from the Claude API agent lesson, and the eval basics of lesson 1. The worked code uses Inspect, an open-source eval framework from the UK AI Security Institute and Meridian Labs. Install it with pip install inspect-ai. Its docs list over 200 ready-made evals you can run before writing your own.

A version note. The pass_k reducer this lesson uses arrived in Inspect 0.3.224 on 20 May 2026, by its changelog. On 6 November 2026 the newest release on PyPI was 0.3.287. Pin the version you use, so two runs a month apart count the same way.

The agent says the flight is booked. Is it?

A chat model's answer is the whole result. An agent's last message is only its account of what it did. Anthropic's post on agent evals names the parts. A task is one test with inputs and success criteria, and a trial is one attempt at it. The transcript, also called the trace or trajectory, records every output, tool call and reasoning step of a trial. The outcome is the state of the environment when the trial ends.

Its example is a flight-booking agent that ends with "Your flight has been booked". The outcome is whether a reservation exists in the environment's SQL database. Grade the row. The message can say anything.

τ-bench, a benchmark Sierra's researchers published in June 2024, grades the same way. Its paper compares the database state at the end of a conversation with an annotated goal state, so any route that reaches the goal passes.

That freedom matters more than it looks. Anthropic reports that Opus 4.5 solved a τ2-bench flight problem by finding a loophole in the policy it was given. The eval as written marked it a failure, yet Anthropic says it came up with a better solution for the user.

Write tasks with an end state code can check

Lesson 1 built a first test set from 20 to 50 real failures: rows two domain experts would grade alike, a reference solution that passes, and tests in both directions. Agent tasks follow the same rules, and a reference solution for each task also proves the graders are wired up correctly. What changes is the success criterion: an end state code can check, such as a database row, a file or a ticket status.

The YAML below is Anthropic's example of a support agent handling a refund for a frustrated customer. The post calls it theoretical: it shows multiple grader types at once for illustration, and says real conversational evals typically use model graders. One task carries four graders: an LLM rubric for tone, a state check on the ticket and the refund, the tool calls it requires and a limit of ten turns. Turns, tokens and latency are tracked without being graded.

graders:
  - type: llm_rubric
    rubric: prompts/support_quality.md
    assertions:
      - "Agent showed empathy for customer's frustration"
      - "Resolution was clearly explained"
      - "Agent's response grounded in fetch_policy tool results"
  - type: state_check
    expect:
      tickets: {status: resolved}
      refunds: {status: processed}
  - type: tool_calls
    required:
      - {tool: verify_identity}
      - {tool: process_refund, params: {amount: "<=100"}}
      - {tool: send_confirmation}
  - type: transcript
    max_turns: 10
tracked_metrics:
  - type: transcript
    metrics:
      - n_turns
      - n_toolcalls
      - n_total_tokens
  - type: latency
    metrics:
      - time_to_first_token
      - output_tokens_per_sec
      - time_to_last_token

1. Collect failures from bug reports, support tickets and your own manual runs, and turn each into a task with its input.
2. Write the end state each task should leave as a check code can run: a database row, a file or a ticket status.
3. Ask a colleague to grade a few outcomes without seeing your verdicts. Rewrite each task where you disagree.
4. Write a reference solution for each task and confirm it passes every grader.
5. Add tasks where the agent should not act, such as a question it can answer without a tool.

Run each task several times: pass@k and pass^k

Model outputs vary between runs, so one trial per task says little. With k trials, pass@k is the chance that at least one succeeds, and pass^k the chance that all k do. At k=1 they are the same number. They drift apart as k grows.

Anthropic works an example. An agent that succeeds on 75% of trials passes all three of three trials about 42% of the time, 0.75 cubed. Report pass@k for a tool where one success matters, since proposing many solutions is fine as long as one works. Report pass^k for an agent customers talk to, because they expect it to work every time.

τ-bench introduced pass^k, and its README shows how fast it falls. In the retail domain, claude-3-5-sonnet-20241022 with tool calling passed 69.2% of tasks on one try and 46.2% when all four tries had to pass. The README now warns that its tasks are outdated and points to τ³-bench. Read the chart as the shape of the drop, not as a current score.

Inspect computes both. The task below is the CTF example from its docs. A react() agent with a shell and a to-do tool hunts for a flag inside a Docker sandbox, and includes() passes the trial if the flag appears in its answer. The epochs line, which this lesson adds, runs each sample five times and reports the mean, pass_at_5 and pass_k_5, reducers Inspect's metrics page lists.

from inspect_ai import Epochs, Task, task
from inspect_ai.agent import react
from inspect_ai.dataset import json_dataset
from inspect_ai.scorer import includes
from inspect_ai.tool import bash, todo_write

@task
def ctf():
    return Task(
        dataset=json_dataset("challenges.json"),
        solver=react(
            prompt=(
                "You are a Capture the Flag player. "
                "Explore the system and find the flag."
            ),
            tools=[bash(), todo_write()],
            attempts=3,
        ),
        scorer=includes(),
        sandbox="docker",
        # this lesson's line: five trials per sample, summed up three ways
        epochs=Epochs(5, ["mean", "pass_at_5", "pass_k_5"]),
    )

• attempts=3 lets the agent submit again inside one trial. Epochs are separate trials, each from a fresh start.
• Run it with inspect eval ctf.py --model and a model name, then open the transcripts with inspect view.
• Start every trial from a clean environment. In some internal evals, Anthropic saw Claude gain an unfair advantage by reading the git history of earlier trials.
• Trials that fail for one shared cause, such as too little memory on the machine, are not independent. Anthropic says the results then stop measuring the agent.

Check the steps that matter in the trace

The outcome says whether the agent succeeded. The trace shows how, and parts of the how count on their own: whether it checked who the customer was before refunding, or called a tool it should never touch. OpenAI calls scoring this trace grading, structured scores or labels on the end-to-end log of decisions, tool calls and reasoning steps.

OpenAI's agent evals page lists the questions it answers. Did the agent pick the right tool? Did a handoff happen when it should have? Did the workflow break an instruction or a safety policy?

Grade too many steps and the eval turns brittle. Anthropic found that checking a fixed sequence of tool calls is too rigid, because agents find valid approaches the designers did not foresee. Pin the order only where a rule demands it, such as identity before refund, and give partial credit on tasks with several parts. A support agent that finds the problem and verifies the customer but fails the refund did better than one that fails at once.

promptfoo checks traces with trajectory assertions. The config below, from its tracing docs, turns on the built-in OpenTelemetry receiver, then checks one test's trace for the tool it used, the arguments it passed and the order of two calls. Its last check asks an LLM judge whether the run reached the goal. By default tool-sequence allows extra steps between the listed ones, and mode: exact forbids them.

# promptfooconfig.yaml (providers and prompts left out)
tracing:
  enabled: true # Required to send OTLP telemetry
  otlp:
    http: {} # Starts the built-in OTLP receiver with the defaults

tests:
  - vars:
      order_id: '123'
    assert:
      - type: trajectory:tool-used
        value: search_orders

      - type: trajectory:tool-args-match
        value:
          name: search_orders
          args:
            order_id: '{{ order_id }}'

      - type: trajectory:tool-sequence
        value:
          steps:
            - search_orders
            - compose_reply

      - type: trajectory:goal-success
        value: 'Determine the shipping status for order {{ order_id }} and tell the user whether it has shipped'
        provider: openai:gpt-6-luna

• Tracing works with the providers promptfoo instruments, OpenAI and Anthropic among them, and with any app that sends OpenTelemetry spans naming its tools.
• For an LLM judge reading a transcript, everything in lesson 2 applies, including the two habits it takes from Anthropic: a way out such as "Unknown", and one judge per dimension.

Read the transcripts before you trust a score

Some low scores are the grader's fault. Opus 4.5 first scored 42% on CORE-Bench. An Anthropic researcher then found grading so rigid it rejected "96.12" when it expected "96.124991…", ambiguous task specs and random tasks no one could reproduce exactly. With those fixed and a less constrained scaffold, the score rose to 95%.

An audit of Terminal-Bench turned up a task that asked for a script without naming its path, while the tests assumed one. The agent could fail through no fault of its own. With frontier models, Anthropic says, a task that fails every trial, 0% pass@100, is most often a broken task and not an incapable agent.

Anthropic's team built tooling to view eval transcripts and reads them regularly. A failed transcript shows whether the agent made a mistake or the grader rejected a valid answer. Passing runs deserve a look too, since a grader that waves through a wrong answer never appears in the failure list.

Then split the suite. Capability evals start at a low pass rate and give you something to climb. Regression evals should pass almost every time, and a drop means something broke. A capability task that passes reliably can move into the regression suite, the set worth running on every change, which the next lesson wires into CI.

Karthik Narasimhan on reliability, Hamel Husain on reading traces

Karthik Narasimhan, a Princeton professor by his X bio and an author of the τ-bench paper, announced the benchmark on X on 20 June 2024. He pitched it as a test of agents' performance and reliability with dynamic user and tool interaction, and pass^k is the reliability part. Anthropic's post on agent evals reached X on 9 January 2026 with a line most agent teams know: the capabilities that make agents useful also make them more difficult to evaluate.

Hamel Husain, who leads Parlance Labs, an AI engineering group, put Anthropic's Claude skill that builds evals to the test in a video posted on 13 October 2026. He and Isaac Flath pointed it at 100 real conversations from an apartment leasing assistant. By the video's description, the skill found real problems in the data. It also had them pick an eval before they had looked at a single conversation, and by the end both of them failed it.

Screenshots of X's own embeds of the posts, taken on 6 November 2026, with the time in UTC. Select one to open the post on X.

Common mistakes

• Grading the agent's last message instead of the state it left behind.
• Running one trial per task and reading a single pass or fail as the level of the agent.
• Quoting pass@k for a support agent people depend on, where pass^k shows how often it lets them down.
• Checking the exact order of every tool call, which fails an agent that finds a valid shorter route.
• Letting trials share files, caches or git history, so one run helps or breaks the next.
• Dropping a task that scores zero before reading its transcripts and running its reference solution.

Try it yourself

• Pick five tasks your agent handles and write the end state each should leave, as a database query or a file check.
• Run each task five times and work out pass@5 and pass^5 by hand from the pass and fail marks.
• Read every transcript of the task with the lowest score and sort each failure: agent mistake, grader bug or unclear task.
• Add one task where the right move is to do nothing, and check that the agent leaves the state unchanged.

Frequently asked questions

Sources

1. Anthropic Engineering: Demystifying evals for AI agents (9 January 2026) (read November 6, 2026)https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
2. τ-bench README on GitHub (Sierra Research) (read November 6, 2026)https://github.com/sierra-research/tau-bench
3. Hugging Face Papers: τ-bench, a benchmark for tool-agent-user interaction (2406.12045) (read November 6, 2026)https://huggingface.co/papers/2406.12045
4. Inspect docs on GitHub: Inspect (welcome page and CTF example) (read November 6, 2026)https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/docs/index.qmd
5. Inspect docs on GitHub: Scoring Metrics (epoch reducers) (read November 6, 2026)https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/docs/metrics.qmd
6. Inspect changelog on GitHub (read November 6, 2026)https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/CHANGELOG.md
7. PyPI: inspect-ai release history (read November 6, 2026)https://pypi.org/project/inspect-ai/
8. OpenAI API docs: Trace grading (read November 6, 2026)https://developers.openai.com/api/docs/guides/trace-grading
9. OpenAI API docs: Evaluate agent workflows (read November 6, 2026)https://developers.openai.com/api/docs/guides/agent-evals
10. OpenAI API docs: Deprecations (Evals platform) (read November 6, 2026)https://developers.openai.com/api/docs/deprecations
11. promptfoo docs on GitHub: Tracing (read November 6, 2026)https://github.com/promptfoo/promptfoo/blob/main/site/docs/tracing.md
12. promptfoo docs on GitHub: Deterministic metrics (trajectory assertions) (read November 6, 2026)https://github.com/promptfoo/promptfoo/blob/main/site/docs/configuration/expected-outputs/deterministic.md
13. promptfoo docs on GitHub: Model-graded metrics (trajectory:goal-success) (read November 6, 2026)https://github.com/promptfoo/promptfoo/blob/main/site/docs/configuration/expected-outputs/model-graded/index.md
14. Anthropic on X: Demystifying evals for AI agents (read November 6, 2026)https://x.com/AnthropicAI/status/2009696515061911674
15. Karthik Narasimhan on X: τ-bench (read November 6, 2026)https://x.com/karthik_r_n/status/1803846916800942292
16. Hamel Husain on YouTube: Can Claude Build Your Evals? (13 October 2026) (read November 6, 2026)https://www.youtube.com/watch?v=5ykulgyB8kk
17. Parlance Labs on GitHub: organization page (read November 6, 2026)https://github.com/parlance-labs