Evals in CI: run LLM evals on every pull request with GitHub Actions

How one prompt line slipped past Anthropic's evals

On 16 April 2026, Anthropic shipped Opus 4.7 in Claude Code along with a new system prompt line meant to cut verbosity. It read: "Length limits: keep text between tool calls to ≤25 words. Keep final responses to ≤100 words unless the task requires more detail." The line had passed multiple weeks of internal testing with no regressions in the evaluations the team ran.

Users then reported that Claude Code's quality had slipped. While investigating, Anthropic ran ablations, removing one prompt line at a time, on a broader set of evaluations. One of them showed a 3% drop for both Opus 4.6 and 4.7, and the line came out in the 20 April release, as the postmortem tells it.

The fix it announced is the subject of this lesson: "We will run a broad suite of per-model evals for every system prompt change to Claude Code." Your app needs the same habit at a smaller size. Eval cases sit in the repository, each change to a prompt gets a run before it merges, and a lower score fails the job. Anthropic's agent evals post already places automated evals in CI/CD, running on each agent change and model upgrade.

Before you start

You need a GitHub repository holding the prompts your app sends, a few test cases written the way lesson 1 shows, and an API key for the model you test. Save the key as a repository secret under Settings, Secrets and variables, Actions. promptfoo needs Node.js 22.22.0 or newer, and its CI docs recommend Node 24 LTS.

On 6 November 2026 the newest promptfoo on npm was 0.125.1, from 25 October. The GitHub Action runs npx promptfoo@latest unless you set its promptfoo-version input. Pin it there, so a release between two pull requests cannot change how they are scored.

Why promptfoo and not OpenAI Evals

OpenAI is winding down its hosted Evals product. Its deprecations page announced the change on 3 June 2026, made existing evals read-only on 31 October and schedules the Evals dashboard and API to shut down on 30 November 2026. Graders documented for eval workflows are part of the same transition.

OpenAI's migration guide recommends promptfoo, which it describes as an open-source CLI and library for evaluating and red-teaming AI apps and agents. OpenAI agreed to acquire Promptfoo on 9 March 2026, and the promptfoo team wrote that the project will remain open source.

The guide moves evals by hand and needs no export feature. You recreate each eval's prompts, provider, test cases and assertions in a promptfooconfig.yaml and check it with promptfoo validate config. The guide also asks you to validate any recreated grader, an LLM judge above all, before you rely on it for regression decisions.

Write the suite as a promptfoo config

A promptfoo config lists the prompts, the providers that run them and the tests. Each test sets variables and makes assertions about the output. The example below comes from promptfoo's model-graded metrics docs. It runs two prompt files against one model and gives every test the same llm-rubric assertion, a judge that passes an answer only if it says it is uncertain or unable to answer. The tests fill in question, and the rubric reads it too.

providers:
  - openai:gpt-6-sol
prompts:
  - file://prompt1.txt
  - file://prompt2.txt
defaultTest:
  assert:
    - type: llm-rubric
      value: 'Says that it is uncertain or unable to answer the question: "{{question}}"'
tests:
  - vars:
      question: What's the weather in New York?
  - vars:
      question: Who won the latest football match between the Giants and 49ers?

• Use code assertions wherever they can decide. contains, equals and is-json need no model call and give the same verdict every run.
• An llm-rubric is an LLM judge, so check its verdicts against your own labels, as lesson 2 does, before it gates a merge.
• When you use repeat, give any exact duplicate tests you define on purpose a unique id or description, as the action's README advises, so its report can tell them apart.
• Run the suite locally with promptfoo eval -c promptfooconfig.yaml, then open the results with promptfoo view.

Run it on every pull request

promptfoo's GitHub Action runs the suite and posts a comment on the pull request with pass and fail counts. The workflow below is the pull request example from its README. It starts only when a file under prompts/ or the config changes, and it hands the changed prompt files to promptfoo with --prompts. Without a prompts input, the config's own prompts run.

name: 'Prompt Evaluation'
on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'promptfooconfig.yaml'

jobs:
  evaluate:
    runs-on: ubuntu-latest
    permissions:
      contents: read # Required for actions/checkout
      pull-requests: write # Ability to post comments on Pull Requests
    steps:
      # Required for promptfoo-action's git usage
      - uses: actions/checkout@v4

      # This cache is optional, but you'll save money and time by setting it up!
      - name: Set up promptfoo cache
        uses: actions/cache@v4
        with:
          path: .promptfoo-cache
          key: ${{ runner.os }}-promptfoo-${{ hashFiles('promptfooconfig.yaml', 'prompts/**') }}
          restore-keys: |
            ${{ runner.os }}-promptfoo-

      - name: Run promptfoo evaluation
        uses: promptfoo/promptfoo-action@v1
        with:
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
          github-token: ${{ secrets.GITHUB_TOKEN }}
          config: 'promptfooconfig.yaml'
          prompts: 'prompts/**'
          cache-path: '.promptfoo-cache'

• The action evaluates only the current checkout. It does not run the base branch and the pull request side by side, so keep an earlier run when you need a before and after.
• If you switch the trigger to pull_request_target, take the README at its word: do not execute untrusted pull request code with a privileged token.

1. Save the API key of your model provider as a repository secret named OPENAI_API_KEY.
2. Commit promptfooconfig.yaml and the prompts/ folder, with the config's prompt paths pointing into it, such as file://prompts/prompt1.txt.
3. Save the workflow below as a YAML file in .github/workflows/.
4. Open a pull request that edits a prompt, and read the comment the action posts.

Decide what fails the build

By its command-line docs, promptfoo eval exits with code 100 when at least one test fails or the pass rate falls below PROMPTFOO_PASS_RATE_THRESHOLD, and with code 1 for any other error. The threshold is a percentage. With none set, no failure is allowed. Any non-zero exit fails the CI step.

That default suits a regression suite, which should pass almost every time. A capability suite starts low by design, so run it as a report. PROMPTFOO_FAILED_TEST_EXIT_CODE overrides the failure code, and 0 lets the job pass. In the action, fail-on-threshold sets the pass percentage a suite needs.

Model graders vary between runs. With repeat: 3 and repeat-min-pass: 2, the action runs each test three times and requires two passes. Tests that fail consistently get flagged, while random grader variance is tolerated.

Size the gate to the noise. A 3% drop on one broader eval was enough for Anthropic to pull a prompt line. In its internal experiments, resource settings alone moved Terminal-Bench 2.0 by 6 percentage points, and the same post says leaderboard gaps under 3 points deserve skepticism until the setup is documented and matched. Keep the runner, the model version and the judge the same between the runs you compare.

# regression suite: every test must pass (no threshold set)
npx promptfoo@0.125.1 eval -c regression.yaml

# capability suite: report only, a failed test exits 0
PROMPTFOO_FAILED_TEST_EXIT_CODE=0 npx promptfoo@0.125.1 eval -c capability.yaml

Keep the calls down: cache, path filters, concurrency

Every eval run sends model calls, and the provider bills each one. promptfoo caches responses on disk, and PROMPTFOO_CACHE_TTL defaults to 1209600 seconds, 14 days. A test whose request has not changed is answered from the cache. GitHub's hosted runners start empty, which is why the workflow above saves .promptfoo-cache with actions/cache, keyed on a hash of the config and the prompts. The action prunes that directory itself, removing entries older than seven days when CI=true.

• The paths filter in the workflow starts the job only when a prompt or the config changes.
• no-cache: true in the action, or --no-cache on the command line, forces fresh calls. Use it on a nightly run, or after your provider updates a model.
• If the provider rate-limits you, promptfoo's CI docs suggest caching and fewer concurrent calls, down to -j 1. The action's max-concurrency defaults to 4.

Inspect or pytest instead of promptfoo

An agent that works in a sandbox fits Inspect better, and the CTF task from lesson 3 runs in a CI job as it is. Inspect's log docs say to check a log's status before you analyse its results. The script below, this lesson's own glue, does that, prints each reducer's accuracy and fails the job when pass^5 drops below 0.8.

A pytest file works too. GitHub's Python guide installs pytest and runs pytest in a workflow step. A test that calls your app and asserts on its answer fails that step like any other test.

# ci_eval.py
import sys

from inspect_ai import eval
from ctf import ctf

log = eval(ctf(), model="anthropic/claude-sonnet-4-6")[0]
if log.status != "success":
    sys.exit(f"eval did not finish: {log.status}")

for score in log.results.scores:
    accuracy = score.metrics["accuracy"].value
    print(score.reducer, accuracy)
    if score.reducer == "pass_k_5" and accuracy < 0.8:
        sys.exit("pass^5 fell below 0.8")

ClaudeDevs, Paige Bailey and Ben Hylak on catching regressions

Anthropic opened the ClaudeDevs account on X to explain product decisions. On 23 April 2026 it told users who had reported a drop in Claude Code's quality that the team had published a post-mortem on the three issues it found, all fixed in v2.1.116 and later. That post-mortem is the story at the top of this lesson.

Paige Bailey, engineering lead for developer relations at Google DeepMind and formerly at GitHub by her X bio, described the pull request loop on X on 11 July 2024. When you check in a change to a prompt, an updated eval is posted on the PR with a link to the promptfoo web viewer. Her post adds that promptfoo also uses caching.

Ben Hylak, CTO of Raindrop by his X bio, read that post-mortem the next day and named the reason for all of this in one line.

Screenshots of X's own embeds of the posts, taken on 6 November 2026, with the time in UTC. X cuts Paige Bailey's post short with Show more. Select one to open the post on X.

Common mistakes

• Running evals by hand before a release, so a prompt edit merged in between ships untested.
• Putting capability tests in the blocking suite, so a set built to start low fails every pull request.
• Gating merges on an LLM judge nobody has checked against people.
• Failing the build on a single run of a flaky judge instead of using repeats.
• Running pull_request_target with secrets on code from a fork.
• Calling a gap a regression when the model version, the judge or the runner changed between runs.
• Keeping the only copy of your graders in OpenAI's Evals dashboard.

Try it yourself

• Write five tests for one prompt, two with code assertions and three with llm-rubric, and run them locally.
• Add the workflow, open a pull request that changes one word of the prompt and read the comment.
• Break a test on purpose, run promptfoo eval and print its exit code with echo $?.
• Set repeat: 3 and repeat-min-pass: 2 on the judge tests and count how many runs flip.

Next course: n8n

This is the last lesson of the AI evals course. The next course, n8n, opens the Automate your work path. Its first lesson builds a four-node workflow that logs the weather to a data table every day. Its third, the n8n AI agent, wires a chat model to tools, which is where a prompt you have tested can run without you.

Frequently asked questions

Sources

1. Anthropic Engineering: An update on recent Claude Code quality reports (23 April 2026) (read November 6, 2026)https://www.anthropic.com/engineering/april-23-postmortem
2. Anthropic Engineering: Quantifying infrastructure noise in agentic coding evals (5 February 2026) (read November 6, 2026)https://www.anthropic.com/engineering/infrastructure-noise
3. Anthropic Engineering: Demystifying evals for AI agents (9 January 2026) (read November 6, 2026)https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
4. OpenAI API docs: Deprecations (Evals platform) (read November 6, 2026)https://developers.openai.com/api/docs/deprecations
5. OpenAI Cookbook: Moving from OpenAI Evals to Promptfoo (read November 6, 2026)https://developers.openai.com/cookbook/examples/evaluation/moving-from-openai-evals-to-promptfoo
6. promptfoo blog on GitHub: Promptfoo is joining OpenAI (9 March 2026) (read November 6, 2026)https://github.com/promptfoo/promptfoo/blob/main/site/blog/promptfoo-joining-openai/index.mdx
7. promptfoo docs on GitHub: Command line (exit codes, environment variables) (read November 6, 2026)https://github.com/promptfoo/promptfoo/blob/main/site/docs/usage/command-line.md
8. promptfoo docs on GitHub: CI/CD integration (read November 6, 2026)https://github.com/promptfoo/promptfoo/blob/main/site/docs/integrations/ci-cd.md
9. promptfoo docs on GitHub: Model-graded metrics (read November 6, 2026)https://github.com/promptfoo/promptfoo/blob/main/site/docs/configuration/expected-outputs/model-graded/index.md
10. promptfoo-action README on GitHub (read November 6, 2026)https://github.com/promptfoo/promptfoo-action
11. npm registry: promptfoo (latest 0.125.1) (read November 6, 2026)https://registry.npmjs.org/promptfoo
12. GitHub Docs: Building and testing Python (read November 6, 2026)https://docs.github.com/en/actions/tutorials/build-and-test-code/python
13. Inspect docs on GitHub: Inspect (welcome page, Docker sandbox in the CTF example) (read November 6, 2026)https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/docs/index.qmd
14. Inspect docs on GitHub: Scoring Metrics (epoch reducers) (read November 6, 2026)https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/docs/metrics.qmd
15. Inspect docs on GitHub: Eval Logs (read November 6, 2026)https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/docs/eval-logs.qmd
16. Inspect source on GitHub: log/_log.py (EvalResults) (read November 6, 2026)https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/log/_log.py
17. ClaudeDevs on X: Claude Code quality post-mortem (read November 6, 2026)https://x.com/ClaudeDevs/status/2047371123185287223
18. Paige Bailey on X: promptfoo and GitHub Actions (read November 6, 2026)https://x.com/DynamicWebPaige/status/1811448653640606011
19. Ben Hylak on X: forward progress without regressions (24 April 2026) (read November 6, 2026)https://x.com/benhylak/status/2047559627400003676