# Evals in CI: run LLM evals on every pull request with GitHub Actions > Run evals in CI by keeping the test cases in your repository and running them on every pull request that touches a prompt, a model or a tool. The promptfoo GitHub Action does this: it runs the suite, comments the results on the pull request and fails the job when the pass rate drops below your gate. A cache stops unchanged tests from calling the model again. Mindket editors · Published 2026-11-06 · Updated 2026-11-06 · https://mindket.com/learn/ai-evals/evals-in-ci ## In short - Keep eval cases and graders in the repository, next to the prompts they test. - Run them on every pull request that changes a prompt, and let a failed eval fail the job. - Gate the regression suite, and run capability tests as a report that never blocks. - Use repeats for flaky model graders, and a cache so unchanged tests are not sent twice. - Move graders out of OpenAI's Evals dashboard before it shuts down. ## How one prompt line slipped past Anthropic's evals On 16 April 2026, Anthropic shipped Opus 4.7 in Claude Code along with a new system prompt line meant to cut verbosity. It read: "Length limits: keep text between tool calls to ≤25 words. Keep final responses to ≤100 words unless the task requires more detail." The line had passed multiple weeks of internal testing with no regressions in the evaluations the team ran. Users then reported that Claude Code's quality had slipped. While investigating, Anthropic ran ablations, removing one prompt line at a time, on a broader set of evaluations. One of them showed a 3% drop for both Opus 4.6 and 4.7, and the line came out in the 20 April release, as [the postmortem](https://www.anthropic.com/engineering/april-23-postmortem) tells it. The fix it announced is the subject of this lesson: "We will run a broad suite of per-model evals for every system prompt change to Claude Code." Your app needs the same habit at a smaller size. Eval cases sit in the repository, each change to a prompt gets a run before it merges, and a lower score fails the job. Anthropic's [agent evals post](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) already places automated evals in CI/CD, running on each agent change and model upgrade. ## Before you start You need a GitHub repository holding the prompts your app sends, a few test cases written the way [lesson 1](https://mindket.com/learn/ai-evals/what-are-llm-evals) shows, and an API key for the model you test. Save the key as a repository secret under Settings, Secrets and variables, Actions. promptfoo needs Node.js 22.22.0 or newer, and [its CI docs](https://github.com/promptfoo/promptfoo/blob/main/site/docs/integrations/ci-cd.md) recommend Node 24 LTS. On 6 November 2026 the newest promptfoo on npm was 0.125.1, from 25 October. The GitHub Action runs `npx promptfoo@latest` unless you set its `promptfoo-version` input. Pin it there, so a release between two pull requests cannot change how they are scored. ## Why promptfoo and not OpenAI Evals OpenAI is winding down its hosted Evals product. [Its deprecations page](https://developers.openai.com/api/docs/deprecations) announced the change on 3 June 2026, made existing evals read-only on 31 October and schedules the Evals dashboard and API to shut down on 30 November 2026. Graders documented for eval workflows are part of the same transition. OpenAI's [migration guide](https://developers.openai.com/cookbook/examples/evaluation/moving-from-openai-evals-to-promptfoo) recommends promptfoo, which it describes as an open-source CLI and library for evaluating and red-teaming AI apps and agents. OpenAI agreed to acquire Promptfoo on 9 March 2026, and the promptfoo team wrote that the project will remain open source. The guide moves evals by hand and needs no export feature. You recreate each eval's prompts, provider, test cases and assertions in a `promptfooconfig.yaml` and check it with `promptfoo validate config`. The guide also asks you to validate any recreated grader, an LLM judge above all, before you rely on it for regression decisions. **From OpenAI Evals to promptfoo** 1. **March 9, 2026** OpenAI agrees to acquire Promptfoo. The promptfoo team says the project will remain open source. 2. **June 3, 2026** Evals platform deprecation announced. OpenAI notifies developers who use the Evals platform. 3. **October 31, 2026** Existing evals become read-only. Evals can be read but no longer changed. 4. **November 30, 2026** Evals dashboard and API shut down. The date OpenAI schedules for the shutdown. Sources: [promptfoo blog on GitHub: Promptfoo is joining OpenAI (9 March 2026)](https://github.com/promptfoo/promptfoo/blob/main/site/blog/promptfoo-joining-openai/index.mdx) (read 2026-11-06), [OpenAI API docs: Deprecations (Evals platform)](https://developers.openai.com/api/docs/deprecations) (read 2026-11-06) ## Write the suite as a promptfoo config A promptfoo config lists the prompts, the providers that run them and the tests. Each test sets variables and makes assertions about the output. The example below comes from promptfoo's [model-graded metrics docs](https://github.com/promptfoo/promptfoo/blob/main/site/docs/configuration/expected-outputs/model-graded/index.md). It runs two prompt files against one model and gives every test the same `llm-rubric` assertion, a judge that passes an answer only if it says it is uncertain or unable to answer. The tests fill in `question`, and the rubric reads it too. ````yaml providers: - openai:gpt-6-sol prompts: - file://prompt1.txt - file://prompt2.txt defaultTest: assert: - type: llm-rubric value: 'Says that it is uncertain or unable to answer the question: "{{question}}"' tests: - vars: question: What's the weather in New York? - vars: question: Who won the latest football match between the Giants and 49ers? ```` - Use code assertions wherever they can decide. `contains`, `equals` and `is-json` need no model call and give the same verdict every run. - An `llm-rubric` is an LLM judge, so check its verdicts against your own labels, as lesson 2 does, before it gates a merge. - When you use `repeat`, give any exact duplicate tests you define on purpose a unique `id` or `description`, as the action's README advises, so its report can tell them apart. - Run the suite locally with `promptfoo eval -c promptfooconfig.yaml`, then open the results with `promptfoo view`. ## Run it on every pull request [promptfoo's GitHub Action](https://github.com/promptfoo/promptfoo-action) runs the suite and posts a comment on the pull request with pass and fail counts. The workflow below is the pull request example from its README. It starts only when a file under `prompts/` or the config changes, and it hands the changed prompt files to promptfoo with `--prompts`. Without a `prompts` input, the config's own prompts run. ````yaml name: 'Prompt Evaluation' on: pull_request: paths: - 'prompts/**' - 'promptfooconfig.yaml' jobs: evaluate: runs-on: ubuntu-latest permissions: contents: read # Required for actions/checkout pull-requests: write # Ability to post comments on Pull Requests steps: # Required for promptfoo-action's git usage - uses: actions/checkout@v4 # This cache is optional, but you'll save money and time by setting it up! - name: Set up promptfoo cache uses: actions/cache@v4 with: path: .promptfoo-cache key: ${{ runner.os }}-promptfoo-${{ hashFiles('promptfooconfig.yaml', 'prompts/**') }} restore-keys: | ${{ runner.os }}-promptfoo- - name: Run promptfoo evaluation uses: promptfoo/promptfoo-action@v1 with: openai-api-key: ${{ secrets.OPENAI_API_KEY }} github-token: ${{ secrets.GITHUB_TOKEN }} config: 'promptfooconfig.yaml' prompts: 'prompts/**' cache-path: '.promptfoo-cache' ```` - The action evaluates only the current checkout. It does not run the base branch and the pull request side by side, so keep an earlier run when you need a before and after. - If you switch the trigger to `pull_request_target`, take the README at its word: do not execute untrusted pull request code with a privileged token. 1. Save the API key of your model provider as a repository secret named `OPENAI_API_KEY`. 2. Commit `promptfooconfig.yaml` and the `prompts/` folder, with the config's prompt paths pointing into it, such as `file://prompts/prompt1.txt`. 3. Save the workflow below as a YAML file in `.github/workflows/`. 4. Open a pull request that edits a prompt, and read the comment the action posts. ## Decide what fails the build By [its command-line docs](https://github.com/promptfoo/promptfoo/blob/main/site/docs/usage/command-line.md), `promptfoo eval` exits with code 100 when at least one test fails or the pass rate falls below `PROMPTFOO_PASS_RATE_THRESHOLD`, and with code 1 for any other error. The threshold is a percentage. With none set, no failure is allowed. Any non-zero exit fails the CI step. That default suits a regression suite, which should pass almost every time. A capability suite starts low by design, so run it as a report. `PROMPTFOO_FAILED_TEST_EXIT_CODE` overrides the failure code, and 0 lets the job pass. In the action, `fail-on-threshold` sets the pass percentage a suite needs. Model graders vary between runs. With `repeat: 3` and `repeat-min-pass: 2`, the action runs each test three times and requires two passes. Tests that fail consistently get flagged, while random grader variance is tolerated. Size the gate to the noise. A 3% drop on one broader eval was enough for Anthropic to pull a prompt line. In its internal experiments, [resource settings alone](https://www.anthropic.com/engineering/infrastructure-noise) moved Terminal-Bench 2.0 by 6 percentage points, and the same post says leaderboard gaps under 3 points deserve skepticism until the setup is documented and matched. Keep the runner, the model version and the judge the same between the runs you compare. - **3%** Drop one broader eval showed for Opus 4.6 and 4.7 after a single system prompt line (Source: [Anthropic Engineering: An update on recent Claude Code quality reports (23 April 2026)](https://www.anthropic.com/engineering/april-23-postmortem), read 2026-11-06) - **6 points** Terminal-Bench 2.0 lift with uncapped resources over the 1x setting (p < 0.01) (Source: [Anthropic Engineering: Quantifying infrastructure noise in agentic coding evals (5 February 2026)](https://www.anthropic.com/engineering/infrastructure-noise), read 2026-11-06) - **3 points** Leaderboard gaps below this deserve skepticism until the setups are documented and matched (Source: [Anthropic Engineering: Quantifying infrastructure noise in agentic coding evals (5 February 2026)](https://www.anthropic.com/engineering/infrastructure-noise), read 2026-11-06) ````bash # regression suite: every test must pass (no threshold set) npx promptfoo@0.125.1 eval -c regression.yaml # capability suite: report only, a failed test exits 0 PROMPTFOO_FAILED_TEST_EXIT_CODE=0 npx promptfoo@0.125.1 eval -c capability.yaml ```` ## Keep the calls down: cache, path filters, concurrency Every eval run sends model calls, and the provider bills each one. promptfoo caches responses on disk, and `PROMPTFOO_CACHE_TTL` defaults to 1209600 seconds, 14 days. A test whose request has not changed is answered from the cache. GitHub's hosted runners start empty, which is why the workflow above saves `.promptfoo-cache` with `actions/cache`, keyed on a hash of the config and the prompts. The action prunes that directory itself, removing entries older than seven days when `CI=true`. - The `paths` filter in the workflow starts the job only when a prompt or the config changes. - `no-cache: true` in the action, or `--no-cache` on the command line, forces fresh calls. Use it on a nightly run, or after your provider updates a model. - If the provider rate-limits you, promptfoo's CI docs suggest caching and fewer concurrent calls, down to `-j 1`. The action's `max-concurrency` defaults to 4. ## Inspect or pytest instead of promptfoo An agent that works in a sandbox fits Inspect better, and the CTF task from [lesson 3](https://mindket.com/learn/ai-evals/evaluate-ai-agents) runs in a CI job as it is. [Inspect's log docs](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/docs/eval-logs.qmd) say to check a log's status before you analyse its results. The script below, this lesson's own glue, does that, prints each reducer's accuracy and fails the job when pass^5 drops below 0.8. A pytest file works too. GitHub's [Python guide](https://docs.github.com/en/actions/tutorials/build-and-test-code/python) installs pytest and runs `pytest` in a workflow step. A test that calls your app and asserts on its answer fails that step like any other test. ````python # ci_eval.py import sys from inspect_ai import eval from ctf import ctf log = eval(ctf(), model="anthropic/claude-sonnet-4-6")[0] if log.status != "success": sys.exit(f"eval did not finish: {log.status}") for score in log.results.scores: accuracy = score.metrics["accuracy"].value print(score.reducer, accuracy) if score.reducer == "pass_k_5" and accuracy < 0.8: sys.exit("pass^5 fell below 0.8") ```` ## ClaudeDevs, Paige Bailey and Ben Hylak on catching regressions Anthropic opened the [ClaudeDevs](https://x.com/ClaudeDevs/status/2047371123185287223) account on X to explain product decisions. On 23 April 2026 it told users who had reported a drop in Claude Code's quality that the team had published a post-mortem on the three issues it found, all fixed in v2.1.116 and later. That post-mortem is the story at the top of this lesson. Paige Bailey, engineering lead for developer relations at Google DeepMind and formerly at GitHub by her X bio, described the pull request loop [on X](https://x.com/DynamicWebPaige/status/1811448653640606011) on 11 July 2024. When you check in a change to a prompt, an updated eval is posted on the PR with a link to the promptfoo web viewer. Her post adds that promptfoo also uses caching. Ben Hylak, CTO of Raindrop by his X bio, read that post-mortem the next day and [named the reason](https://x.com/benhylak/status/2047559627400003676) for all of this in one line. > anyone building agents gets how hard it is to make forward progress without regressions. > > Ben Hylak, CTO of Raindrop, by his X bio, [X](https://x.com/benhylak/status/2047559627400003676) (read 2026-11-06) [![ClaudeDevs (@ClaudeDevs) on X, 23 April 2026: Over the past month, some of you reported Claude Code's quality had slipped. We investigated, and published a post-mortem on the three issues we found. All are fixed in v2.1.116+ and we've reset usage limits for all subscribers.](https://mindket.com/blog/shots/x-claudedevs-quality-postmortem.webp?v=106613ecd524)](https://x.com/ClaudeDevs/status/2047371123185287223) ClaudeDevs on X, 23 April 2026 [![Paige Bailey (@DynamicWebPaige) on X, 11 July 2024, praising promptfoo's integrations with Gemini APIs and GitHub Actions. When you check in a change to your prompt, an updated eval is automatically posted on the PR with a link to the promptfoo web viewer. Promptfoo also uses caching to store…](https://mindket.com/blog/shots/x-dynamicwebpaige-promptfoo-actions.webp?v=dd8e1b783d15)](https://x.com/DynamicWebPaige/status/1811448653640606011) Paige Bailey on X, 11 July 2024 > Screenshots of X's own embeds of the posts, taken on 6 November 2026, with the time in UTC. X cuts Paige Bailey's post short with Show more. Select one to open the post on X. ## Common mistakes - Running evals by hand before a release, so a prompt edit merged in between ships untested. - Putting capability tests in the blocking suite, so a set built to start low fails every pull request. - Gating merges on an LLM judge nobody has checked against people. - Failing the build on a single run of a flaky judge instead of using repeats. - Running `pull_request_target` with secrets on code from a fork. - Calling a gap a regression when the model version, the judge or the runner changed between runs. - Keeping the only copy of your graders in OpenAI's Evals dashboard. ## Try it yourself - Write five tests for one prompt, two with code assertions and three with `llm-rubric`, and run them locally. - Add the workflow, open a pull request that changes one word of the prompt and read the comment. - Break a test on purpose, run `promptfoo eval` and print its exit code with `echo $?`. - Set `repeat: 3` and `repeat-min-pass: 2` on the judge tests and count how many runs flip. ## Next course: n8n This is the last lesson of the AI evals course. The next course, n8n, opens the Automate your work path. [Its first lesson](https://mindket.com/learn/n8n/n8n-first-workflow) builds a four-node workflow that logs the weather to a data table every day. Its third, [the n8n AI agent](https://mindket.com/learn/n8n/n8n-ai-agent), wires a chat model to tools, which is where a prompt you have tested can run without you. ## Check yourself 1. Why did Anthropic pull a line it had added to the Claude Code system prompt in April 2026? A. Users asked for longer answers B. A broader set of evals showed a 3% drop for Opus 4.6 and 4.7 C. The line broke the API D. The line failed a unit test Answer: B. The line had passed weeks of internal testing. Ablations on a broader eval set found the drop, and Anthropic reverted the line on 20 April. 2. One test fails and no pass-rate threshold is set. What does `promptfoo eval` do? A. Exits with code 0 and prints a warning B. Retries the test until it passes C. Exits with a failure code, which fails the CI step D. Skips the failing test Answer: C. With no threshold set, no failure is allowed, so the command exits with its failure code and the step fails. 3. What do `repeat: 3` and `repeat-min-pass: 2` do in the promptfoo GitHub Action? A. Run each test three times and require at least two passes B. Retry the whole workflow three times C. Run three tests at once D. Require two reviewers on the pull request Answer: A. Each test runs three times and needs two passes, so random grader variance is tolerated while tests that fail consistently are flagged. 4. Which statement about the promptfoo GitHub Action is true? A. It compares the base branch with the pull request B. It keeps its cache between runners on its own C. It runs only on push events D. It evaluates only the current checkout Answer: D. The README says it evaluates only the current checkout, and the cache persists only when the workflow adds actions/cache. ## Frequently asked questions ### How do I run LLM evals in GitHub Actions without promptfoo? Write them as pytest tests or an Inspect script and run them in an ordinary job step, since any step that exits non-zero fails the job. Claude Code can also run in GitHub Actions, as [its lesson](https://mindket.com/learn/claude-code-advanced/claude-code-github-actions) shows, though that job reviews and writes code rather than scoring your prompts. ### What happens to my OpenAI Evals? They have been read-only since 31 October 2026, and the Evals dashboard and API are scheduled to shut down on 30 November 2026. OpenAI's migration guide recommends promptfoo and rebuilds each eval by hand, with no export step. Recreate each one in a promptfoo config, compare its first results with the old ones, and keep the config in your repo so the graders outlive the dashboard. ### How much does it cost to run evals in CI? You pay for model calls, not for promptfoo, which is open source. Each test calls the provider once for every prompt and provider pair, a model-graded assertion adds a judge call, and repeats multiply the lot. Caching and the paths filter decide how often that happens. ### Should a flaky eval block a merge? Not on one run. Make a test fail most of its repeats before it blocks, and move one that keeps flipping to a report-only suite. Then read its outputs to learn whether the test, the judge or your app is at fault. ### Is promptfoo or Inspect better for evals in CI? promptfoo fits prompts and app outputs, because its action comments on the pull request and its tests need no code. For an agent you test over many trials, Inspect fits better: it can run each task in a Docker sandbox, and its epoch reducers report pass@k and pass^k. ## Sources - [Anthropic Engineering: An update on recent Claude Code quality reports (23 April 2026)](https://www.anthropic.com/engineering/april-23-postmortem) (read 2026-11-06) - [Anthropic Engineering: Quantifying infrastructure noise in agentic coding evals (5 February 2026)](https://www.anthropic.com/engineering/infrastructure-noise) (read 2026-11-06) - [Anthropic Engineering: Demystifying evals for AI agents (9 January 2026)](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) (read 2026-11-06) - [OpenAI API docs: Deprecations (Evals platform)](https://developers.openai.com/api/docs/deprecations) (read 2026-11-06) - [OpenAI Cookbook: Moving from OpenAI Evals to Promptfoo](https://developers.openai.com/cookbook/examples/evaluation/moving-from-openai-evals-to-promptfoo) (read 2026-11-06) - [promptfoo blog on GitHub: Promptfoo is joining OpenAI (9 March 2026)](https://github.com/promptfoo/promptfoo/blob/main/site/blog/promptfoo-joining-openai/index.mdx) (read 2026-11-06) - [promptfoo docs on GitHub: Command line (exit codes, environment variables)](https://github.com/promptfoo/promptfoo/blob/main/site/docs/usage/command-line.md) (read 2026-11-06) - [promptfoo docs on GitHub: CI/CD integration](https://github.com/promptfoo/promptfoo/blob/main/site/docs/integrations/ci-cd.md) (read 2026-11-06) - [promptfoo docs on GitHub: Model-graded metrics](https://github.com/promptfoo/promptfoo/blob/main/site/docs/configuration/expected-outputs/model-graded/index.md) (read 2026-11-06) - [promptfoo-action README on GitHub](https://github.com/promptfoo/promptfoo-action) (read 2026-11-06) - [npm registry: promptfoo (latest 0.125.1)](https://registry.npmjs.org/promptfoo) (read 2026-11-06) - [GitHub Docs: Building and testing Python](https://docs.github.com/en/actions/tutorials/build-and-test-code/python) (read 2026-11-06) - [Inspect docs on GitHub: Inspect (welcome page, Docker sandbox in the CTF example)](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/docs/index.qmd) (read 2026-11-06) - [Inspect docs on GitHub: Scoring Metrics (epoch reducers)](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/docs/metrics.qmd) (read 2026-11-06) - [Inspect docs on GitHub: Eval Logs](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/docs/eval-logs.qmd) (read 2026-11-06) - [Inspect source on GitHub: log/_log.py (EvalResults)](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/log/_log.py) (read 2026-11-06) - [ClaudeDevs on X: Claude Code quality post-mortem](https://x.com/ClaudeDevs/status/2047371123185287223) (read 2026-11-06) - [Paige Bailey on X: promptfoo and GitHub Actions](https://x.com/DynamicWebPaige/status/1811448653640606011) (read 2026-11-06) - [Ben Hylak on X: forward progress without regressions (24 April 2026)](https://x.com/benhylak/status/2047559627400003676) (read 2026-11-06) [Browse agents](https://mindket.com/agents)