Before you start
A test set is a short list of inputs with what a good reply to each one looks like. Running every prompt version on the same inputs is the only way to tell whether a rewrite helped or just read better on the one case you tried. An assertion is an automatic check on a reply, such as "contains this word" or "a model judges it against this rule".
Bring the prompt you want to improve, its rewrite from lesson 2 and a handful of inputs. Include inputs where the current prompt fails and at least one where it works. A rewrite can fix one case and break another, and you only see that if both kinds are in the set. The evals lesson goes deeper on test sets and graders.
OpenAI's cookbook says it is winding down its Evals product and recommends Promptfoo for continuing evaluation work. That is one reason this lesson ends on promptfoo.
Three ways to compare prompt versions
The routes use the same idea at different scales. The comparison below is this lesson's own. The Console column follows Anthropic's course, which covers human grading, and a cell reads "not covered" where the course says nothing. The promptfoo column follows its docs.
| Spreadsheet | Anthropic Console | promptfoo | |
|---|---|---|---|
| Needs a terminal | No | No | Yes |
| Checks replies automatically | No | scored by hand | Yes |
| Shows two versions side by side | Partly | Yes | Yes |
| Works with other vendors' models | Yes | not covered | Yes |
| Runs in CI | No | not covered | Yes |
Sources: Anthropic on GitHub: Prompt evaluations course, lesson 2, Workbench evaluations (read November 6, 2026), promptfoo docs on GitHub: Intro (read November 6, 2026), promptfoo docs on GitHub: Getting started (read November 6, 2026)
Start in a spreadsheet
A spreadsheet is enough for the first round, and a short one at that. promptfoo can later read the same sheet as a CSV or Excel file, so nothing you write here is wasted. The table lays it out for the shoe-shop assistant in promptfoo's example, one row per input.
- List a handful of inputs in the first column, with the ones your prompt fails on first.
- Write in the second column what a good reply does, before you run anything.
- Run each prompt version on every input, paste the reply, and mark pass or fail.
- Count the passes for each version, then read the failures. The count alone hides which inputs broke.
| Input | What a good reply does | Version 1: pass? | Version 2: pass? |
|---|---|---|---|
| Can you help me find a specific product on your website? | Helps with the product search and does not mention being an AI | yes or no | yes or no |
| Do you have any promotions or discounts currently available? | Answers about promotions, keeps it short and does not mention being an AI | yes or no | yes or no |
Compare versions in Anthropic's Console
Anthropic's prompt evaluations course teaches human-graded evaluations in the Anthropic Workbench, which was the Console's prompt tool when the course was written. The buttons it names may have moved since. The course says the Workbench is a good place to start with human grading before you move to code-graded or model-graded evals.
Its example prompt translates code from any language into Python, with the source code and its language as variables written {{SOURCE_CODE}} and {{SOURCE_LANGUAGE}}. Version 1 only asks for the translation. Version 2, below, adds a required output format, and version 1 is the same prompt without the format instructions at the end.
You are a skilled programmer tasked with translating code from one programming language to Python. Your goal is to produce an accurate and idiomatic Python translation of the provided source code.
Here is the source code to translate:
<source_code>
{{SOURCE_CODE}}
</source_code>
The source code is written in the following language:
<source_language>
{{SOURCE_LANGUAGE}}
</source_language>
Please translate this code to Python.
Format your response as follows:
<python_code>
Your Python translation here
</python_code>
Only output the <python_code> tags without any other text content- Write the prompt with double-brace variables, then open the variables button and type a test value for each one.
- Press Run and read the output.
- Switch to the evaluate view with the toggle at the top. Press Add Row for each new case and Run Remaining to run the rest.
- Give each output a score. The course gave version 1 three out of five for preambles, a format that is hard to parse and long explanations.
- Edit the prompt, which becomes version 2, and run the remaining cases again. The course scored all of version 2's outputs five out of five.
- Press Add Comparison, pick version 1 and read the outputs and scores side by side.
Compare versions with promptfoo
promptfoo is an open-source CLI and library for evaluating and red-teaming LLM apps. Its intro calls the goal test-driven LLM development, not trial and error, and says the evals run on your machine and talk directly to the model. The package record on npm lists Node 22.22.0 or newer for the version published as 0.125.1.
The docs' prompt-quality example compares two customer-support prompts for a shoe company. A prompts file holds the versions, and a config holds the tests and checks. Both files appear below, each under a comment line, as the example in the repository holds them.
# prompts.txt
You're an ecommerce chat assistant for a shoe company.
Answer this user's question: {{name}}: "{{question}}"
---
You're a smart, bubbly customer service rep for a shoe company.
Answer this user's question concisely: {{name}}: "{{question}}"
# promptfooconfig.yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Automatic response evaluation using LLM rubric scoring
prompts:
- file://prompts.txt
providers:
- openai:gpt-6-sol
defaultTest:
assert:
- type: llm-rubric
value: Do not mention that you are an AI or chat assistant
- type: javascript
# Shorter is better
value: Math.max(0, Math.min(1, 1 - (output.length - 100) / 900));
tests:
- vars:
name: Bob
question: Can you help me find a specific product on your website?
- vars:
name: Jane
question: Do you have any promotions or discounts currently available?- A line of three dashes between two prompts makes them two prompt versions, so every test runs on both.
{{name}}and{{question}}are variables, filled from each test'svars.defaultTest.assertruns on every test: anllm-rubricthat a model grades against a plain rule, and ajavascriptscore that rewards shorter replies.
- Create the example with
npx promptfoo@latest init --example eval-self-grading, then move into the new directory. - Set your key with
export OPENAI_API_KEY=your-key-here. The example'sprovidersline names an OpenAI model, and the docs say each provider has its own setup requirements. - Run
npx promptfoo@latest eval. It substitutes the variables, runs both prompts and prints the results in the terminal. The command-line docs list a--no-cacheflag that stops results being read from or written to the disk cache. - Open the results in your browser with
npx promptfoo@latest view. - To test your own prompts, replace the two in
prompts.txtand the tests inpromptfooconfig.yaml. The command-line docs list a--testsflag that takes the path to a CSV file of test cases.
Choose a check for each input
The docs say assertions are optional, and that many people get value from reviewing outputs by hand in the web viewer. Add one where the rule is clear, and skip it where it is not. In a CSV test file, an __expected column holds one assertion per row. A value with no prefix means equals, and prefixes such as contains: and llm-rubric: pick the other types.
input,__expected
"Hello world","contains: Hello"
"Calculate 5 * 6","equals: 30"
"What's the weather?","llm-rubric: Provides weather information"| Check | Passes when | Needs |
|---|---|---|
| equals | The reply matches the expected text exactly | Nothing else |
| contains, icontains | The reply contains a string or number, and icontains ignores case | Nothing else |
| javascript | A function you write says the reply is fine, such as a length limit | Nothing else |
| llm-rubric | A model grades the reply against a rubric you write | A model provider to do the grading |
| similar | Embedding similarity to a reference is above a threshold | A provider and a threshold |
Read the results before you switch
promptfoo shows the results as a grid of inputs by prompt versions, which its docs call a matrix view for comparing outputs across many prompts. Read it column by column, then read the failures. Our rule of thumb: the new version has to hold every input the old one passed and fix at least one it failed. A version that wins on average but breaks an input you care about has not beaten the old one.
A model that grades can be wrong too. OpenAI's migration guide says to validate any recreated grader, especially an LLM-as-a-judge grader, before you rely on it for regression decisions. The judge lesson shows how, and the CI lesson runs the same file on every change. After a config edit that throws an error, the guide suggests promptfoo validate config -c promptfooconfig.yaml before you retry.
Common mistakes
- Judging a prompt on the one input you used while writing it.
- Changing three things in version 2, so you cannot say which change helped.
- Writing the pass rule after you have seen the replies.
- Trusting an
llm-rubricscore before you have read a few of its verdicts yourself. - Running promptfoo with a single prompt. The point is the comparison, so keep the old version in the file as the baseline.
Try it yourself
- Run the example once as it is and open the viewer.
- Replace the two prompts with your current prompt and the rewrite from lesson 2.
- Replace the two tests with five of your own inputs, one assertion each.
- Run it, change one line of the rewrite, and run it again.
Next: automatic prompt optimization with DSPy
You have compared versions by hand. Lesson 4 lets DSPy search for better versions itself, scored by a metric much like the checks above.
Check yourself
In a promptfoo prompts file, how do you put two prompt versions in one file?
Show answer
C. Put a line of three dashes between them The docs say to put three dashes on their own line between prompts in a .txt file, and every test then runs on each one.
In a promptfoo CSV, what does a value with no prefix in the __expected column mean?
Show answer
B. equals The test cases page says values without a type prefix default to equals.
In Anthropic's evaluation course, how do you see two prompt versions' outputs and scores side by side?
Show answer
D. Press Add Comparison and pick the earlier version The course presses Add Comparison, selects version 1 and then reads the outputs and human-graded scores of both versions together.
Which promptfoo check needs a model provider to do the grading?
Show answer
C. llm-rubric The assertions page says some assertions, including similarity and llm-rubric, require an LLM provider.
Frequently asked questions
How do I test a prompt?
Run each version on the same set of real inputs, then compare the replies against a pass rule you wrote first. A spreadsheet, Anthropic's Console and promptfoo do this at three different scales.
What is promptfoo, and what else does it do?
It is an open-source tool, listed under the MIT license on npm. Besides comparing prompts, its intro lists automated red teaming and use in CI/CD. OpenAI's cookbook names it as the way to carry on after the Evals product.
How many test cases do I need?
The sources set no target. OpenAI's prompt optimizer asks for at least three dataset rows, and promptfoo's example has two tests. Start small, then add a case each time a prompt fails in real use.
Can promptfoo test Claude and other vendors' models?
Yes. Its docs list more than 60 providers, including OpenAI, Anthropic and Google, and open-source models such as Llama.
Do I need to code to test prompts?
No for a spreadsheet or the Console's evaluate view. promptfoo needs a terminal and a YAML file, but its test cases are declarative, so they read like a table.
Sources
- Anthropic on GitHub: Prompt evaluations course, lesson 2, Workbench evaluations https://github.com/anthropics/courses/blob/master/prompt_evaluations/02_workbench_evals/02_workbench_evals.ipynb
- promptfoo docs on GitHub: Intro https://github.com/promptfoo/promptfoo/blob/main/site/docs/intro.md
- promptfoo docs on GitHub: Getting started https://github.com/promptfoo/promptfoo/blob/main/site/docs/getting-started.md
- promptfoo docs on GitHub: Prompt configuration https://github.com/promptfoo/promptfoo/blob/main/site/docs/configuration/prompts.md
- promptfoo docs on GitHub: Test cases https://github.com/promptfoo/promptfoo/blob/main/site/docs/configuration/test-cases.md
- promptfoo docs on GitHub: Assertions and metrics https://github.com/promptfoo/promptfoo/blob/main/site/docs/configuration/expected-outputs/index.md
- promptfoo docs on GitHub: Command line https://github.com/promptfoo/promptfoo/blob/main/site/docs/usage/command-line.md
- promptfoo on GitHub: eval-self-grading example https://github.com/promptfoo/promptfoo/tree/main/examples/eval-self-grading
- OpenAI API docs: Prompt optimizer https://developers.openai.com/api/docs/guides/prompt-optimizer
- npm registry: promptfoo package record https://registry.npmjs.org/promptfoo/latest
- OpenAI Cookbook: Moving from OpenAI Evals to Promptfoo https://developers.openai.com/cookbook/examples/evaluation/moving-from-openai-evals-to-promptfoo
- YouTube: Anthropic, Evaluate prompts in the Anthropic Console https://www.youtube.com/watch?v=KIGBsQqZcNA