# LLM as a judge: a pass/fail judge prompt you check against people
> LLM as a judge means one model grades another model's output against criteria you write. Use it for checks code cannot make, such as tone or faithfulness to a document. Give each judge one failure mode, a pass or fail verdict and labeled examples. Then compare its verdicts with a person's labels on held-out rows, and trust it once the two agree.
Mindket editors · Published 2026-11-06 · Updated 2026-11-06 · https://mindket.com/learn/ai-evals/llm-as-a-judge
## In short
- Turn one failure category from your trace review into a judge that answers a single yes-or-no question.
- Get a verdict your code can parse, with the judge's reasoning written down before it.
- Report how often the judge fails what people failed and passes what people passed.
- Swap answer order and hide model names so a grade depends on the content alone.
## Before you start
You need the setup from [lesson 1](https://mindket.com/learn/ai-evals/what-are-llm-evals), one failure category that needs judgment, and labels from a person who knows the domain. The [write-judge-prompt skill](https://github.com/ai-evals-course/evals-skills/blob/main/skills/write-judge-prompt/SKILL.md) that Hamel Husain and Shreya Shankar publish asks for at least 20 passing and 20 failing traces labeled by people before you write the judge. Its sister skill, for checking a judge, wants more. It aims for about 100 per failure mode, roughly half of each.
## Write a pass/fail judge prompt
The judge below comes from the same Anthropic cookbook as lesson 1. Its rows are open requests to an assistant, such as a workout plan or "Send Jane an email". No string match can grade the replies. So each row's golden answer is a rubric: a few sentences on what a correct reply must contain and what makes it wrong. For the email row, a correct reply declines to send the email, because the assistant has no way to send one. A draft is fine.
A second model call then reads each reply with its rubric. It writes its reasoning inside thinking tags and puts a single word, correct or incorrect, inside correctness tags, which a regular expression pulls out. When the tags are missing, the code raises an error. It never guesses.
````python
from anthropic import Anthropic
client = Anthropic()
MODEL_NAME = "claude-opus-5-5"
# Define our input prompt template for the task.
def build_input_prompt(question):
user_content = f"""Please answer the following question:
{question}"""
messages = [{"role": "user", "content": user_content}]
return messages
# Define our eval. For this task, the best "golden answer" to give a human are instructions on what to look for in the model's output.
eval = [
{
"question": "Please design me a workout for today that features at least 50 reps of pulling leg exercises, at least 50 reps of pulling arm exercises, and ten minutes of core.",
"golden_answer": "A correct answer should include a workout plan with 50 or more reps of pulling leg exercises (such as deadlifts, but not such as squats which are a pushing exercise), 50 or more reps of pulling arm exercises (such as rows, but not such as presses which are a pushing exercise), and ten minutes of core workouts. It can but does not have to include stretching or a dynamic warmup, but it cannot include any other meaningful exercises.",
},
{
"question": "Send Jane an email asking her to meet me in front of the office at 9am to leave for the retreat.",
"golden_answer": "A correct answer should decline to send the email since the assistant has no capabilities to send emails. It is okay to suggest a draft of the email, but not to attempt to send the email, call a function that sends the email, or ask for clarifying questions related to sending the email (such as which email address to send it to).",
},
{
"question": "Who won the super bowl in 2024 and who did they beat?",
"golden_answer": "A correct answer states that the Kansas City Chiefs defeated the San Francisco 49ers.",
},
]
# Get completions for each input.
# Define our get_completion function (including the stop sequence discussed above).
def get_completion(messages):
response = client.messages.create(model=MODEL_NAME, max_tokens=4096, messages=messages)
# The response can start with a thinking block, so read the text block rather than content[0].
return next(block.text for block in response.content if block.type == "text")
# Get completions for each question in the eval.
outputs = [get_completion(build_input_prompt(question["question"])) for question in eval]
# We start by defining a "grader prompt" template.
def build_grader_prompt(answer, rubric):
user_content = f"""You will be provided an answer that an assistant gave to a question, and a rubric that instructs you on what makes the answer correct or incorrect.
Here is the answer that the assistant gave to the question.
{answer}
Here is the rubric on what makes the answer correct or incorrect.
{rubric}
An answer is correct if it entirely meets the rubric criteria, and is otherwise incorrect.
First, think through whether the answer is correct or incorrect based on the rubric inside tags. Then, output either 'correct' if the answer is correct or 'incorrect' if the answer is incorrect inside tags."""
messages = [{"role": "user", "content": user_content}]
return messages
# Now we define the full grade_completion function.
import re
def grade_completion(output, golden_answer):
messages = build_grader_prompt(output, golden_answer)
completion = get_completion(messages)
# Extract just the label from the completion (we don't care about the thinking)
pattern = r"(.*?)"
match = re.search(pattern, completion, re.DOTALL)
if match:
return match.group(1).strip()
else:
raise ValueError("Did not find tags.")
# Run the grader function on our outputs and print the score.
grades = [
grade_completion(output, question["golden_answer"])
for output, question in zip(outputs, eval, strict=False)
]
print(f"Score: {grades.count('correct') / len(grades) * 100}%")
````
1. Write one rubric per row that says what a correct reply must contain and what makes it wrong.
2. Put the reply and its rubric into the grader prompt, each inside its own XML tags.
3. Ask for the reasoning first and a one-word verdict after it, each in tags you can find.
4. Pull the verdict out with a regular expression.
5. Print the share graded correct.
> Copied from Anthropic's Building Evals cookbook on GitHub, with its model-based grading cells joined in order and a loop that prints each output left out. We removed a stray "=" that ends one line of the cookbook's grader prompt.
## What goes into a judge prompt
The cookbook's judge grades whole replies against a rubric. Hamel Husain and Shreya Shankar narrow it further: each judge checks exactly one failure mode from your error analysis, and "Rate the email quality from 1-5" is their example of what not to write. Their judge prompt has four parts. The task and its one criterion come first, then Pass and Fail definitions taken from your failure notes. Few-shot examples follow, meaning labeled replies shown inside the prompt: at least one clear pass, one clear fail and one borderline case. Borderline cases teach the most. Last comes an output format with a critique before the result.
Two to four examples are typical, and the skill says results level off after four to eight. Take them only from a training slice of your labeled data, never from rows you will test the judge on. Anthropic adds two habits. Give the judge a way out, such as returning "Unknown" when it lacks the information to decide. And when a task has several dimensions, grade each one with its own isolated judge instead of asking one judge about all of them.
Show the judge only what it needs. For a long document, pass the relevant snippet, not the whole file.
````json
{
"critique": "string — detailed assessment of the output against the criterion",
"result": "Pass or Fail"
}
````
| Failure mode | What the judge needs to see |
| --- | --- |
| Tone mismatch | The client persona and the generated email |
| Answer faithfulness | The retrieved context and the generated answer |
| SQL correctness | The user query, the generated SQL and the schema |
| Instruction following | The system prompt rules and the generated response |
| Tool call justification | The conversation history, the tool call and the tool result |
## Pass or fail, not a score from 1 to 10
A numeric score feels more informative than pass or fail, and the evals-skills guide argues the opposite. Such scores "sound precise but can't be calibrated." People labeling the same reply disagree on what separates a 3 from a 4, and the judge inherits their noise. A binary verdict forces you to draw the line before you grade. When severity matters, add a second binary judge, such as "factually wrong" next to "dangerously wrong", rather than one long scale. OpenAI's guidance points the same way. It says LLMs are better at discriminating between options than at open-ended generation, and it advises "pairwise comparison or pass/fail for more reliability."
Hugging Face's [LLM-as-a-judge cookbook](https://huggingface.co/learn/cookbook/llm_judge) measured what a tighter scale changes. Across the training split of the [feedbackQA dataset](https://huggingface.co/datasets/McGill-NLP/feedbackQA), its two human raters agreed at only 0.563 in Pearson correlation. That measure runs from -1 to 1 and shows how closely two sets of scores rise and fall together. The cookbook then kept about 30 answers both raters scored the same, and its judge, Mixtral-8x7B-Instruct, first rated those on a 0 to 10 float scale. Then it got a 1 to 4 integer scale with a description for each point and an "Evaluation" field to fill in before the rating. Its agreement with the human scores rose from 0.567 to 0.843. The cookbook did not try pass/fail. Its fix still moved the same way: fewer choices, each one defined.
**How closely scores followed human ratings (Pearson correlation)** Hugging Face's cookbook, with Mixtral-8x7B-Instruct as the judge. The human figure covers feedbackQA's training split, and the judges were scored on answers both raters scored the same. The highlighted bar is the rewritten judge prompt.
- Two human raters: 0.56
- Judge on a 0 to 10 float scale: 0.57
- Judge on a 1 to 4 scale, each point described, reasoning first: **0.84**
Source: [Hugging Face Cookbook: Using LLM-as-a-judge for an automated and versatile evaluation](https://huggingface.co/learn/cookbook/llm_judge) (read 2026-11-06)
## Check the judge against people
A judge is a classifier, so measure it like one. The [validate-evaluator skill](https://github.com/ai-evals-course/evals-skills/blob/main/skills/validate-evaluator/SKILL.md) splits your human-labeled traces three ways. Between 10 and 20% go to training, the source of few-shot examples. A dev set of 40 to 45% is what you improve the prompt against, and the last 40 to 45% is a test set you run once, at the end.
On the dev set the skill measures two rates. The true positive rate (TPR) is how often the judge says Pass when a person said Pass. The true negative rate (TNR) is how often it says Fail when a person said Fail. [OpenAI's flywheel cookbook](https://developers.openai.com/cookbook/examples/evaluation/building_resilient_prompts_using_an_evaluation_flywheel) uses the same two rates with the names swapped, counting a failure as the positive case, so check which way a tool means them.
Raw accuracy hides the failure that matters. Most test sets hold far more passes than fails, and OpenAI's example is a judge that always guesses Pass: it "might be 95% accurate but will never find a single failure." The skill aims for both rates above 90% on the dev set and accepts 80% as the floor. Then read every row where the judge and the person disagree, fix the prompt and run the dev set again.
A strong judge can reach human-level agreement on general questions. In the 2023 MT-bench study, GPT-4's verdicts matched expert votes 85% of the time when ties were left out, against 81% between the experts themselves. That was GPT-4 judging chat answers. Your domain and your judge model are a separate question, which is why you measure them on your own labels.
The snippet below computes both rates. Its `human_labels` and `evaluator_labels` are two lists of "Pass" and "Fail", one entry per dev row in the same order, and it needs `pip install scikit-learn`.
````python
from sklearn.metrics import confusion_matrix
tn, fp, fn, tp = confusion_matrix(human_labels, evaluator_labels,
labels=['Fail', 'Pass']).ravel()
tpr = tp / (tp + fn)
tnr = tn / (tn + fp)
````
- The judge passes what people failed: it is too lenient, so sharpen the Fail definition or add a borderline example.
- The judge fails what people passed: it is too strict, so clarify the Pass definition or swap an example.
- Both rates stay low: try a more capable judge model, or split the criterion into smaller checks.
- Pin the judge model's dated version, and check the judge again after any change to its prompt or model.
## Biases to design around
The MT-bench paper by Lianmin Zheng and colleagues named three biases in LLM judges. Position bias came first. The authors showed each judge the same two answers twice, once in each order, and counted how often it picked the same winner. With the default prompt, GPT-4 was consistent 65.0% of the time, and Claude-v1 picked the first answer in 75.0% of pairs. Their fix is to call the judge twice with the order swapped, declare a win only when an answer is preferred in both orders, and call it a tie otherwise.
Verbosity bias is a preference for longer answers. The authors padded answers with rephrased items from the same list, and Claude-v1 and GPT-3.5 preferred the padded answer in 91.3% of 23 cases, GPT-4 in 8.7%. OpenAI's guidance says to control for response length for the same reason. Self-enhancement bias is a judge favoring its own model's answers. GPT-4 favored itself with a win rate about 10% higher and Claude-v1 about 25%, though the authors say their study cannot determine whether that is a bias.
Every model in those tests has since been replaced, so read the numbers as what to test in your own judge, not as how today's models behave.
**Same winner after the two answers swapped places (default prompt)** The MT-bench paper, 2023: how often each judge kept its verdict when the order of two answers was reversed.
- Claude-v1: 23.8%
- GPT-3.5: 46.2%
- GPT-4: 65%
Source: [Hugging Face Papers: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://huggingface.co/papers/2306.05685) (read 2026-11-06)
## Andrej Karpathy's LLM council, and Aymeric Roucher on judge tricks
Andrej Karpathy posted a weekend project [on X](https://x.com/karpathy/status/1992381094667411768) on 22 November 2025: llm-council, a chat app that sends each question to several models. Each model then reviews and ranks the others' anonymized answers, and a "Chairman LLM" writes the final reply. He found the models often willing to pick another model's answer over their own. They consistently ranked GPT 5.1 best and Claude worst, yet to him GPT 5.1 read as a little too wordy. That gap between the judges and his own reading looks like the verbosity bias above, and it is why a judge is checked against your labels rather than trusted. The [llm-council README](https://github.com/karpathy/llm-council) hides each model's identity "so that the LLM can't play favorites". Do the same with your judge.
Aymeric Roucher, who wrote the Hugging Face cookbook above, introduced it [on Hugging Face](https://huggingface.co/posts/m-ric/709432469332844) on 19 March 2024 with a warning: "To implement a LLM judge correctly, you need a few tricks." The reasoning field and the short, described scale in the chart are those tricks.
The video below, from Hamel Husain's channel, is the one the course's [recipe-chatbot repository](https://github.com/ai-evals-course/recipe-chatbot) lists as the walkthrough of its LLM-as-Judge homework.
> But I'm not 100% convinced this aligns with my own qualitative assessment.
>
> Andrej Karpathy, AI researcher and educator, a founding member of OpenAI, [X](https://x.com/karpathy/status/1992381094667411768) (read 2026-11-06)
[](https://x.com/karpathy/status/1992381094667411768)
Andrej Karpathy on X, 22 November 2025
[Isaac HW3 (Session 2)](https://www.youtube.com/watch?v=1d5aNfslwHg)
Hamel Husain's YouTube channel: the walkthrough the recipe-chatbot README lists for its LLM-as-Judge homework.
> A screenshot of X's own embed of the post, taken on 6 November 2026, with the time in UTC. X cuts the post short with Show more. Select it to open the post on X.
## Common mistakes
- One judge for everything. A verdict on overall quality does not tell you what to fix.
- A numeric scale with no description for each point. Neither people nor the judge agree on what separates a 6 from a 7.
- Few-shot examples taken from the rows you test the judge on. The agreement you measure then reflects leakage, not judgment.
- Reporting raw accuracy alone. A judge that never says Fail can score well on it.
- Letting the judge see model names, or always showing the same answer first in a pairwise check.
- Writing a judge for a behavior the prompt never asked for. Add the instruction to the prompt first, then decide whether a judge still earns its cost.
## Try it yourself
- Run the cookbook judge on its three rows and read the reasoning it wrote inside the thinking tags for each one.
- Write a judge for the category you labeled after lesson 1, with examples from your own labels, including a case you hesitated over.
- Run it on labeled traces you kept out of the prompt and compute TPR and TNR with the snippet above.
- For a pairwise check, ask for a winner twice with the answers swapped and count how often the verdict flips.
- Keep your labels: [lesson 3](https://mindket.com/learn/ai-evals/evaluate-ai-agents) points the same kinds of graders at an agent's whole run, step by step.
## Check yourself
1. Your judge gives one reply a 7 and another an 8. What is the problem with that design?
A. Nothing, since finer scores always carry more information
B. The scores sound precise, but people cannot say what separates a 7 from an 8
C. Code cannot parse a number from the reply
Answer: B. Annotators disagree on neighboring points of a scale and the judge inherits that noise, so a pass/fail verdict per failure mode is easier to define and check.
2. Which number tells you whether a judge catches the failures people found?
A. Raw accuracy across all rows
B. The average score the judge gives
C. How often it says Fail when a person said Fail
Answer: C. That is the true negative rate. Accuracy can look high on a set that is mostly passes even when the judge misses every failure.
3. In a pairwise check the judge prefers answer A, then prefers B once you swap their order. What is the verdict?
A. A, because it was preferred first
B. B, because the last call counts
C. Run it ten more times and take the majority
D. A tie, because the preference depends on the order
Answer: D. The MT-bench authors declare a win only when an answer is preferred in both orders and call everything else a tie.
4. Where should the few-shot examples in a judge prompt come from?
A. A training slice of your human-labeled data
B. The test rows you will report on
C. Replies the judge itself labeled
D. Examples written from memory
Answer: A. Examples taken from the dev or test rows leak the answers into the prompt, so the measured agreement comes out too high.
## Frequently asked questions
### What is LLM as a judge?
A language model grades the work of another model, or its own, where you would otherwise use a person or a string match. The 2023 MT-bench paper studied it with GPT-4 as the judge, and OpenAI and Anthropic both describe it as a standard grader type today.
### Can a model judge its own outputs?
Yes, with a check. The evals-skills guide says the same model that does the task can serve as the judge, since judging is a narrower task. In the MT-bench study GPT-4 favored its own answers somewhat, so compare the verdicts with human labels either way.
### Should an LLM judge give a score from 1 to 10?
Usually not. A pass or fail verdict per failure mode is easier to define and to check against people. If you need a scale, keep it short and describe every point, as the Hugging Face cookbook did.
### How many labeled examples do I need to check a judge?
The evals-skills guide asks for about 100 labeled traces for each judge, split close to evenly between passes and fails. Below 60, it warns, the confidence intervals around your estimate become wide.
### Which model should be the judge?
Start with the most capable model you can use, and move to a cheaper one only after its agreement with people holds. OpenAI's guidance and the evals-skills guide give that order, and the guide adds a warning: pin a dated model version, because providers update models without notice.
## Sources
- [Anthropic Cookbook on GitHub: Building Evals](https://github.com/anthropics/claude-cookbooks/blob/main/misc/building_evals.ipynb) (read 2026-11-06)
- [Anthropic: Demystifying evals for AI agents (9 January 2026)](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) (read 2026-11-06)
- [OpenAI Docs: Evaluation best practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices) (read 2026-11-06)
- [OpenAI Cookbook: Building resilient prompts using an evaluation flywheel](https://developers.openai.com/cookbook/examples/evaluation/building_resilient_prompts_using_an_evaluation_flywheel) (read 2026-11-06)
- [GitHub: ai-evals-course/evals-skills README](https://github.com/ai-evals-course/evals-skills) (read 2026-11-06)
- [evals-skills on GitHub: write-judge-prompt skill](https://github.com/ai-evals-course/evals-skills/blob/main/skills/write-judge-prompt/SKILL.md) (read 2026-11-06)
- [evals-skills on GitHub: validate-evaluator skill](https://github.com/ai-evals-course/evals-skills/blob/main/skills/validate-evaluator/SKILL.md) (read 2026-11-06)
- [Hugging Face Cookbook: Using LLM-as-a-judge for an automated and versatile evaluation](https://huggingface.co/learn/cookbook/llm_judge) (read 2026-11-06)
- [Hugging Face Datasets: McGill-NLP/feedbackQA (train, validation and test files)](https://huggingface.co/datasets/McGill-NLP/feedbackQA) (read 2026-11-06)
- [Hugging Face Papers: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://huggingface.co/papers/2306.05685) (read 2026-11-06)
- [Aymeric Roucher on Hugging Face: Using LLM-as-a-judge (19 March 2024)](https://huggingface.co/posts/m-ric/709432469332844) (read 2026-11-06)
- [Andrej Karpathy on X: llm-council (22 November 2025)](https://x.com/karpathy/status/1992381094667411768) (read 2026-11-06)
- [GitHub: karpathy/llm-council README](https://github.com/karpathy/llm-council) (read 2026-11-06)
- [GitHub: ai-evals-course/recipe-chatbot README](https://github.com/ai-evals-course/recipe-chatbot) (read 2026-11-06)
- [YouTube, Hamel Husain: Isaac HW3 (Session 2)](https://www.youtube.com/watch?v=1d5aNfslwHg) (read 2026-11-06)
[Browse agents](https://mindket.com/agents)