How to evaluate RAG: retrieval metrics, faithfulness and a small test set

Before you start

You need Python 3, a pipeline that exposes its retrieved chunks and its answer, such as the one from lesson 1, and an API key for the judge model. Install Ragas with pip install ragas. The examples on the Ragas pages call OpenAI, and the testset guide shows how to wrap AWS, Google Cloud, Azure or any other LangChain provider with LangchainLLMWrapper.

One version note. The Ragas pages this lesson follows are dated 9 December 2025. They still say the older ragas.metrics classes, such as LLMContextRecall, form a legacy API that "will be deprecated in version 0.4" and removed in 1.0. PyPI shows that 0.4.0 had already shipped on 3 December 2025, and on 5 November 2026 the newest release was 0.4.3, from 13 January 2026. The metric code below uses the ragas.metrics.collections API the docs recommend for new projects. Pin the version you test with, for example pip install ragas==0.4.3, so an upgrade cannot shift your scores between two runs.

Test retrieval and the answer separately

A wrong answer has two possible causes. The retriever may not have found the passage, or the model may have ignored or distorted it. Ragas splits its metrics along that line, so a score tells you which half to fix. Context recall and context precision grade the retrieved chunks. Faithfulness and answer relevancy grade the response. Noise sensitivity looks at both.

Build a small evaluation set

Ragas wants four fields for each question. user_input is the question, retrieved_contexts the chunks your retriever returned, response the answer your model gave and reference a golden answer. Its getting-started guide collects them by running each query through the pipeline and loading the rows into an EvaluationDataset. It calls the reference optional, but three of the metrics above take one.

Anthropic's guidance on building evals carries over. Mirror the real mix of your task. Include edge cases such as irrelevant or nonexistent input, overly long input and ambiguous cases where even humans would struggle to agree. Prefer more questions with automated grading over fewer that people grade by hand.

from ragas import EvaluationDataset

# sample_queries and expected_responses: your own questions and reference answers
dataset = []

for question, reference in zip(sample_queries, expected_responses):
    chunks = retrieve(question)          # your retrieval step: a list of chunk texts
    response = answer(question, chunks)  # your generation step
    dataset.append(
        {
            "user_input": question,
            "retrieved_contexts": chunks,
            "response": response,
            "reference": reference,
        }
    )

evaluation_dataset = EvaluationDataset.from_list(dataset)

1. Write down the questions your users ask, including a few your documents cannot answer.
2. Write a reference answer for each, one or two sentences taken from your documents.
3. Run each question through your pipeline and keep the chunks it retrieved and the response it gave.
4. Load the rows into an EvaluationDataset, or keep them as a list of dicts.
5. Freeze the set. Add a row whenever you meet a new failure, and never drop a row to make a score look better.

Score retrieval: recall and precision

Context recall asks whether the retriever missed anything. Ragas breaks the reference into claims and checks each one against the retrieved context, so the score is the claims the context supports divided by the claims in the reference. Because it compares against something, it always needs a reference.

Context precision asks whether the useful chunks come first. It is the mean of precision@k over the retrieved chunks, where precision@k counts the true positives among the first k chunks. Order matters, and the Ragas page shows how much. With one relevant chunk and one irrelevant chunk, its Context Utilization score halves when the irrelevant chunk comes first.

from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import ContextPrecision, ContextRecall

client = AsyncOpenAI()  # reads OPENAI_API_KEY
llm = llm_factory("gpt-4o-mini", client=client)  # the judge model

recall = ContextRecall(llm=llm)
precision = ContextPrecision(llm=llm)

recalls, precisions = [], []
for row in dataset:
    recalls.append(
        recall.score(
            user_input=row["user_input"],
            retrieved_contexts=row["retrieved_contexts"],
            reference=row["reference"],
        ).value
    )
    precisions.append(
        precision.score(
            user_input=row["user_input"],
            reference=row["reference"],
            retrieved_contexts=row["retrieved_contexts"],
        ).value
    )

print("context recall", sum(recalls) / len(recalls))
print("context precision", sum(precisions) / len(precisions))

• No reference answer: ContextUtilization compares each chunk with the generated response instead.
• No LLM calls: NonLLMContextRecall and NonLLMContextPrecisionWithReference compare the retrieved chunks with reference_contexts through a string similarity measure such as Levenshtein distance. The precision one needs the rapidfuzz package.
• By ID: IDBasedContextRecall and IDBasedContextPrecision compare retrieved_context_ids with reference_context_ids. In the docs, 1 of 4 reference IDs retrieved gives a recall of 0.25, and 2 of 4 retrieved IDs found in the reference gives a precision of 0.5.

Score the answer: faithfulness and relevancy

Faithfulness splits the response into statements and checks each against the retrieved context. The Ragas example gives a context saying Einstein was born on 14 March 1879. The answer "Einstein was born in Germany on 20th March 1879" splits into two statements. The first is supported and the second is not, so the score is 1 of 2, one half.

Answer relevancy works from the question and the response alone. A model writes artificial questions from the response, three by default, and the score is the mean cosine similarity between their embeddings and the embedding of the real question. By design it ignores factual accuracy and penalizes answers that are incomplete or carry unnecessary detail. The score usually falls between 0 and 1, but cosine similarity can go as low as -1, so Ragas does not guarantee the range.

from ragas.embeddings.base import embedding_factory
from ragas.metrics.collections import AnswerRelevancy, Faithfulness

embeddings = embedding_factory("openai", model="text-embedding-3-small", client=client)

faithfulness = Faithfulness(llm=llm)
relevancy = AnswerRelevancy(llm=llm, embeddings=embeddings)

faithful, relevant = [], []
for row in dataset:
    faithful.append(
        faithfulness.score(
            user_input=row["user_input"],
            response=row["response"],
            retrieved_contexts=row["retrieved_contexts"],
        ).value
    )
    relevant.append(
        relevancy.score(user_input=row["user_input"], response=row["response"]).value
    )

print("faithfulness", sum(faithful) / len(faithful))
print("answer relevancy", sum(relevant) / len(relevant))

• NoiseSensitivity is the share of claims in the response that the reference does not support, so lower is better. Ragas computes it over the relevant chunks by default and over the irrelevant ones with mode="irrelevant".
• Ragas's example gives a score of 0.333, one incorrect claim out of three, for a response that adds a claim the reference never makes.
• Faithfulness has a variant, FaithfulnesswithHHEM, that replaces the claim check with Vectara's HHEM-2.1-Open classifier, which Ragas describes as small, free and open source.

Read the scores and pick a lever

A low score points at one half of the pipeline. Low context recall means the passage never reached the model. Low context precision with good recall means it was found but ranked below noise. Low faithfulness with good retrieval points at generation, and LlamaIndex lists prompt changes among the first things to try for hallucinations or badly formatted output.

When the judge needs checking

Most of the metrics above use a model as the judge. Anthropic ranks grading methods by speed, reliability and scale and says to choose the fastest, most reliable and most scalable one that works. Code-based grading is the fastest and most reliable but lacks nuance. Human grading is the most flexible and slow and expensive. LLM-based grading is fast and flexible, and Anthropic says to test it for reliability first, then scale.

• Write a detailed, clear rubric, for example that the answer must mention a named product in its first sentence, or it counts as incorrect.
• Make the output specific. Tell the judge to answer only "correct" or "incorrect", or to score from 1 to 5.
• Let the judge reason first. Anthropic says a grader with thinking turned on does better, particularly on complex judgments.

Generate more test questions

Writing questions by hand does not scale, so Ragas can generate them from your documents. Its guide describes two steps. It builds a knowledge graph from your documents and enriches it with extractors such as entities and keyphrases, then it turns the graph into scenarios that become questions with reference answers. Export the result with to_pandas() and keep the questions you judge fit. We'd read each generated question and its reference before trusting it, since a model wrote both.

The concepts page sorts RAG queries into four kinds. The default distribution is half single-hop specific questions and a quarter each of multi-hop abstract and multi-hop specific.

import openai
from langchain_community.document_loaders import DirectoryLoader
from langchain_openai import ChatOpenAI
from ragas.embeddings import OpenAIEmbeddings
from ragas.llms import LangchainLLMWrapper
from ragas.testset import TestsetGenerator

docs = DirectoryLoader("docs/", glob="**/*.md").load()

generator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o"))
generator_embeddings = OpenAIEmbeddings(client=openai.OpenAI())

generator = TestsetGenerator(llm=generator_llm, embedding_model=generator_embeddings)
dataset = generator.generate_with_langchain_docs(docs, testset_size=10)
dataset.to_pandas()

• Single-hop specific: one source answers a fact, such as the year Einstein published the theory of relativity.
• Single-hop abstract: one source, but the answer is interpretive, such as how the theory changed our view of time and space.
• Multi-hop specific: two or more sources, such as which scientist influenced Einstein's work and what theory they proposed.
• Multi-hop abstract: several sources combined into a broad answer, such as how theories of relativity have evolved since the original publication.

Shreya Shankar on checking the judge, Hamel Husain on generic metrics

Shreya Shankar, an incoming assistant professor of computer science at Carnegie Mellon by her X bio, called evals "arguably the hardest part of LLMOps" on X on 22 April 2024. Models make mistakes, so we check them with other models, and she asked who validates the validators. Her team's paper names an effect it calls criteria drift: people need criteria to grade outputs, but grading outputs helps them define the criteria.

So grade a sample of your rows by hand before you trust faithfulness, or a judge of your own, on the rest. Each row where you and the judge disagree points to a criterion the rubric lacks. Expect the rubric to change as you read, as it did for the people in the paper's study.

Hamel Husain, who works on evals at OpenAI by his X bio, warned on X on 4 June 2024 against generic evals such as conciseness, toxicity and factual correctness run on data nobody reads. In July 2026 he wrote on X about tests he and a co-author ran on real production data, of tools that scan traces with AI. The tools often caught issues people missed but missed problems that need domain expertise and taste, so he advises using them with you in the loop. Before you change a setting, read the rows behind your lowest faithfulness and context recall.

Screenshots of X's own embeds of the posts, taken on 5 November 2026, with the time in UTC. X's embed leaves out the paper link at the end of Shreya Shankar's post. Select one to open the post on X.

Common mistakes

• Scoring only the final answer. Without the retrieved chunks you cannot tell a retrieval miss from a generation error.
• Asking for context recall without a reference. It always needs one.
• Treating answer relevancy as accuracy. It ignores factual accuracy by design, so read it beside faithfulness.
• Reading one run on a handful of questions as a trend, when a single flipped answer moves every mean.
• Switching the judge model between two runs you want to compare. Keep the judge fixed so only your pipeline changes.
• Copying an old tutorial that imports classes from ragas.metrics and calls single_turn_ascore. The docs call that the legacy API.

Try it yourself

• Write ten questions for a set of documents you know, two of them unanswerable, and add reference answers.
• For three questions, mark by hand which chunk IDs hold the answer and compute ID-based recall as found IDs over reference IDs.
• Rerun the set after you change the chunk size from lesson 3. Compare the means, then the per-question rows.
• Move an irrelevant chunk to the front of retrieved_contexts for one row and watch context precision fall.

Next

This is the last lesson of the RAG course. The next course, advanced prompt engineering, starts the AI engineering path: its first lesson teaches a model with examples, which is also how you steer the answers your RAG pipeline writes. To compare other people's work with your own questions, browse the agents on Mindket.

Frequently asked questions

Sources

1. Ragas Docs: Installation (read November 4, 2026)https://docs.ragas.io/en/stable/getstarted/install/
2. Ragas Docs: Evaluate a simple RAG system (read November 4, 2026)https://docs.ragas.io/en/stable/getstarted/rag_eval/
3. Ragas Docs: Context Recall (read November 5, 2026)https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/
4. Ragas Docs: Context Precision (read November 4, 2026)https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/
5. Ragas Docs: Faithfulness (read November 4, 2026)https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/
6. Ragas Docs: Response Relevancy (read November 4, 2026)https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/answer_relevance/
7. Ragas Docs: Noise Sensitivity (read November 4, 2026)https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/noise_sensitivity/
8. Ragas Docs: Testset Generation for RAG (guide) (read November 4, 2026)https://docs.ragas.io/en/stable/getstarted/rag_testset_generation/
9. Ragas Docs: Testset Generation for RAG (concepts) (read November 4, 2026)https://docs.ragas.io/en/stable/concepts/test_data_generation/rag/
10. Claude Docs: Define success criteria and build evaluations (read November 4, 2026)https://platform.claude.com/docs/en/test-and-evaluate/develop-tests
11. LlamaIndex Docs: Basic strategies (read November 5, 2026)https://developers.llamaindex.ai/python/framework/optimizing/basic_strategies/basic_strategies/
12. PyPI: ragas release history (read November 5, 2026)https://pypi.org/project/ragas/
13. Shreya Shankar on X: Who validates the validators? (read November 5, 2026)https://x.com/sh_reya/status/1782425960366940333
14. arXiv: Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences (read November 5, 2026)https://arxiv.org/abs/2404.12272
15. Hamel Husain on X: Generic evals (read November 5, 2026)https://x.com/HamelHusain/status/1798024532995052010
16. Hamel Husain on X: Do Automated Evals Work? (14 July 2026) (read November 5, 2026)https://x.com/HamelHusain/status/2077042379392213377