Mindket
English
English简体中文
Sign inSign up
StoreMCPSkillsCalculatorLearnPriceBlog
Discover
Research308Writing59Business95Legal13Finance & accounting33Marketing & SEO172Developer1,379Productivity382Education81Companions14
Help and support
Language
English简体中文
IntermediateFree

AI evals: measure whether your LLM app and agent work

Mindket editors · Updated · 4 lessons · 39 min

A black banner titled "AI evals" over the line "Test sets, LLM judges and agent evals".

This free AI evals course has four lessons for people who ship LLM apps or AI agents. You will turn real failures into a test set and grade it in code or with an LLM judge you check against people. Then you score an agent's whole run and run the evals in CI, so a prompt change that lowers the score fails the build before it merges.

What you'll be able to do

  • Explain what an eval is and build a first test set from your app's own failures.
  • Write an LLM-as-a-judge grader and check how often it agrees with people.
  • Evaluate an AI agent on its final result and on the steps it took.
  • Run evals in CI on every change, so a change that makes results worse fails the build.
All courses

Who it's for

For developers and product people who ship an LLM app or an AI agent. You need Python 3 for lessons 1 to 3, Node.js 22.22.0 or newer for promptfoo in lesson 4, and an API key for the model you test and for a judge model.

Lessons

  1. 1What are LLM evals? A first test set from real failuresWhat LLM evals are and how to start one: read real traces, turn the failures into a first test set and grade each check with code or a model.9 min
  2. 2LLM as a judge: a pass/fail judge prompt you check against peopleHow to use an LLM as a judge: write a pass/fail judge prompt, measure how often it agrees with people and design around position and length bias.10 min
  3. 3How to evaluate AI agents: end states, trials and tracesHow to evaluate AI agents: tasks with an end state code can check, several trials scored as pass@k or pass^k, trace grading and reading transcripts.10 min
  4. 4Evals in CI: run LLM evals on every pull request with GitHub ActionsRun LLM evals in CI on every pull request: a promptfoo config in your repo, its GitHub Action, a pass-rate gate, repeats for flaky graders and a cache.10 min

Frequently asked questions

Is this AI evals course free?

Yes. Every lesson is free to read. Running the evals calls model APIs, and those providers may bill for their use.

What is an eval in AI?

An eval is a test for an AI system: an input, a way to grade the output and a score you track over time. It tells you whether a change to a prompt, a model or a tool made things better or worse.

Is an LLM judge reliable?

Only once you check it. Lesson 2 has you compare the judge with labels from people and fix its prompt until they agree, because a judge can be wrong in its own ways.

How is this different from the RAG evaluation lesson?

The RAG lesson scores retrieval and answers with Ragas. This course covers evals for any LLM app or agent, including multi-step agents and evals that run in CI.

What should I learn after this course?

The n8n course is next. Its first lesson builds a scheduled four-node workflow that logs the weather, and its third adds an AI Agent node, where a prompt you have tested can run.

Share this course

Test an agent before you rely on it

Browse the agents on Mindket, each listed with a preview, and try them on your own questions before you build your own.

Browse agents

All courses · All articles