# Test a Claude skill: trigger checks, evals and a no-skill baseline

> Test an agent skill two ways. A trigger check asks whether the agent loads the skill on the right prompts. An eval asks whether the output is better with the skill than without it. Both need realistic prompts and fresh sessions, and Anthropic's skill-creator plugin automates most of the loop.

Mindket editors · Published 2026-11-04 · Updated 2026-11-05 · https://mindket.com/learn/agent-skills/test-an-agent-skill

## In short

- A trigger check asks whether the skill loads. An eval asks whether the result is better for it. They need different tests.
- A trigger set is about 20 prompts, half that should load the skill and half near-misses that should not, each run three times.
- An eval runs each realistic prompt with the skill and without it, each run in a fresh session.
- An assertion that passes with and without the skill measures the model, not the skill. Drop it.

## Before you start

You need Claude Code, the `reviewing-api-changes` skill from lessons 1 and 2 (any skill works) and `jq` for the script. Claude Code v2.1.233 or later has the validate command used below. Any agent that can run a prompt without a human and show which skills it used follows the same method. The Agent Skills guide says most clients offer logs, tool call histories or verbose output for that.

The sample prompts and test cases were written for this course. The script and the file formats come from the Agent Skills guides, and the commands from Claude Code's documentation, listed under Sources.

> Every fact here comes from the pages under Sources, read on 4 November 2026. This lesson is not hands-on testing.

## Run a trigger check in six steps

A trigger check asks one question: does the agent open the skill on prompts that call for it and leave it alone on prompts that don't? The [Agent Skills guide](https://agentskills.io/skill-creation/optimizing-descriptions) says the most useful positive prompts are the ones where the skill would help but the link isn't obvious from the wording, because those are where the description decides the outcome.

The negatives matter more than they look. "Write a fibonacci function" tests nothing against an API-review skill. A prompt that shares words with it, such as a request to write a new spec, does.

````json
[
  { "query": "my teammate renamed /users/{id} to /accounts/{id} in api/openapi.yaml, is that safe to merge?", "should_trigger": true },
  { "query": "can you diff v1.yaml and v2.yaml of our payments API before we release", "should_trigger": true },
  { "query": "i changed a field from string to integer in the swagger file, anything i should worry about", "should_trigger": true },
  { "query": "write an OpenAPI spec for a new /orders endpoint", "should_trigger": false },
  { "query": "convert this OpenAPI yaml file to json", "should_trigger": false },
  { "query": "explain what a breaking change is in semantic versioning", "should_trigger": false }
]

````

1. Confirm the skill parses. Run `claude plugin validate ~/.claude/skills`, then ask Claude Code `What skills are available?` and check that the skill is listed.
2. Write about 20 prompts into `eval_queries.json`, in the shape above: 8 to 10 that should trigger the skill and 8 to 10 that should not. Vary phrasing, detail and how directly the prompt names the skill's domain.
3. Make the prompts read like real ones. The guide suggests file paths, personal context, specific column or company names, casual language and the occasional typo.
4. Split the set before you start fixing anything: about 60% for training and 40% held back for validation, with positives and negatives in both. Shuffle once and keep the split fixed.
5. Run every prompt three times, each in a fresh session, and record whether the skill loaded. The guide calls three a reasonable starting point, since one prompt can fire the skill on one run and miss on the next.
6. Score it. A should-trigger prompt passes when its trigger rate is above 0.5, and a should-not-trigger prompt passes when its rate is below it.

## Script the runs, then fix the description

Twenty prompts at three runs each make 60 invocations, which is why the guide says to script them. Its example sets the number of runs and the skill name at the top, asks Claude Code for JSON output and looks for a Skill tool call with `jq`. Change `SKILL_NAME` to your skill. For another client, replace the detection inside `check_triggered`.

When prompts fail, edit the description from the training failures only. A should-trigger prompt that misses means the description may be too narrow. A should-not-trigger prompt that fires means it may be too broad, and the fix is to say what the skill does not do. Copying words from a failed prompt is overfitting. Find the general kind of request it stands for.

Keep an eye on length, because descriptions tend to grow during tuning and the specification caps them. If several rounds change nothing, try a different sentence structure. The guide also says five iterations is usually enough, and that a stuck score may mean the queries are the problem, not the description.

````bash
#!/bin/bash
QUERIES_FILE="${1:?Usage: $0 <queries.json>}"
SKILL_NAME="my-skill"
RUNS=3

# This example uses Claude Code's JSON output to check for Skill tool calls.
# Replace this function with detection logic for your agent client.
# Should return 0 (success) if the skill was invoked, 1 otherwise.
check_triggered() {
  local query="$1"
  claude -p "$query" --output-format json 2>/dev/null \
    | jq -e --arg skill "$SKILL_NAME" \
      'any(.messages[].content[]; .type == "tool_use" and .name == "Skill" and .input.skill == $skill)' \
      > /dev/null 2>&1
}

count=$(jq length "$QUERIES_FILE")
for i in $(seq 0 $((count - 1))); do
  query=$(jq -r ".[$i].query" "$QUERIES_FILE")
  should_trigger=$(jq -r ".[$i].should_trigger" "$QUERIES_FILE")
  triggers=0

  for run in $(seq 1 $RUNS); do
    check_triggered "$query" && triggers=$((triggers + 1))
  done

  jq -n \
    --arg query "$query" \
    --argjson should_trigger "$should_trigger" \
    --argjson triggers "$triggers" \
    --argjson runs "$RUNS" \
    '{query: $query, should_trigger: $should_trigger, triggers: $triggers, runs: $runs, trigger_rate: ($triggers / $runs)}'
done | jq -s '.'

````

**The tuning loop from the Agent Skills guide**

1. Evaluate. Score the current description on the training set and on the validation set.
2. Find the failures. Read only the training failures. Keep validation results out of the edit.
3. Revise. Broaden or narrow the description to cover the general case, not the failed wording.
4. Repeat. Go again until the training set passes or the gains stop.
5. Pick the winner. Choose the iteration with the best validation pass rate. It may not be the last one.

## Evals: is the output better than no skill?

Seeing a skill trigger tells you Claude found it, not that it did what you meant. [Claude Code's documentation](https://code.claude.com/docs/en/skills) says to measure the two separately. For output quality the check is a baseline comparison: run each prompt in a fresh session with the skill, run it again with the skill off, and compare. Each run needs a clean context, so the agent follows only what the skill says and not what you remember from writing it.

To turn a personal or project skill off for the second run, set it to `"off"` in `skillOverrides`. That setting doesn't apply to plugin skills. For those, `claude plugin eval` repeats each run with no plugin loaded. When you improve an existing skill, the [evals guide](https://agentskills.io/skill-creation/evaluating-skills) uses the old version as the baseline: copy it to a snapshot folder first, and point the baseline run there.

Start with two or three test cases in `evals/evals.json`. Each has a realistic prompt, a plain-words description of success and any input files. Write assertions after the first round, once you can see what good output looks like. "The output file is valid JSON" can be checked, "the output is good" can't, and an exact required phrase fails correct answers worded differently. Judge tone and design by eye.

````json
{
  "skill_name": "reviewing-api-changes",
  "evals": [
    {
      "id": 1,
      "prompt": "I renamed the status field to state in api/openapi.yaml and made customer_id required on POST /orders. Is this safe to ship?",
      "expected_output": "A review that flags both changes as breaking for existing clients and suggests a safer rollout.",
      "files": ["evals/files/openapi-before.yaml", "evals/files/openapi-after.yaml"],
      "assertions": [
        "The review names the renamed status field as a breaking change",
        "The review flags the new required customer_id field",
        "The review lists each change with a suggested fix"
      ]
    }
  ]
}

````

| Result | What it tells you | What to do |
| --- | --- | --- |
| Passes with and without the skill | The model handles it alone | Remove or replace the assertion |
| Fails in both | The assertion is broken, the case is too hard, or it checks the wrong thing | Fix it before the next iteration |
| Passes with the skill, fails without | The skill adds value here | Find which instruction or script made the difference |
| Passes on some runs only | The case is flaky, or the instructions are open to two readings | Add an example or more specific guidance |

## Let skill-creator run the loop

Once the manual version works, Anthropic's [skill-creator plugin](https://github.com/anthropics/skills/blob/main/skills/skill-creator/SKILL.md) automates the comparison inside Claude Code. In a terminal, install it from the official marketplace with the first line below, then ask for an evaluation in your own words. If Claude Code answers that the marketplace isn't found, run `/plugin marketplace add anthropics/claude-plugins-official` and retry.

Its own description loop follows the guide's numbers: a 60 and 40 split, three runs for each prompt and at most five iterations, choosing the description by its held-out score. Anthropic's [announcement](https://claude.com/blog/improving-skill-creator-test-measure-and-refine-agent-skills) says that running description tuning on its document-creation skills improved triggering on 5 of 6 public ones. It adds a caveat worth keeping. If the base model passes your evals without the skill, the skill's techniques may now be in the model, and the skill may no longer be needed.

Anthropic has two tools for this, and their formats aren't interchangeable. The skill-creator plugin iterates on one skill inside a conversation. For a plugin's skills, `claude plugin eval` scores each run with graders and exits non-zero below a threshold, so a CI job can fail on it.

````text
/plugin install skill-creator@claude-plugins-official

evaluate my reviewing-api-changes skill with skill-creator
````

- Test cases saved in `evals/evals.json`, with one subagent per case so each run starts clean.
- Grading of each assertion into `grading.json`, with the evidence for every pass or fail.
- A benchmark of pass rate, time and tokens, with the skill and without it, in `benchmark.json`.
- A blind A/B comparison of two versions of the skill, to confirm an edit helped before you keep it.
- Description tuning, and an HTML viewer where you read each output and leave feedback for the next round.

**The testing tools for skills, by date**

1. **March 3, 2026** skill-creator learns evals. Anthropic adds evals, benchmarks and description tuning to the plugin.
2. **August 14, 2026** A bare skills folder gets validated. Claude Code 2.1.233 lets `claude plugin validate` check `.claude/skills` and report SKILL.md files whose frontmatter fails to parse.
3. **September 11, 2026** claude plugin eval arrives. Claude Code 2.1.269 runs a plugin's eval suite and writes scored results as JSON and an HTML report.
4. **October 14, 2026** A refused run fails the suite. Claude Code 2.1.299 scores a run that ends on an API error as 0, and the suite exits 1.

Sources: [Claude blog: Improving skill-creator](https://claude.com/blog/improving-skill-creator-test-measure-and-refine-agent-skills) (read 2026-11-05), [Claude Code changelog](https://code.claude.com/docs/en/changelog) (read 2026-11-05)

## Boris Cherny on fresh context windows, and a skill-creator walkthrough

Boris Cherny, who created Claude Code at Anthropic, [explained on X](https://x.com/bcherny/status/2031151689219321886) in March 2026 why a second context window catches what the first one misses. One agent can cause a bug and another, on the same model, can find it, much as a coworker reviewing your code spots what you could not. That is the reason every eval case in this lesson runs in a fresh session, and why skill-creator gives each case its own subagent.

Chase Hannegan's Chase AI channel put out [a video on the skill-creator update](https://www.youtube.com/watch?v=UxfeF4bSBYI) to go with his March 2026 written guide to testing and improving skills. Watch it after one manual round of trigger checks and evals, so you can see which parts of the loop the plugin takes over.

> One way to make the result even better is to use separate context windows.
>
> Boris Cherny, Creator of Claude Code, Anthropic, [X](https://x.com/bcherny/status/2031151689219321886) (read 2026-11-05)

[![Boris Cherny (@bcherny) on X, 9 March 2026. Roughly, the more tokens you throw at a coding problem, the better the result is. We call this test time compute. One way to make the result even better is to use separate context windows.](https://mindket.com/blog/shots/x-bcherny-separate-context-windows.webp?v=55a5159fc6e0)](https://x.com/bcherny/status/2031151689219321886)

Boris Cherny on X, 9 March 2026

[Claude Code Skills Just Got a MASSIVE Upgrade](https://www.youtube.com/watch?v=UxfeF4bSBYI)

Chase AI on YouTube: the video that goes with its guide to testing and improving skills with skill-creator.

> A screenshot of X's own embed of the post, taken on 5 November 2026, with the time in UTC. X cuts the post short with Show more. Select it to open the post on X.

## Common mistakes

- Testing in the session where you wrote the skill. Your own context fills the gaps the instructions leave.
- Trusting one run. The same prompt can load the skill once and miss the next time.
- Writing easy negatives. "What's the weather today?" has no keyword overlap and proves nothing.
- Tuning on every prompt you have. With no held-out set, the description fits the prompts you tuned on, and the best round may be an earlier one.
- Testing on one model. Anthropic's guide says to try every model you plan to use, because what works for Opus may need more detail for Haiku.

## Try it yourself

Grow the six-prompt sheet from lesson 1 into a full trigger set, with as many near-misses as prompts that should load the skill. Run each prompt three times and write the trigger rate beside it. Then write two eval cases for the skill, run each with the skill and with it switched off, and mark which assertions change.

Keep what you find. A pass that appears only with the skill shows you what the skill is worth. A pass that appears in both runs shows you what you can cut.

- Does every should-trigger prompt fire in more than half its runs?
- Does every near-miss stay below half?
- Does each eval case have at least one assertion that fails without the skill?

## Next

A skill you can test is a skill you can share. [Lesson 4](https://mindket.com/learn/agent-skills/install-agent-skills) covers installing skills in Claude Code, Codex and other agents, personal folders and project folders, and plugins.

The [Claude Code subagents](https://mindket.com/blog/claude-code-subagents) post covers the subagents skill-creator uses to give each eval run a clean context. A shorter take on the trigger check, with the same prompt counts, is in [how to create an Agent Skill](https://mindket.com/blog/how-to-create-agent-skill) under its trigger tests.

## Check yourself

1. What does an eval measure that a trigger check does not?

   A. Whether the skill file parses
   B. Whether the agent loads the skill on the right prompts
   C. Whether the output is better with the skill than without it

   Answer: C. A trigger check asks whether the skill loads, and an eval asks whether the result is better for it, so they need different tests.

2. How should you run each trigger prompt?

   A. Three times, each in a fresh session
   B. Once, in the session where you wrote the skill
   C. Once, then again only if it fails

   Answer: A. One prompt can load the skill on one run and miss on the next, and your own session fills the gaps the instructions leave.

3. An assertion passes both with the skill and without it. What should you do with it?

   A. Keep it, because a pass is a pass
   B. Raise the trigger rate threshold
   C. Add more instructions to the skill
   D. Remove or replace it

   Answer: D. An assertion that passes in both runs measures the model, not the skill.

4. Which prompts make the best should-not-trigger cases?

   A. A short unrelated question such as "What's the weather today?"
   B. Near-misses that share words with the skill, such as a request to write a new spec
   C. Only the prompts that failed in the last round

   Answer: B. An easy negative proves nothing, while a near-miss that shares words with the skill tests whether the description is precise.

## Frequently asked questions

### Should the trigger prompts and the eval cases be the same set?

No. A trigger prompt needs only a `should_trigger` label, and it passes or fails on whether the skill loaded. An eval case carries an expected output and any input files, and it measures what the skill did once it was loaded. The guides keep them in separate files.

### Does Claude Code report how often a skill is used?

Yes. Run `/skill-doctor` (v2.1.252 or later) to see what each skill costs and how often it gets used. It also flags listed skills that were never invoked. For a skill in a plugin, an eval case with a `tool_used: Skill` grader measures the trigger rate across prompts.

### What counts as a baseline run?

The same prompt without the skill for a new skill, or with a snapshot of the previous version when you are improving one. The evals guide copies the old version to a snapshot folder before editing and saves that run under `old_skill/outputs/` in place of `without_skill/`.

### What if the output is just as good without my skill?

Compare the cost before you keep it. The guide's benchmark sets what the skill costs in time and tokens against what it adds to the pass rate, and says that doubling token use for a 2-point gain might not be worth it. Anthropic adds that a pass without the skill may mean the model has absorbed what the skill used to teach.

### How do I make a trigger run cheaper?

If your client supports it, stop a run as soon as the outcome is clear: the agent either consulted the skill or started working without it. The guide says this can significantly reduce the time and cost of the full set, which runs dozens of invocations at the starting numbers.

### How do I test a skill in Codex or another agent?

The method is the same, and each client shows skill use in its own way. OpenAI's [Codex guide](https://learn.chatgpt.com/docs/build-skills) says to test prompts against the description to confirm trigger behavior, and offers `/skills` or `$` for calling a skill explicitly. Codex also picks up skill edits on its own, and a restart is the fix when one doesn't show.

## Sources

- [Agent Skills: Optimizing skill descriptions](https://agentskills.io/skill-creation/optimizing-descriptions) (read 2026-11-04)
- [Agent Skills: Evaluating skill output quality](https://agentskills.io/skill-creation/evaluating-skills) (read 2026-11-04)
- [Claude Code docs: Skills](https://code.claude.com/docs/en/skills) (read 2026-11-04)
- [Anthropic skills repository: skill-creator](https://github.com/anthropics/skills/blob/main/skills/skill-creator/SKILL.md) (read 2026-11-04)
- [Claude blog: Improving skill-creator](https://claude.com/blog/improving-skill-creator-test-measure-and-refine-agent-skills) (read 2026-11-04)
- [Claude Platform docs: Skill authoring best practices](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices) (read 2026-11-04)
- [OpenAI: Build skills (Codex)](https://learn.chatgpt.com/docs/build-skills) (read 2026-11-04)
- [Claude Code changelog](https://code.claude.com/docs/en/changelog) (read 2026-11-05)
- [Boris Cherny on X: separate context windows, 9 March 2026](https://x.com/bcherny/status/2031151689219321886) (read 2026-11-05)
- [Chase AI on YouTube: Claude Code Skills Just Got a MASSIVE Upgrade](https://www.youtube.com/watch?v=UxfeF4bSBYI) (read 2026-11-05)

[Browse kits](https://mindket.com/kits)