Test a Claude skill: trigger checks, evals and a no-skill baseline

Before you start

You need Claude Code, the reviewing-api-changes skill from lessons 1 and 2 (any skill works) and jq for the script. Claude Code v2.1.233 or later has the validate command used below. Any agent that can run a prompt without a human and show which skills it used follows the same method. The Agent Skills guide says most clients offer logs, tool call histories or verbose output for that.

The sample prompts and test cases were written for this course. The script and the file formats come from the Agent Skills guides, and the commands from Claude Code's documentation, listed under Sources.

Every fact here comes from the pages under Sources, read on 4 November 2026. This lesson is not hands-on testing.

Run a trigger check in six steps

A trigger check asks one question: does the agent open the skill on prompts that call for it and leave it alone on prompts that don't? The Agent Skills guide says the most useful positive prompts are the ones where the skill would help but the link isn't obvious from the wording, because those are where the description decides the outcome.

The negatives matter more than they look. "Write a fibonacci function" tests nothing against an API-review skill. A prompt that shares words with it, such as a request to write a new spec, does.

[
  { "query": "my teammate renamed /users/{id} to /accounts/{id} in api/openapi.yaml, is that safe to merge?", "should_trigger": true },
  { "query": "can you diff v1.yaml and v2.yaml of our payments API before we release", "should_trigger": true },
  { "query": "i changed a field from string to integer in the swagger file, anything i should worry about", "should_trigger": true },
  { "query": "write an OpenAPI spec for a new /orders endpoint", "should_trigger": false },
  { "query": "convert this OpenAPI yaml file to json", "should_trigger": false },
  { "query": "explain what a breaking change is in semantic versioning", "should_trigger": false }
]

1. Confirm the skill parses. Run claude plugin validate ~/.claude/skills, then ask Claude Code What skills are available? and check that the skill is listed.
2. Write about 20 prompts into eval_queries.json, in the shape above: 8 to 10 that should trigger the skill and 8 to 10 that should not. Vary phrasing, detail and how directly the prompt names the skill's domain.
3. Make the prompts read like real ones. The guide suggests file paths, personal context, specific column or company names, casual language and the occasional typo.
4. Split the set before you start fixing anything: about 60% for training and 40% held back for validation, with positives and negatives in both. Shuffle once and keep the split fixed.
5. Run every prompt three times, each in a fresh session, and record whether the skill loaded. The guide calls three a reasonable starting point, since one prompt can fire the skill on one run and miss on the next.
6. Score it. A should-trigger prompt passes when its trigger rate is above 0.5, and a should-not-trigger prompt passes when its rate is below it.

Script the runs, then fix the description

Twenty prompts at three runs each make 60 invocations, which is why the guide says to script them. Its example sets the number of runs and the skill name at the top, asks Claude Code for JSON output and looks for a Skill tool call with jq. Change SKILL_NAME to your skill. For another client, replace the detection inside check_triggered.

When prompts fail, edit the description from the training failures only. A should-trigger prompt that misses means the description may be too narrow. A should-not-trigger prompt that fires means it may be too broad, and the fix is to say what the skill does not do. Copying words from a failed prompt is overfitting. Find the general kind of request it stands for.

Keep an eye on length, because descriptions tend to grow during tuning and the specification caps them. If several rounds change nothing, try a different sentence structure. The guide also says five iterations is usually enough, and that a stuck score may mean the queries are the problem, not the description.

#!/bin/bash
QUERIES_FILE="${1:?Usage: $0 <queries.json>}"
SKILL_NAME="my-skill"
RUNS=3

# This example uses Claude Code's JSON output to check for Skill tool calls.
# Replace this function with detection logic for your agent client.
# Should return 0 (success) if the skill was invoked, 1 otherwise.
check_triggered() {
  local query="$1"
  claude -p "$query" --output-format json 2>/dev/null \
    | jq -e --arg skill "$SKILL_NAME" \
      'any(.messages[].content[]; .type == "tool_use" and .name == "Skill" and .input.skill == $skill)' \
      > /dev/null 2>&1
}

count=$(jq length "$QUERIES_FILE")
for i in $(seq 0 $((count - 1))); do
  query=$(jq -r ".[$i].query" "$QUERIES_FILE")
  should_trigger=$(jq -r ".[$i].should_trigger" "$QUERIES_FILE")
  triggers=0

  for run in $(seq 1 $RUNS); do
    check_triggered "$query" && triggers=$((triggers + 1))
  done

  jq -n \
    --arg query "$query" \
    --argjson should_trigger "$should_trigger" \
    --argjson triggers "$triggers" \
    --argjson runs "$RUNS" \
    '{query: $query, should_trigger: $should_trigger, triggers: $triggers, runs: $runs, trigger_rate: ($triggers / $runs)}'
done | jq -s '.'

Evals: is the output better than no skill?

Seeing a skill trigger tells you Claude found it, not that it did what you meant. Claude Code's documentation says to measure the two separately. For output quality the check is a baseline comparison: run each prompt in a fresh session with the skill, run it again with the skill off, and compare. Each run needs a clean context, so the agent follows only what the skill says and not what you remember from writing it.

To turn a personal or project skill off for the second run, set it to "off" in skillOverrides. That setting doesn't apply to plugin skills. For those, claude plugin eval repeats each run with no plugin loaded. When you improve an existing skill, the evals guide uses the old version as the baseline: copy it to a snapshot folder first, and point the baseline run there.

Start with two or three test cases in evals/evals.json. Each has a realistic prompt, a plain-words description of success and any input files. Write assertions after the first round, once you can see what good output looks like. "The output file is valid JSON" can be checked, "the output is good" can't, and an exact required phrase fails correct answers worded differently. Judge tone and design by eye.

{
  "skill_name": "reviewing-api-changes",
  "evals": [
    {
      "id": 1,
      "prompt": "I renamed the status field to state in api/openapi.yaml and made customer_id required on POST /orders. Is this safe to ship?",
      "expected_output": "A review that flags both changes as breaking for existing clients and suggests a safer rollout.",
      "files": ["evals/files/openapi-before.yaml", "evals/files/openapi-after.yaml"],
      "assertions": [
        "The review names the renamed status field as a breaking change",
        "The review flags the new required customer_id field",
        "The review lists each change with a suggested fix"
      ]
    }
  ]
}

Let skill-creator run the loop

Once the manual version works, Anthropic's skill-creator plugin automates the comparison inside Claude Code. In a terminal, install it from the official marketplace with the first line below, then ask for an evaluation in your own words. If Claude Code answers that the marketplace isn't found, run /plugin marketplace add anthropics/claude-plugins-official and retry.

Its own description loop follows the guide's numbers: a 60 and 40 split, three runs for each prompt and at most five iterations, choosing the description by its held-out score. Anthropic's announcement says that running description tuning on its document-creation skills improved triggering on 5 of 6 public ones. It adds a caveat worth keeping. If the base model passes your evals without the skill, the skill's techniques may now be in the model, and the skill may no longer be needed.

Anthropic has two tools for this, and their formats aren't interchangeable. The skill-creator plugin iterates on one skill inside a conversation. For a plugin's skills, claude plugin eval scores each run with graders and exits non-zero below a threshold, so a CI job can fail on it.

/plugin install skill-creator@claude-plugins-official

evaluate my reviewing-api-changes skill with skill-creator

• Test cases saved in evals/evals.json, with one subagent per case so each run starts clean.
• Grading of each assertion into grading.json, with the evidence for every pass or fail.
• A benchmark of pass rate, time and tokens, with the skill and without it, in benchmark.json.
• A blind A/B comparison of two versions of the skill, to confirm an edit helped before you keep it.
• Description tuning, and an HTML viewer where you read each output and leave feedback for the next round.

Boris Cherny on fresh context windows, and a skill-creator walkthrough

Boris Cherny, who created Claude Code at Anthropic, explained on X in March 2026 why a second context window catches what the first one misses. One agent can cause a bug and another, on the same model, can find it, much as a coworker reviewing your code spots what you could not. That is the reason every eval case in this lesson runs in a fresh session, and why skill-creator gives each case its own subagent.

Chase Hannegan's Chase AI channel put out a video on the skill-creator update to go with his March 2026 written guide to testing and improving skills. Watch it after one manual round of trigger checks and evals, so you can see which parts of the loop the plugin takes over.

A screenshot of X's own embed of the post, taken on 5 November 2026, with the time in UTC. X cuts the post short with Show more. Select it to open the post on X.

Common mistakes

• Testing in the session where you wrote the skill. Your own context fills the gaps the instructions leave.
• Trusting one run. The same prompt can load the skill once and miss the next time.
• Writing easy negatives. "What's the weather today?" has no keyword overlap and proves nothing.
• Tuning on every prompt you have. With no held-out set, the description fits the prompts you tuned on, and the best round may be an earlier one.
• Testing on one model. Anthropic's guide says to try every model you plan to use, because what works for Opus may need more detail for Haiku.

Try it yourself

Grow the six-prompt sheet from lesson 1 into a full trigger set, with as many near-misses as prompts that should load the skill. Run each prompt three times and write the trigger rate beside it. Then write two eval cases for the skill, run each with the skill and with it switched off, and mark which assertions change.

Keep what you find. A pass that appears only with the skill shows you what the skill is worth. A pass that appears in both runs shows you what you can cut.

• Does every should-trigger prompt fire in more than half its runs?
• Does every near-miss stay below half?
• Does each eval case have at least one assertion that fails without the skill?

Next

A skill you can test is a skill you can share. Lesson 4 covers installing skills in Claude Code, Codex and other agents, personal folders and project folders, and plugins.

The Claude Code subagents post covers the subagents skill-creator uses to give each eval run a clean context. A shorter take on the trigger check, with the same prompt counts, is in how to create an Agent Skill under its trigger tests.

Frequently asked questions

Sources

1. Agent Skills: Optimizing skill descriptions (read November 4, 2026)https://agentskills.io/skill-creation/optimizing-descriptions
2. Agent Skills: Evaluating skill output quality (read November 4, 2026)https://agentskills.io/skill-creation/evaluating-skills
3. Claude Code docs: Skills (read November 4, 2026)https://code.claude.com/docs/en/skills
4. Anthropic skills repository: skill-creator (read November 4, 2026)https://github.com/anthropics/skills/blob/main/skills/skill-creator/SKILL.md
5. Claude blog: Improving skill-creator (read November 4, 2026)https://claude.com/blog/improving-skill-creator-test-measure-and-refine-agent-skills
6. Claude Platform docs: Skill authoring best practices (read November 4, 2026)https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices
7. OpenAI: Build skills (Codex) (read November 4, 2026)https://learn.chatgpt.com/docs/build-skills
8. Claude Code changelog (read November 5, 2026)https://code.claude.com/docs/en/changelog
9. Boris Cherny on X: separate context windows, 9 March 2026 (read November 5, 2026)https://x.com/bcherny/status/2031151689219321886
10. Chase AI on YouTube: Claude Code Skills Just Got a MASSIVE Upgrade (read November 5, 2026)https://www.youtube.com/watch?v=UxfeF4bSBYI