Evaluating Agent Skill Selection

Coding agents work best when they get the right context at the right time. Not everything at once, not nothing, but what’s relevant to the task at hand. Most of the mechanisms for delivering that context are passive: an AGENTS.md file in the repo root is loaded on every turn, a directory-scoped AGENTS.md is loaded when the agent works in that directory, and a block comment at the top of a file is loaded when the agent opens it. None of these require the agent to make a decision. The context is just there.

Skills are different. A skill is a bundle of instructions and resources that an agent tool loads based on task relevance. The agent sees a catalog of available skills (their names and descriptions), and the tool decides whether to load one mid-task. That decision point is what makes skills powerful and also what makes them fragile. Every other context mechanism triggers automatically. Skills are the one mechanism where the agent has to choose correctly.

The Selection Problem

Jude Gao at Vercel published an analysis of AGENTS.md vs. skills that quantifies how often this goes wrong. They built evals targeting Next.js 16 APIs that aren’t in model training data, then tested whether agents would invoke a skill containing the correct documentation. With no explicit instructions to use the skill, agents failed to invoke it 56% of the time. The pass rate was 53%, identical to having no docs at all.

| Configuration | Pass Rate | vs Baseline | | --- | --- | --- | | Baseline (no docs) | 53% | — | | Skill (default behavior) | 53% | +0pp | | Skill with explicit instructions | 79% | +26pp | | AGENTS.md docs index (8KB) | 100% | +47pp |

Adding explicit instructions to use the skill pushed the pass rate to 79%, but even that was fragile. Subtle wording differences like “You MUST invoke the skill” vs. “Explore project first, then invoke skill” produced meaningfully different results from the same skill and same docs. Meanwhile, a compressed 8KB docs index embedded directly in AGENTS.md hit 100% without the agent needing to make any selection decision at all.

Skills and passive context solve different problems, and they complement each other. AGENTS.md is better for broad, always-on context while skills are better for task-specific workflows. But that 56% miss rate is the number to sit with. Skills are context-efficient because they only load when relevant, but that efficiency comes at the cost of reliability. An AGENTS.md file never fails to load. A skill can. The question is how often, and for which prompts. That’s what evals answer.

What a Skill Selection Eval Actually Tests

A selection eval isn’t a quiz. You don’t ask the agent “which skill would you pick for this task?” You register a real load_skill tool and observe whether the agent calls it when given a prompt. That’s the behavior you care about.

The eval answers a simple question: given this prompt and these available skills, does the agent load the right one? It can also answer the inverse: does the agent correctly leave all skills unloaded when none are relevant?

This is the same discipline as unit testing. You write a function, you write a test that calls it and asserts the output. You write a skill, you write an eval that presents it and asserts the selection. The mechanics are different but the principle is identical.

Testing Skills with Skills Dojo

I built Skills Dojo specifically for this problem. It’s a CLI toolkit for writing and running selection evals against agent skills. You define your evals in YAML, run dojo run, and get a pass/fail report with model reasoning logs for every case.

Note: Skills Dojo currently requires GitHub Copilot as the model provider. Support for additional providers is planned.

Skills Dojo follows the agent skills specification. It discovers skills by scanning for SKILL.md files and runs each eval by presenting the prompt and available skills to an evaluator model, then asserting whether the correct skill was selected.

Setup

Install the CLI globally:

npm install -g skills-dojo

Structure your skills directory with a SKILL.md and an evals/selection.yaml alongside it:

skills/
  code-review/
    SKILL.md
    evals/
      selection.yaml

A minimal SKILL.md looks like this:

---
name: code-review
description: Review code changes for security, performance, and correctness issues.
---

# Code Review

Analyzes diffs and pull requests...

The description field is what the agent sees when deciding whether to select the skill. It’s also what you’ll be tuning.

Your selection.yaml defines the prompts and assertions:

timeout: 30
skills: all

evals:
  - name: should-select-code-review
    prompt: "Review this pull request for potential security issues and suggest improvements."

  - name: should-not-select-code-review
    prompt: "Write a Python function that calculates the Fibonacci sequence."
    assert: none

When assert is omitted, Dojo defaults to asserting that the skill the eval lives under was selected. assert: none tests that the agent correctly identifies the prompt as out of scope. Then run:

dojo run

Variants

The Vercel data showed that wording matters. “You MUST invoke the skill” vs. “Explore first, then invoke” produced dramatically different results. The same is true for the description field in your SKILL.md. Variants let you test that directly.

variants:
  - name: concise
    value: Write and optimize SQL queries across all major database dialects.

  - name: verbose
    value: >
      Write correct, performant SQL across all major data warehouse and database
      dialects including Snowflake, BigQuery, Databricks, PostgreSQL, MySQL, and
      SQL Server.

evals:
  - name: should-select-sql-queries
    prompt: "Write a query that finds the top 10 customers by revenue using a window function."

Each eval runs once with the current description, then once per variant. Results are displayed in a matrix so you can see how each wording performs against the same prompt. This is the primary way to tune a skill’s description without guessing.

Decoys

Decoys are fake skills injected into the available pool to test discrimination. They answer the question: does the agent select your skill because it’s relevant, or just because it’s the only option?

evals:
  - name: select-with-decoys
    prompt: "Review this pull request for potential security issues."
    decoys:
      - name: code-formatter
        value: Automatically format code to match style guidelines.
      - name: code-explainer
        value: Explain what a piece of code does in plain English.

A skill that passes without decoys but fails with them has a description that’s too broad or ambiguous. The model is picking it by elimination. Decoys force the selection to be based on actual relevance.

Reports

Every run saves a JSON report and a reasoning log to skills/<name>/evals/reports/<run-id>/. The reasoning log is where the useful debugging happens. When a selection fails, you can see exactly what the model was thinking before it picked the wrong skill or skipped them all. That feedback directly informs how you rewrite the description or restructure the skill instructions.

Writing Evals Is the Same Discipline as Writing Tests

Skills occupy a unique position in how agents get context. Passive context files load automatically. Skills depend on the agent recognizing the task and choosing to load them. That decision point is what makes skills context-efficient, and it’s also what makes them unreliable without testing.

The Vercel numbers show a 56% miss rate with no explicit instructions. Even with carefully tuned instructions, they topped out at 79%. Those are real failure rates on real tasks. If your skill is critical to getting correct output, you need to know whether agents are actually selecting it before you rely on it.

Writing evals for your skills is the same thing as writing tests for your code. You wouldn’t ship a function without verifying it behaves correctly. Don’t ship a skill without verifying it gets selected. The Skills Dojo docs cover the full eval schema and configuration options. Jude Gao’s Vercel article is worth reading in full for the broader context on how passive context and skills fit together.