Grill Skill

Interview the user to design a skill's eval, then loop run → measure → improve until it ships. Requires caliper (pipx install caliper-eval if missing). Commands, spec skeleton, and expect/assert guidance: REFERENCE.md.

Entry point

/grill-skill [path] — optional path to a SKILL.md.

• Path given — use it.
• No path — look for SKILL.md in the cwd; if found, confirm before proceeding, else ask where it is.

Phase 1 — Understand

Read the SKILL.md. Summarize what it does, when it triggers, and what a successful run looks like. Ask the user to confirm your reading. Wait for confirmation before continuing.

Phase 2 — Detect eval mode

Look for *.eval.yaml beside the SKILL.md (try <dir-name>.eval.yaml first).

• None → New eval. Found → Gap-fill.

Interview one question at a time and wait for each answer. Never invent the user's answers or write the spec before interviewing.

New eval — three tasks

Elicit three tasks, one question at a time:

1. Happy path — the most common successful use. What did the agent do, and what would confirm it worked?
2. Edge case — a tricky-but-valid input that might trip the raw agent.
3. Adversarial — what the skill should refuse or avoid.

Turn each answer into a task: a realistic prompt, an observable expect, and an assert when the outcome is checkable (see REFERENCE.md). Show the proposed YAML and confirm before writing.

Write the spec beside SKILL.md, named <dir-name>.eval.yaml, with skill.path: ./SKILL.md and claude-code as the default backend for both skill and judge unless the SKILL.md targets another.

Gap-fill

Read the existing spec and report its tasks. Ask what behaviors are missing or under-tested before proposing or writing anything — even if the user only asked you to inspect it, report first, then ask. Sharpen each gap into a task, show it, and confirm before writing it in.

Whose setup is measured

Runs load the user's own customizations by default (user skills, plugins, rules, settings and connectors; see REFERENCE.md for backend exceptions), which answers "does my skill work in my agent?". Isolate (--no-user-customizations, or user_customizations: false in the spec) when comparing backends or models, when the number leaves this machine (shared, published, compared with someone else's run), or when measuring the bare agent: each setup is different, so otherwise part of the delta is the setups. --ablate of the user's own skill needs no isolation, since both runs load the same setup.

Always tell the user which mode ran and what it loaded, from the report header's user customizations: line (absent means isolated), and relay any fix caliper compare suggests about it.

Phase 3 — First run

Validate the spec, then run at k=1 (commands in REFERENCE.md). Show the results. Fix any harness or config error (not a task failure) before asking the user what to do next.

Phase 4 — Iterate

Ask whether to iterate or finish.

• Iterate — after the user edits their SKILL.md, re-run at k=3 and show results. Loop back.
• Done — suggest an --ablate <skill-name> run plus a caliper compare to prove the skill beats the raw agent, then remind the user to commit SKILL.md and the .eval.yaml together. Mention that the ablated run is worth keeping: it cannot move when the skill's text changes, so later iterations re-diff against it instead of re-running it.