bench-it
Turn a specific production failure reported by the owner into a discriminating benchmark case: generate a genericized case with frozen evidence from the failing transcript, validate it, run it across the model matrix, and accept only cases that discriminate (some models pass, some fail). Requires a sample run shown to the owner before any batch run.
When to use
The owner complains about a specific assistant failure ("it said X, but that wasn't true / not supported / contradicts its instructions"). This skill is NOT for generic "benchmark this model" requests.
How bench-it works
How bench-it works
- Trim the failing session transcript so the problematic message is last.
- Generate the intake brief:
bench/intake.py brief --complaint "..." --transcript <path> [--full]— pipe it into an isolated subagent (intake model: glm-5.3, a config knob). - The subagent writes
data/cases/<case-id>/(case.json, prompt.md with frozen evidence, rubric.md, notes.md;--fulladds prompt-full.md). Validate:bench/intake.py validate <case-dir> --leak-terms <real proper nouns>. - Sample run first (standing rule: one sample shown to the owner before any batch):
bench/run_case.py <case-dir> --models <m1>,<m2> --sample-only - Batch + discrimination check:
bench/run_case.py <case-dir> --models ... --n 3 [--jury] [--until-discriminating]. A case is accepted only if some models pass and some fail; revise (budget 3) or park asflaky. - On tag push, CI publishes the skill to ClawHub.
What bench-it judges
What bench-it judges
| Failure class | Judge | Origin |
|---|---|---|
claim_follows_from_evidence |
ClaimFollowsFromEvidence |
2026-08-27 GFN verification-theater incident: real citations, clean format, unsupported conclusion. First run: glm-5.3 3P, claude-sonnet-4-6 3P, deepseek-v4-flash 1P/2F, minimax-m2.7 3F — discriminating on revision 0. |
swedish_fabrication, kcal_budget_framing, transit_connection_feasibility, english_stickiness |
see above + EnglishStickiness |
production incidents 2026-05..08 |
Procedure
- Trim the transcript. Copy the failing session's
.jsonl, keep messages up to and including the problematic assistant reply (it must be the last message). Store underprivate-transcripts/(git-ignored). - Generate the intake brief with
bench/intake.py brief --complaint "<the owner's complaint>" --transcript <trimmed.jsonl> [--full]. - Spawn an isolated subagent (intake model: glm-5.3, a config knob in
intake.py) with that brief. It writes
data/cases/<case-id>/. - Validate:
bench/intake.py validate <case-dir> --leak-terms <comma,separated,real,proper,nouns>. - Sample run (standing rule — show the owner before any batch):
bench/run_case.py <case-dir> --models <m> --sample-only - Batch + discrimination check:
bench/run_case.py <case-dir> --n 3 [--jury] [--full] [--until-discriminating]. Not discriminating → revise (budget 3), then park asflaky. - Report the scorecard to the owner.
Rules
- One turn, no live tools: all evidence is frozen in the case prompt.
- Genericization preserves the failure mode, not the trivia.
- Case context may include standing instructions (AGENTS.md rules live in the failing turn) — instruction-following failures need the instruction in the frozen context.
- Old models stay in the matrix (known-floor anchors, e.g. minimax-m2.7).
- Private transcripts and the full-tier payload are never committed or pushed; only genericized cases are tracked.
- Repo:
patello/bench-iton GitHub; skill published to ClawHub automatically on tag push.