Metadata

name: fable-eval
description: Evaluate changes to agent prompts, skills, routing policies, and harnesses against reproducible baselines, held-out suites, and regression benchmarks. Use when optimizing agent system prompts, measuring skill triggering accuracy, evaluating routing changes, or running benchmark regressions — even if the user does not explicitly say "fable-eval" (e.g. "benchmark this prompt", "evaluate agent behavior", "test skill performance", "run the eval suite"). Do NOT use for routine application unit tests (use fable-verify).
version: 1.3.0
pack: evolution
inputs: ["candidate_modification"]
requires: ["reproducible_baseline"]
produces: ["eval_verdict","regression_evidence"]
gates: ["baseline_frozen","holdout_tested","rollback_defined"]
fallback: fable-plan
mutatesWorkspace: false
parallelSafe: true
neural_links: {"precursors":["skill-creator"],"continuations":["fable-plan","fable-execute"],"lateral_peers":["fable-verify"],"recovery":"fable-recover"}

Fable Eval

Measure whether an agent-control change improves the behavior it claims to improve without quietly overfitting the benchmark or breaking neighboring behavior.

Mission

An eval is a decision instrument, not a scoreboard. It needs a frozen comparison point, representative semantic families, oracle isolation, explicit failure costs, and a rollback decision.

A candidate should not win because the prompts resemble its instructions, because holdouts leaked into authoring, or because one average score hides a severe regression.

Activate When

changing Skills, prompts, routers, hooks, agent profiles, policies, or model-control logic;
comparing candidate prompt/agent configurations;
measuring trigger precision/recall or action compliance;
validating a new behavioral maturity claim;
investigating whether an apparent improvement is robust or benchmark-specific.

Do Not Activate When

verifying ordinary application behavior (fable-verify);
authoring a Skill before its intended behavior is clear (skill-creator);
running a one-off subjective prompt demo with no acceptance decision.

Evaluation Classification

Change	Primary eval risk
Router/trigger	false positives, false negatives, precedence
Skill instruction	action correctness, forbidden shortcuts, boundary behavior
Spark/next-action	top-1 action, unsafe suggestion, silence precision
Hook/guard	enforcement, false blocking, bypasses
Prompt/persona	task quality + regressions + instruction conflicts
Tool policy	correct tool choice, unsafe/missing action
Model/config	quality/latency/cost variance across representative tasks

Protocol

Stage 1 — Define the decision before running tests

State:

candidate being evaluated;
baseline/control;
exact behavior expected to improve;
metrics and thresholds;
unacceptable regressions;
rollback action.

Avoid inventing metrics after seeing results.

Stage 2 — Build semantic scenario families

Cover distinct decisions, not wording variants. Include as applicable:

straightforward positive case;
non-trigger/boundary case;
ambiguous competing action;
adversarial shortcut pressure;
partial/contradictory evidence;
failure/recovery path;
legacy/constrained environment;
unseen holdout.

Record family coverage separately from raw prompt count.

Stage 3 — Freeze baseline and corpus identity

Bind the run to:

corpus hash/version;
candidate/baseline identity;
evaluator/scorer version;
provider/model/config where external;
timestamp/repository revision.

Changing the subject, oracle, corpus, or scoring logic invalidates direct comparability unless explicitly normalized.

Stage 4 — Protect the oracle

Provider-facing requests must not reveal expected actions, forbidden actions, category labels, holdout identity, scoring implementation, or answer-bearing metadata.

Do not draft the candidate while repeatedly reading holdout failures. Promote discovered cases into a future checked corpus and preserve a new unseen holdout.

Stage 5 — Execute baseline and candidate consistently

Use the same task inputs, tool availability, context budget, temperature/configuration, timeout policy, and scoring rules where comparison requires them.

Capture provider errors/timeouts as failures or explicit unavailable states; do not fill missing outputs from the oracle.

Stage 6 — Score by slices, not average alone

Inspect:

overall metric;
each semantic family;
negative/adversarial forbidden violations;
high-cost regressions;
variance/repeated-run stability where stochasticity is material;
routing confusion pairs where applicable.

A 1% average gain is not acceptable if it introduces a severe release/security/recovery regression.

Stage 7 — Investigate suspicious gains

Check for:

prompt leakage;
duplicated/near-duplicate scenarios;
benchmark-specific keyword matching;
changed tool/context budget;
scorer drift;
cherry-picked seeds/runs;
examples copied into the candidate.

Stage 8 — Decide and preserve rollback

Verdict:

ACCEPT: thresholds met, no prohibited regression, evidence representative/fresh;
REJECT: candidate regresses required behavior or fails threshold;
INCONCLUSIVE: evidence lacks breadth/stability/holdout integrity.

Record baseline artifact so rollback remains possible.

Decision Rules

Semantic family breadth matters more than raw scenario count.
Surface rewrites of one case do not create independent coverage.
Holdouts stop being holdouts once used repeatedly to tune the candidate.
Compare slices before averages; safety-critical forbidden violations can veto a higher average score.
If stochastic variance could change the decision, repeat enough runs to estimate stability rather than cherry-picking one seed.
If provider/runtime errors differ between candidate and baseline, separate infrastructure failure from behavior score.
Never preserve an old maturity result after the evaluated Skill/corpus/oracle changes unless freshness validation proves identity.
Do not lower thresholds after a candidate fails simply to ship it.

Invariants

Baseline is reproducible/frozen before candidate judgment.
Oracle/holdout data remains hidden from the evaluated agent.
Candidate and baseline are compared under equivalent conditions where claimed.
Missing/failed provider outputs are never replaced by expected answers.
Every acceptance decision has a rollback path.
High-cost regressions remain visible even when aggregate score improves.

Failure Taxonomy

Benchmark overfit

Candidate improves checked cases but fails unseen family/holdout. Increase semantic breadth and reject/generalize candidate.

Oracle leakage

Expected/forbidden/category data reaches provider/candidate authoring loop. Discard contaminated evidence and create fresh blind cases.

Metric blindness

Average improves while important slice worsens. Use per-family and veto metrics.

Non-comparable runs

Different model/tool/context/scorer settings produce apparent gain. Re-run under controlled conditions.

High variance

Repeated runs change verdict. Increase samples/control nondeterminism or mark inconclusive.

Corpus drift

Skill/scenario/oracle changed after evidence capture. Mark evidence stale and rerun.

Anti-Patterns

five paraphrases counted as five independent tests;
reading holdouts while tuning every candidate;
accepting on average score alone;
changing thresholds after seeing failure;
treating provider timeout as skipped rather than failed/incomplete;
evaluating only positive examples;
copying expected action vocabulary in a way that reveals the answer per case;
claiming M4 because an evidence JSON file exists even though corpus hash changed.

Eval Report

Candidate / baseline:
Corpus + hashes:
Provider/config:
Semantic families:
Metrics + thresholds:
Per-family results:
Forbidden/high-cost regressions:
Variance/repeats:
Leakage/comparability checks:
Verdict: ACCEPT | REJECT | INCONCLUSIVE
Rollback:
Evidence freshness:

Completion Criteria

Evaluation completes when:

baseline and candidate identities are explicit;
semantic coverage and blind holdout integrity are credible;
metrics are inspected by meaningful slices;
regressions/forbidden actions are not hidden by averages;
verdict and rollback are evidence-backed;
evidence freshness is tied to the exact evaluated corpus/control.

Progressive Resources

Deep guide: references/benchmark-design-and-overfit-control.md
Existing protocol: references/eval-harness-protocol.md
Example: examples/eval-regression-run.md

fable-eval

安装