AI Agent Evaluation Lab

Evaluate AI agents and workflows using structured, deterministic frameworks.

Turn subjective feedback into measurable scorecards, failure analysis, prioritized fixes, and clear release recommendations.

Use only user-provided information or synthetic examples derived from the user’s description.

Quick Start

clawhub install agent-evals-lab
text
evals customer support agent for SaaS onboarding and billing questions
evals --regression prompt update for support agent with before/after outputs
evals --compare model A vs model B for retrieval-heavy support workflows
Best For
AI agent builders

Prompt engineers

Agent ops teams

QA and evaluation workflows

Release reviews before deployment

Quick Reference
Need	Use
Standard evaluation report	evals [agent or workflow summary]
Before/after regression analysis	evals --regression [change summary]
Model or workflow bake-off	evals --compare [model A vs model B]
Production readiness review	evals --readiness [agent summary]
Conservative scoring	evals --strict [summary]
Structured export	evals --json [summary]
When to Use
Use this skill when the user asks to:

evaluate an AI agent

audit agent quality

benchmark one model against another

detect regressions after a change

review tool reliability

analyze workflow failures

check production readiness

decide whether a change is safe enough to ship

Safety Constraints
text
- Use only information explicitly provided by the user
- Do not read local files, credentials, system config, or private repos automatically
- Do not download or execute external scripts unless explicitly provided and approved
- If test data is missing, generate synthetic cases only from the user’s description
Modes
text
evals [agent or workflow summary]
evals --regression [change summary]
evals --compare [model A vs model B]
evals --readiness [agent summary]
evals --strict [agent or workflow summary]
evals --json [agent or workflow summary]
If no mode is specified, use the standard evaluation flow.

Inputs
text
agent purpose
target tasks
representative test cases
expected outcomes
constraints           | latency / cost / risk tolerance
environment notes     | models / tools / channels
before_outputs        | optional
after_outputs         | optional
If test cases are missing, generate synthetic cases based only on the
task description.

Evaluation Dimensions
Score each case on a 1–5 scale with short evidence notes:

text
Correctness      — How accurate is the response?
Relevance        — How well does it address the task?
Actionability    — Can the user act on the output effectively?
Risk Flags       — Are there safety, compliance, or irreversible-action concerns?
Tool Reliability — Were tools used correctly and reliably?
Workflow
Step 1 — Build the Evaluation Set

Each case should include:

text
- id
- task type
- input
- expected outcome
- criticality
- tool context if relevant
Step 2 — Score Outputs

Capture:

text
- output quality
- failure notes
- tool-use behavior
- evidence for scores
Step 3 — Identify Failure Clusters

Examples:

text
instruction_following
reasoning_quality
tool_call_failure
format_noncompliance
safety_boundary_failure
hallucination
retrieval_miss
Step 4 — Prioritize Fixes

Rank by:

text
- expected impact
- implementation effort
- risk reduction value
Step 5 — Compute Deltas

If before/after evidence exists, compute:

text
- score deltas
- improvements
- regressions
- critical-case outcomes
Step 6 — Apply Readiness Gates

Use deterministic thresholds before recommending deployment.

Deterministic Gates
Hard gates include:

text
- high-risk workflows failing minimum score thresholds
- low tool reliability averages
- synthetic-only evidence in high-risk mode
- unresolved safety-boundary failures
Strict mode should enforce a more conservative final recommendation.

Output Template
text
## Executive Summary
[Score snapshot, strengths, and main failure modes]

## Evaluation Scope
- Agent purpose:
- Task set:
- Constraints:
- Evidence quality:

## Scorecard
- Correctness:
- Relevance:
- Actionability:
- Risk flags:
- Tool reliability:

## Failure Map
### cluster_name
- frequency:
- severity:
- likely cause:
- representative cases:

## Top Fixes
- Fix 1 | expected impact | effort
- Fix 2 | expected impact | effort
- Fix 3 | expected impact | effort

## Regression Plan
- cases to rerun:
- success thresholds:
- rollback or follow-up checks:

## Before / After Delta
- overall score delta:
- tool reliability delta:
- critical-case delta:

## Recommendation
Go / Conditional Go / No-Go

## Rationale
[Why this recommendation follows from the evidence]
JSON Output
json
{
  "agent_purpose": "",
  "mode": "default",
  "evidence_quality": "medium",
  "scorecard": {
    "correctness": 0,
    "relevance": 0,
    "actionability": 0,
    "risk_flags": 0,
    "tool_reliability": 0
  },
  "failure_clusters": [
    {
      "name": "",
      "frequency": 0,
      "severity": "medium",
      "likely_cause": "",
      "cases": []
    }
  ],
  "top_fixes": [
    {
      "fix": "",
      "expected_impact": "high",
      "effort": "medium"
    }
  ],
  "regression_plan": {
    "cases_to_rerun": [],
    "success_thresholds": [],
    "follow_up_checks": []
  },
  "before_after_delta": {
    "overall_score_delta": 0,
    "tool_reliability_delta": 0,
    "critical_case_delta": 0
  },
  "recommendation": "Conditional Go",
  "rationale": ""
}
Limits
This skill does not:

guarantee production success

replace monitoring or live experiments

infer real-world impact perfectly from a small suite

compensate for weak synthetic-only evidence in high-risk situations

Tips
Use --regression when you have before/after outputs.

Use --compare for model or workflow bake-offs.

Use --readiness before shipping to production-like environments.

Mark critical cases explicitly.

Keep synthetic cases separate from real ones in the report.

Treat safety-boundary failures as high priority even if averages look good.

Do not call something “improved” without delta evidence.

Prefer fewer, high-signal benchmark cases over many weak ones.

Author
Vassiliy Lakhonin