openclaw-eval-skill

Evaluation framework for any OpenClaw skill. No claude CLI dependency — all agent execution runs through sessions_spawn + sessions_history.

Scope: Works with CLI tool skills, conversational skills, and API integration skills.

Runtime Actions Disclosure

This skill performs the following actions during evaluation:

NOT performed automatically: Gateway restart, config modification, skill installation. These require manual user action (see "Bundled Test Skill" section).

Quick Eval

Just say:

evaluate weather

The agent will:

1. Run scripts/resolve_paths.py weather to find all paths
2. Execute trigger rate + quality compare with detected evals
3. Output results to eval-workspace/weather/iter-N/

Options:

• evaluate weather trigger — trigger rate only
• evaluate weather quality — quality compare only
• evaluate github --mode all — explicit mode

What gets auto-detected:

• Skill path: from OpenClaw built-in skills or registered extraDirs
• Evals: from evals/{skill-name}/ or fallback to evals/example-*.json
• Output: next available iter-N directory

First step for agent: Run the resolver to get paths:

python scripts/resolve_paths.py {skill-name} --mode {trigger|quality|all}

Use the JSON output to fill in paths for the workflows below.

Bundled Test Skill: fake-tool

A test skill (test-skills/fake-tool/) is included for validating trigger rate detection. It simulates a fictional "Zephyr API" that models cannot know from training.

Manual setup required: The agent will NOT automatically install fake-tool or restart your gateway. If you want to test with fake-tool:

1. Copy fake-tool to your skills directory:

   cp -r test-skills/fake-tool ~/.openclaw/workspace/skills/
   

2. Restart OpenClaw gateway (from terminal):

   openclaw gateway restart
   

3. Verify registration:

   python scripts/resolve_paths.py fake-tool
   

If step 3 returns a valid path, fake-tool is ready. If "not found", check that your ~/.openclaw/openclaw.json includes the skills directory in skills.load.extraDirs.

Evaluation Scenarios

Tier 1: Core (Always Run)

Tier 2: Optional (Run When Needed)

Tier 3: Future (Roadmap)

How This Skill Works

Two-layer architecture:

Layer 1: Agent (main OpenClaw session) — YOU ARE HERE
  → Reads evals.json
  → Calls sessions_spawn to run subagents
  → Calls sessions_history to collect results
  → Writes raw data to workspace/

Layer 2: Python analysis scripts (run via exec)
  → Read the raw data from workspace/
  → Compute statistics
  → Generate reports

Python scripts (analyze_*.py) are data processors — they cannot call sessions_spawn. The agent drives the workflow.

Usage

Follow USAGE.md for all workflows.

Quick reference:

Each workflow follows the same pattern:

1. Agent spawns subagents using sessions_spawn
2. Agent collects histories using sessions_history
3. Agent writes raw data to workspace/{skill}/iter-{n}/raw/
4. Agent runs analysis script via exec

Core Principles

1. Never modify the evaluated skill — observe only, give recommendations
2. Keep eval records in workspace — output goes to eval-workspace/<skill-name>/iteration-N/
3. Keep full records — save full_history.json (including tool_use + tool_result)

agents/ Reference

Directory Structure

eval-workspace/<skill-name>/
├── evals.json                    ← Eval definition (shared across iterations)
└── iteration-1/
    ├── raw/
    │   ├── histories/            ← Trigger test session histories
    │   └── transcripts/          ← Quality compare transcripts
    ├── trigger_results.json      ← analyze_triggers output
    ├── quality_results.json      ← analyze_quality output
    └── diagnostics/
        └── RECOMMENDATIONS.md

evals.json Format

Quality Compare (prompt + assertions):

{
  "skill_name": "my-skill",
  "evals": [
    {
      "id": 1,
      "name": "onboarding-fresh",
      "prompt": "Check the weather in Tokyo",
      "context": "Clean machine, no prior setup. For grader only.",
      "expected_output": "Install → configure → verify profile",
      "assertions": [
        {
          "id": "a1-1",
          "description": "Install command executed",
          "type": "output_contains",
          "value": "pip install"
        },
        {
          "id": "a1-2",
          "description": "Profile verified after setup",
          "type": "output_contains",
          "value": "profile current",
          "priority": true
        }
      ]
    }
  ]
}

Trigger Rate (query + expected):

{
  "id": 1,
  "name": "direct-weather",
  "query": "What's the weather in Singapore?",
  "expected": true,
  "category": "positive"
}

Assertion Types

Priority assertions ("priority": true): any failure → overall=FAIL. Gap assertions ("note": "Best practice..."): failure = skill design gap.

Issue Priority (grader output)

🔴 P0 Critical  — Core functionality broken
🟠 P1 High      — Significantly impacts usability
🟡 P2 Medium    — Room for improvement
🟢 P3 Low       — Minor polish

Behavior Anomaly Tracking

Grader records these signals beyond assertions:

Key Constraints

• sandbox="inherit" — subagents inherit skill registration environment
• cleanup="keep" — history must be retained for trigger detection
• Skill must be in a real directory under skills.load.extraDirs (symlinks rejected)