ExpertPack Eval

作为ExpertPack配套工具，核心功能分两类：一是盲测前沿模型，计算该包中模型无法自主回答的内容占比（EK比率）；二是对加载该包的代理运行自动化评估集，由LLM打分。典型场景为评估ExpertPack的内容独特性与质量，结果可写入Obsidian前置元数据查询。需OpenRouter API密钥，输入包路径等参数，输出带分数的YAML文件，依赖Python3。

brianhearn

热度 62.65

最近更新 2026-05-17 14:57:35

分类：数据依赖：代码执行 / 第三方 API / 读写文件应用场景：数据处理 / 报告撰写 / 自动化脚本

安装技能

复制下面的话术发给你的 Agent，它会通过 secure-skills 检索并安装该技能。

执行命令安装此 skill： npx secure-skills-cli add expertpack-eval

Metadata

name: expertpack-eval
description: Measure ExpertPack EK (Esoteric Knowledge) ratio and run automated quality evals. Use when: (1) Measuring what percentage of a pack's content frontier LLMs cannot produce on their own, (2) Running automated eval sets against a pack-powered agent with LLM-as-judge scoring. Requires OpenRouter API key (auto-resolved from OpenClaw auth or OPENROUTER_API_KEY env var). Companion to the main expertpack skill. Triggers on: 'EK ratio', 'measure EK', 'blind probe', 'eval expertpack', 'pack quality eval', 'run eval', 'esoteric knowledge ratio'. Note: packs are Obsidian-compatible — eval results (ek_score) can be added to file frontmatter and queried in Obsidian via Dataview.
metadata: {"openclaw":{"homepage":"https://expertpack.ai","requires":{"bins":["python3"]}}}

ExpertPack Eval

Measure and evaluate ExpertPack quality. Companion to the core expertpack skill.

Note: This skill makes external API calls to OpenRouter for blind probing and LLM-as-judge scoring. Requires an API key.

1. Measure EK Ratio

Blind-probe frontier models to measure what percentage of a pack's propositions they cannot answer without the pack loaded:

python3 {skill_dir}/scripts/eval-ek.py <pack-path> [--models model1,model2] [--sample N] [--output FILE]

Default models: GPT-4.1-mini, Claude Sonnet 4.6, Gemini 2.0 Flash (via OpenRouter)
API key: Auto-resolves from OpenClaw auth profiles or OPENROUTER_API_KEY env var
Judge model: Claude Sonnet (GPT-4.1-mini is unreliable as judge — defaults to "partial")
Output: YAML with per-proposition scores and aggregate ratio

Interpretation:

EK Ratio	Meaning
0.80+	Exceptional — almost entirely esoteric
0.60–0.79	Strong — majority esoteric
0.40–0.59	Mixed — significant GK padding
0.20–0.39	Weak — most content already in weights
< 0.20	Minimal value-add

Add measured ratio to manifest.yaml:

ek_ratio:
  value: 0.72
  measured: "2026-03-12"
  models: ["gpt-4.1-mini", "claude-sonnet-4-6", "gemini-2.0-flash"]
  propositions_tested: 142

2. Run Quality Eval

Automated eval against a pack-powered agent endpoint:

python3 {skill_dir}/scripts/run-eval.py \
  --questions <eval-set.yaml> \
  --endpoint <ws://host:port/path> \
  --output <results.yaml> \
  --label "baseline"

Build eval set: 30+ questions (basic, intermediate, advanced, out-of-scope)
Fix one dimension at a time: structure → agent training → model
Re-run after each change to verify improvement

Learn more: expertpack.ai · GitHub