Aa Benchmarking Framework

该 Skill 规划为 LLM 多维度评估框架：按可配置权重合成准确率、延迟、成本、一致性等维度得分，检测 Pareto 非支配前沿，绘制雷达图，并计划支持统计显著性检验、LangFuse 评估数据接入与 CSV/JSON 导出。当前状态为草稿，尚未实现，仓库内仅有说明文档，无代码文件。

nissan

热度 53.11

最近更新 2026-05-11 17:37:31

分类：数据依赖：代码执行应用场景：数据处理 / 报告撰写 / 研究

安装技能

复制下面的话术发给你的 Agent，它会通过 secure-skills 检索并安装该技能。

执行命令安装此 skill： npx secure-skills-cli add aa-benchmarking-framework

Metadata

name: aa-benchmarking-framework
version: 0.1.0
status: draft
description: Composite scoring and efficiency frontier analysis for LLM evaluation — combines multiple quality dimensions (accuracy, latency, cost, consistency) into a single Pareto-optimal ranking. Use when comparing models or agent configurations across competing objectives, building evaluation dashboards, or identifying the efficiency frontier for model selection. Implements weighted composite scores, Pareto frontier detection, and radar chart visualisation for multi-dimensional LLM benchmarking.
requires: {"env":[],"bins":["python3"]}
metadata: {"openclaw":{"primaryEnv":"production","network":{"outbound":false},"security_notes":"Draft skill — not yet published"}}

Last used: 2026-03-24 Memory references: 1 Status: Active

AA Benchmarking Framework

STATUS: DRAFT — This skill is planned but not yet fully implemented.

What This Does

Provides a systematic framework for multi-dimensional LLM evaluation using composite scoring, efficiency frontier analysis, and Pareto optimality. Rather than ranking models on a single metric, it helps identify which models are non-dominated — i.e., no other model is better on all dimensions simultaneously. Designed for teams that need principled model selection beyond simple leaderboard rankings.

Planned Capabilities

Composite scoring with configurable dimension weights (accuracy, latency, cost, recall, F1)
Pareto frontier detection across any two or more evaluation dimensions
Radar/spider chart visualisation for multi-dimensional comparison
Statistical significance testing across benchmark runs (t-test, Mann-Whitney U)
Integration with LangFuse for trace-based evaluation data ingestion
Export to CSV/JSON for downstream analysis

When To Use

Choosing between 3+ LLM providers on competing objectives (e.g. GPT-4o vs Claude 3.5 vs Gemini)
Building an evaluation dashboard for recurring model benchmarks
Presenting model selection rationale to stakeholders with visual evidence
Running efficiency frontier analysis to identify cost-optimal models for a quality threshold