LLM Evaluator Pro

该 Skill 通过 Langfuse 获取 trace，并借 OpenRouter 调用 GPT-5-nano 进行 LLM-as-a-Judge 评估，对相关性、准确性、幻觉和有用性给出 0–1 评分；支持单条评分、指定评估器、批量回填和测试模式，结果写入 Langfuse 可观测性面板。

aiwithabidi

热度 80.26

最近更新 2026-05-10 17:37:54

分类：数据依赖：代码执行 / 第三方 API / 内部调用大模型应用场景：数据处理 / 自动化脚本 / 工作流 / API 集成

安装技能

复制下面的话术发给你的 Agent，它会通过 secure-skills 检索并安装该技能。

执行命令安装此 skill： npx secure-skills-cli add llm-evaluator-pro

Metadata

name: llm-evaluator
version: 1.0.0
description: LLM-as-a-Judge evaluator via Langfuse. Scores traces on relevance, accuracy, hallucination, and helpfulness using GPT-5-nano as judge. Supports single trace scoring, batch backfill, and test mode. Integrates with Langfuse dashboard for observability. Triggers: evaluate trace, score quality, check accuracy, backfill scores, test evaluator, LLM judge.
license: MIT
compatibility: {"openclaw":">=0.10"}
metadata: {"openclaw":{"requires":{"bins":["python3"],"env":["OPENROUTER_API_KEY","LANGFUSE_PUBLIC_KEY","LANGFUSE_SECRET_KEY"]}}}

LLM Evaluator ⚖️

LLM-as-a-Judge evaluation system powered by Langfuse. Uses GPT-5-nano to score AI outputs.

When to Use

Evaluating quality of search results or AI responses
Scoring traces for relevance, accuracy, hallucination detection
Batch scoring recent unscored traces
Quality assurance on agent outputs

Usage

# Test with sample cases
python3 {baseDir}/scripts/evaluator.py test

# Score a specific Langfuse trace
python3 {baseDir}/scripts/evaluator.py score <trace_id>

# Score with specific evaluator only
python3 {baseDir}/scripts/evaluator.py score <trace_id> --evaluators relevance

# Backfill scores on recent unscored traces
python3 {baseDir}/scripts/evaluator.py backfill --limit 20

Evaluators

Evaluator	Measures	Scale
relevance	Response relevance to query	0–1
accuracy	Factual correctness	0–1
hallucination	Made-up information detection	0–1
helpfulness	Overall usefulness	0–1

Credits

Built by M. Abidi | agxntsix.ai YouTube | GitHub Part of the AgxntSix Skill Suite for OpenClaw agents.

📅 Need help setting up OpenClaw for your business? Book a free consultation