LLM Evaluator

LLM-as-a-Judge 评测工具，通过 Langfuse 读取 trace，调用 OpenRouter 上的 GPT-5-nano 评委模型，对相关性、准确性、幻觉与有用性四项指标输出 0-1 评分；支持单条 trace 评分、按指定评委评分以及批量回填历史未评分 trace，并提供内置样例自测。

aiwithabidi

热度 20.10

最近更新 2026-05-10 17:37:49

分类：数据依赖：代码执行 / 第三方 API应用场景：数据处理 / 自动化脚本

安装技能

复制下面的话术发给你的 Agent，它会通过 secure-skills 检索并安装该技能。

执行命令安装此 skill： npx secure-skills-cli add agxntsix-llm-evaluator

Metadata

name: LLM Evaluator
version: 1.0.0
description: LLM-as-a-Judge evaluation system with Langfuse integration
author: aiwithabidi

LLM Evaluator ⚖️

LLM-as-a-Judge evaluation system using Langfuse. Score AI outputs on relevance, accuracy, hallucination, and helpfulness. Backfill scoring on historical traces. Uses GPT-5-nano for cost-efficient judging.

Usage

# Test with sample cases
python3 scripts/evaluator.py test

# Score a specific Langfuse trace
python3 scripts/evaluator.py score <trace_id>

# Score with a single evaluator
python3 scripts/evaluator.py score <trace_id> --evaluators relevance

# Backfill scores on recent unscored traces
python3 scripts/evaluator.py backfill --limit 20

Evaluators

relevance (0-1) — How relevant is the response to the query?
accuracy (0-1) — Is the response factually correct?
hallucination (0-1) — Does the response contain fabricated information?
helpfulness (0-1) — How useful is the response?

Requirements

OPENROUTER_API_KEY environment variable (for GPT-5-nano judge)
LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY environment variables
LANGFUSE_HOST — your Langfuse instance URL
Python 3.10+
langfuse, requests packages

Credits

Built by AgxntSix — AI ops agent by M. Abidi 🌐 agxntsix.ai | Part of the AgxntSix Skill Suite for OpenClaw agents