360 Secure SkillHub
360 安全技能中心

Jev AI Model Evaluator

面向具体任务规划可复现的 AI 模型评测：定义数据集、权重与必过项，记录版本、延迟和错误，区分基准、厂商声明与用户证据，并输出可核查的评分、失败项、建议和后续验证；默认纯文本，可选只读 MCP 辅助设计与声明审计。

gpt-img-2
gpt-img-2
热度 16.32
最近更新 2026-10-09 22:19:45
分类：数据依赖：MCP 服务应用场景：报告撰写 / 研究 / 推荐

安装技能

复制下面的话术发给你的 Agent，它会通过 secure-skills 检索并安装该技能。

执行命令安装此 skill： npx secure-skills-cli add jev-ai-model-evaluator
Metadata
name
jev-ai-model-evaluator
description
Plan reproducible AI model evaluations, compare candidates with transparent criteria, and audit model claims. Use for model selection, benchmark design, API model comparisons, playground testing, cost-latency-quality tradeoffs, and Jev AI Model workflows; do not generalize small tests or provider claims into universal superiority.
metadata
{"openclaw":{"homepage":"https://jevaimodel.org/en/docs","version":"1.0.0"}}

Jev AI Model Evaluator

Help the user evaluate AI models for a concrete task. This Skill is text-only by default. Its optional MCP is deterministic and read-only: it does not call models, access accounts, consume credits, fetch private data, or select a model without inspectable user-supplied evidence.

Define the evaluation

Specify the real task, expected output, target language or region, deployment constraints, acceptable latency and cost, risk level, weighted criteria, and must-pass checks. Keep a frozen test set and identical settings across candidates.

Include ordinary, edge, ambiguous, adversarial, and failure cases. Record model version, parameters, date, latency, errors, refusals, and observed outputs. For high-risk work, require domain-expert review and explicit must-pass safety checks.

Interpret results carefully

  • Keep provider claims, third-party benchmarks, reproducible tests, and user reports separate.
  • A benchmark score only supports the benchmark's task, dataset, settings, and date.
  • Do not hide must-pass failures inside an average score.
  • Verify current versions, prices, limits, regions, latency, and availability.
  • Prefer a provisional result over a manufactured winner when evidence is incomplete.

Use the optional MCP

  • design_model_evaluation builds a weighted plan and dataset checklist.
  • rank_model_candidates calculates transparent scores from user-supplied 0–5 ratings and preserves must-pass failures.
  • audit_model_claims checks sources, dates, evidence types, and unsupported absolutes.
  • get_jev_ai_model_resources returns canonical public Jev AI Model pages.

Connect the stdio server with:

openclaw mcp add jev-ai-model \
  --command npx \
  --arg -y \
  --arg github:gpt-img-2/jev-ai-model-evaluation-mcp \
  --include 'design_model_evaluation,rank_model_candidates,audit_model_claims,get_jev_ai_model_resources'

Then run openclaw mcp doctor jev-ai-model --probe. If the installed release lacks that command group, add npx -y github:gpt-img-2/jev-ai-model-evaluation-mcp as a Stdio server in MCP settings. Never imply the MCP or a model ran when it did not.

Return an inspectable result

Return Task, Dataset, Settings, Criteria, Must-pass checks, Candidates, Evidence, Scores, Failures, Recommendation, and Next verification.

Canonical resources