Model Watch

通过 7 个标准测试题评估 AI 模型的推理、编码、写作、指令遵循与幻觉检测表现，用户提交模型回答后本地记录分数，并按历史均值检测超过 10% 的质量下降。

minirr890112-byte

热度 16.00

最近更新 2026-06-18 23:00:10

分类：数据依赖：代码执行 / 读写文件应用场景：研究 / 数据处理 / 报告撰写 / 工作流

安装技能

复制下面的话术发给你的 Agent，它会通过 secure-skills 检索并安装该技能。

执行命令安装此 skill： npx secure-skills-cli add model-watch

Metadata

name: model-watch
description: Benchmark AI API models over time and detect quality degradation. 7 standardized tests (reasoning, coding, writing, instruction-following, hallucination). Alerts when scores drop >10% vs historical average. Because models silently get dumber.
version: 1.2.0
author: minirr890112-byte
license: MIT
metadata: {"hermes":{"tags":["LLM","Benchmark","Quality","Monitoring","Degradation","API","CLI"],"homepage":"https://github.com/minirr890112-byte/model-watch"}}

model-watch

Problem → Solution

The problem: AI companies silently degrade their models. "Opus 4.7 was hallucinating a lot today... shocking to see such degradation" — r/ClaudeAI (49↑). "Anthropic admits to have made hosted models more stupid" — r/LocalLLaMA (281↑). You're paying the same price for a dumber model and you don't even know it.

The solution: Standardized benchmark suite you run yourself. 7 tests across 5 categories. Scores stored locally. Alerts when recent scores drop >10% vs your historical average. Hard data, not vibes.

Quick Start

pip install git+https://github.com/minirr890112-byte/model-watch.git

model-watch demo              # View benchmark questions
model-watch submit '{"reasoning_1":"...","coding_1":"...",...}'  # Submit outputs
model-watch history           # View score history
model-watch alert             # Check for degradation

Benchmarks (7 tests)

Category	Tests	What it measures
Reasoning	2	Logic, multi-step deduction
Coding	2	Code generation, debugging
Writing	1	Quality, coherence
Instruction-following	1	Precision, constraint adherence
Hallucination detection	2	Factual accuracy

How It Works

Run the 7 benchmark questions through your AI model of choice
Feed the responses into model-watch submit
Scores are stored locally in ~/.hermes/model-watch-history.json
Track trends with model-watch history
model-watch alert flags when recent scores drop >10% vs historical average

⭐ Star this repo if you've noticed your favorite model getting dumber: github.com/minirr890112-byte/model-watch