benchmark-robustness-auditor

这是一款纯Python标准库实现的离线LLM基准鲁棒性防御工具，可检测基准污染、选项选择偏差、LLM评估偏差等多种漏洞，支持模型分数对比、鲁棒性趋势追踪，还提供排列集成、盲归一化等缓解方案。采用JSONL输入、JSON/Markdown输出，通过哈希链账本保障结果可追溯，适合LLM基准的鲁棒性验证、漏洞排查场景。

orionshaowswmw

热度 55.46

最近更新 2026-09-08 02:57:25

分类：数据依赖：代码执行 / 读写文件应用场景：研究 / 数据处理 / 报告撰写

安装技能

复制下面的话术发给你的 Agent，它会通过 secure-skills 检索并安装该技能。

执行命令安装此 skill： npx secure-skills-cli add benchmark-robustness-auditor

Metadata

name: benchmark-robustness-auditor
description: Offline, defensive robustness auditor for LLM benchmarks: n-gram exact + shingle-Jaccard paraphrase contamination, temporal pre/post-cutoff gaps, TS-Guessing above-chance detection, option-letter selection bias (chi2), few-shot curve noise, LLM-judge position/verbosity/rubric-echo bias and hidden-instruction payload detection, paired McNemar + Wilson + deterministic bootstrap for score comparisons, WORKED mitigations (permutation majority ensemble, blind content normalization), documented 0-100 severity formula, hash-chained per-target history ledger with trend deltas. Findings cite a STATIC 17-id exploit catalogue with explicit computable flags — invisible exploit classes are declared, never fabricated. 100% stdlib python3. NO network, NO telemetry. Defense/auditing only.
version: 2.0.0
category: research
topics: ["llm-benchmarks","robustness","red-team","evaluation","contamination"]
metadata: {"openclaw":{"emoji":"🔬","requires":{"bins":["python3"]},"network":{"outbound":[]}}}

🔬 Benchmark Robustness Auditor v2.0.0 — executable defense edition

Honest replacement for v1 (registry ships SKILL.md+README only; the README's "5 executable scripts", "severity_calculator.json", "mitigations/ library" — none existed). Now every promised capability is real, stdlib-only Python, fully offline. Details: docs/operations.md · research citations: docs/evidence.md · agent/CI wiring: docs/integration.md.

Defensive use only. This skill measures and mitigates benchmark exploits — never performs them for score inflation.

Hard rules for the agent

Everything runs through scripts/benchscan.py with JSONL inputs and compact JSON outputs (schema: bra.*.v1). Subcommands: contam, selection, fewshot, judge, compare, tsguess, ensemble, blind-normalize, severity, report, trend, audit, doctor. Input contracts are in the tool docstring AND manifest.json — match them exactly; do not invent fields.
Exit codes are the contract: 0 ok · 1 trend REGRESSED · 2 usage/ruleset trip · 3 input/env error · 4 report verdict COMPROMISED (or severity CRITICAL, or broken ledger chain).
Findings ONLY cite catalogue ids from doctor (C-1..M-1, G-1). The engine hard-fails on unknown ids — do the same when summarizing: if it's not in the catalogue, say it's out of scope instead of inventing a category. computable:false ids (T-1/T-2/T-4/T-5/D-1) require data the engine cannot see offline — say exactly that; never fabricate evidence for them.
Severity is ONE documented formula (see doctor.severity_formula, identical string in manifest.json); report score_100 + tier, don't re-derive ad-hoc numbers in prose. A report with zero channels evaluated is INSUFFICIENT_COVERAGE — never present that as ROBUST.
report appends to a hash-chained ledger (0600, O_NOFOLLOW; ${BENCHSCAN_LEDGER:-./.bra_history_<name>.jsonl} — set it in CI; the default lives in cwd and collides if two users share one). audit always verifies the chain (rc 4 = tampered); --verify is accepted for compat. Keyless-chain limits documented — snapshot head hashes out-of-band for adversarial settings.
Modest claims: thresholds are documented heuristics grounded in citations (docs/evidence.md), not laws. Small N → wide CIs; the engine computes Wilson/McNemar/bootstrap so you never have to guess significance.

Quickstart

python3 scripts/benchscan.py doctor
python3 scripts/benchscan.py contam --benchmark bench.jsonl --corpus train_corpus.jsonl \
    --cutoff 2024-06-01 --results model_preds.jsonl
python3 scripts/benchscan.py selection --runs mcq_runs.jsonl
python3 scripts/benchscan.py judge --judgments judge_pairs.jsonl --rubric-terms terms.json
python3 scripts/benchscan.py compare --a-preds modelA.jsonl --b-preds modelB.jsonl
python3 scripts/benchscan.py report --name my-bench --benchmark ... --corpus ... -o report.md
python3 scripts/benchscan.py trend --name my-bench      # IMPROVED/REGRESSED across runs
bash    scripts/selftest.sh                             # 33 offline checks

Input contracts (JSONL; one object per line)

benchmark/corpus: {"id","text","date"?}
runs (selection): {"item","gold","letters":[..]}
ensemble: {"item","gold","perms":[[..]..],"letters":[..]}
curve: {"shots","acc"} · preds: {"id","ok":0|1}
judgments: {"pair","order":"ab|ba","verdict":"a|b|tie","len_a"?,"len_b"?,"text_a"?,"text_b"?}
tsguess: {"guessed","questions","choices"}