llm-verify
llm-verify is a single-binary CLI that runs a black-box check against any LLM
API endpoint and answers what the user actually wants to know: am I getting the
model I am paying for, am I being overcharged, and how many relays sit on this
path?
When to use this
Reach for this skill when the user asks anything like:
- "Is this relay/proxy trustworthy?" "Is this key really Claude?"
- "Has the model been downgraded?" "It feels dumber than it used to."
- "Is the billing right?" "The token counts look off."
- "Check this API endpoint for me."
- Debugging an endpoint that is slow, drops its stream, or breaks tool calls.
How to run it
llm-verify --base-url <URL> --api-key <KEY> --model <MODEL_ID>
| Flag | Meaning |
|---|---|
--protocol anthropic|openai |
Inferred from the URL and model name if omitted |
--depth fast|balanced|forensic |
Default balanced; forensic samples more — slower and costlier, but firmer |
--claimed-model <ID> |
Use when the vendor's advertised name differs from the ID you request |
--lang en|zh |
Report language; follows the system locale by default |
-o report.html |
HTML report path |
--json report.json |
Also emit machine-readable JSON |
--no-open |
Do not open a browser |
Credentials can also come from .env or the environment:
LLM_VERIFY_BASE_URL, LLM_VERIFY_API_KEY, LLM_VERIFY_MODEL.
Exit codes
| Code | Meaning |
|---|---|
| 0 | Clean |
| 1 | Failing score, or a suspicious / counterfeit / inconclusive verdict |
| 2 | A hard gate tripped (silent fallback, shared-pool forwarding, tier downgrade, wrapper injection, cache replay, hidden prompt, response replay) |
Suitable as a CI gate as-is.
Reading the result
The tool reports two independent axes. Do not conflate them.
- Authenticity: genuine / genuine-with-defects / relayed / suspicious / counterfeit / inconclusive
- Origin: direct from vendor / cloud platform / subscription-derived / relay / reconstructed channel / undetermined
A real model behind a relay is "relayed", not "counterfeit". That is a longer path, not a substituted model.
Reporting back to the user
- Lead with the conclusion, then the evidence. They want to know whether they can rely on it, not all 40 probe lines.
- Call out hard gates separately. They are facts no weighted score excuses.
- Not tested is not passed. Say plainly which probes were skipped.
- Carry the confidence across, especially for identity: adjacent versions inside one tier are genuinely hard to separate, and the tool abstains when the evidence is thin. Do not supply a verdict it declined to give.
- Do not convict on the tool's behalf. One tier apart is within sampling noise; the tool does not accuse there, and neither should you.
Limits to state honestly
- Resolution stops at tier granularity (flagship / mid / light). Adjacent versions inside a tier cannot be separated without distribution baselines.
- The tier call depends on sampling; use
--depth forensicwhen it matters. - An injected system prompt contaminates identity fingerprints, which is why the contract layer runs first and downgrades identity confidence when it finds injection.
- It cannot prove the server-side weights are the official ones — only that behaviour does or does not match expectations.
- Quantised builds (int4 / fp8) can only be given a probability, never a verdict.