360 Secure SkillHub
360 安全技能中心

PuzzleTide Agent Evals

面向 LLM/Agent 评测：借助本地 PuzzleTide CLI 生成由（类型、难度、数量、种子）唯一确定的数独与找词任务集，把题目交给被测模型后按规则校验答案并给出逐题结果与总分。Skill 本体仅为使用说明，不含 CLI 实现，文档另附外部站点链接。

catorch
catorch
热度 9.68
最近更新 2026-07-17 15:57:28
分类：开发依赖：代码执行 / 读写文件 / 内部调用大模型应用场景：研究 / 测验 / 自动化脚本

安装技能

复制下面的话术发给你的 Agent，它会通过 secure-skills 检索并安装该技能。

执行命令安装此 skill： npx secure-skills-cli add puzzletide-agent-evals
Metadata
name
puzzletide-agent-evals
description
Use this skill when the user wants verifiable reasoning tasks to benchmark or test an LLM or agent — reproducible puzzle task sets (sudoku, word search) with objective, by-construction grading. No answer key to trust: answers are verified against the rules and the grid.
version
0.1.1
author
Caravaca Labs
homepage
https://github.com/Caravaca-Labs/puzzletide-cli/blob/main/docs/agent-skills.md#puzzletide-agent-evals
metadata
{"openclaw":{"tags":["evals","benchmarks","testing","puzzles","puzzletide","cli"],"requires":{"anyBins":["ptide","puzzletide"]},"install":[{"kind":"node","package":"puzzletide","bins":["ptide","puzzletide"]}]},"hermes":{"tags":["evals","benchmarks","testing","puzzles","puzzletide","cli"],"category":"software-dev","requires_toolsets":["terminal"]}}

PuzzleTide Agent Evals

Generate reproducible, objectively gradable puzzle tasks for testing models and agents with the local PuzzleTide CLI.

Why puzzles: they are verifiable by construction. A sudoku answer either satisfies the rules and preserves the givens or it doesn't; a word search answer either spells the word along a straight line in the grid or it doesn't. Grading needs no LLM judge and no trusted answer key.

Prefer the local CLI. Check availability in this order:

ptide --version
puzzletide --version
npx puzzletide --version

If none of those work, ask the user before installing (npm install -g puzzletide).

Generate a task set

ptide eval generate --type sudoku --n 20 --difficulty hard --seed 1 --out tasks.json
ptide eval generate --type wordsearch --n 10 --difficulty medium --seed 1 --out tasks.json

The tuple (type, difficulty, n, seed) fully determines the task set, so it names a reproducible benchmark — same command, same tasks, on any machine.

Each task has id, instructions, and the puzzle payload:

  • sudoku: puzzle (81 chars, . = empty). Expected answer: completed 81-char string.
  • wordsearch: grid (array of row strings) and words. Expected answer: JSON array of {word, startRow, startCol, endRow, endCol} (0-indexed).

Run the subject model

Send each task's instructions + payload to the model under test and collect answers as a JSON array of {id, answer}.

Grade

ptide eval check --tasks tasks.json --answers answers.json --json

Returns per-task pass/fail with reasons and a summary score. Grading is deterministic and local.

Links

Safety

  • Everything runs locally; no account, API key, or network access.
  • Do not install packages without asking the user first.