research-harness

Version: 1.3.2 Cognitive discipline for AI-native scientific experimentation — guardrails, not recipes.

When to Use

Trigger this skill when the user:

• Sets up a new AI-native research experiment repo
• Designs controlled experiments with LLM agents
• Needs reproducible evaluation, statistics, and error analysis
• Wants to structure a research workspace for long-running agent collaboration
• Wants agent-safe research governance that prevents overclaiming
• Says anything like: "research harness", "experiment framework", "AI科研", "对照实验", "评分体系", "可复现性", "科研workflow", "agent协作科研", "可控实验", "效应量"

Core Philosophy

This skill does not prescribe what experiment to run. It prescribes how to think while running it.

Research agents fail not because they lack capability, but because they:

• Scale before validating the minimum loop
• Overclaim what the data proves
• Treat surprising results as methodology failures before checking the execution chain
• Delete failed runs to make progress look cleaner
• Change baselines or rubrics silently
• Invent designs from thin air without tracing them to published precedents or stated assumptions

The antidote is cognitive discipline — a set of non-negotiable mental habits enforced by repo structure, not by prompt reminders. Detailed reasoning for each discipline is in references/scientific-thinking.md.

A critical addition from real-world practice: governance before scale. When a closed loop produces strong signals, the most tempting mistake is to expand immediately. The correct move is to lock down claim boundaries, audit artifacts, fix provenance gaps, and only then scale. A dedicated calibration phase between closed-loop and full-scale expansion is a sign of maturity, not delay.

Six Cognitive Disciplines

Disciplines 1-2: experiment design. 3-4: scoring & statistics. 5: critical reasoning. 6: methodology accountability.

Seven Governance Rules

Details in references/agent-collaboration.md.

Phase Workflow

Phase 0 · Scaffold

Goal: Set up the three-layer repo and root entry files.

• thinking-space/ — research direction, claims, decisions (human)
• execution-layer/ — briefs, logs, results, drafts (agent)
• code-workshop/ — runnable artifacts, packages

Root files: AGENTS.md (workspace map), PLAN.md (phase panel), WORKFLOW.md (procedure), harness/README.md (governance).

Directory skeleton and rationale: references/repo-architecture.md.

Phase 1 · Harden

Goal: Make the repo self-checking before formal execution.

1. Module contracts — Each core module gets a CONTRACT.md (purpose, inputs, outputs, invariants, local validator). Template in references/repo-architecture.md.
2. Local validators — scripts/validate_<module>.py per module; scripts/validate_repo_state.py as aggregator. Gate rule: 0 FAIL before any formal run.
3. Multi-level audit — Structure validation as layered gates: schema-valid → evidence-governed → planning-useful → experiment-ready. Use a default mode (WARN only) during early phases and a strict mode (FAIL on gaps) before scaling. This keeps the repo moving without compromising the expansion gate.
4. Experiment manifest — experiments/results/manifest.csv as run-level provenance ledger (run_id, wave, task_id, group, model, version metadata, status, retry_of, git_commit).
5. Protected surfaces — Baselines, rubrics, raw results, scoring config, schema. Require version bump + proposal to change.

Phase 2 · Design

Goal: Design attributable controlled experiments with grounded methodology.

• Progressive building: minimum artifacts → schema validation → small task set → dry run → scoring → expand. Design details in references/experiment-design.md.
• Controlled groups: Baseline → incremental treatments. Adjacent groups differ by exactly one variable.
• Gold checklists: Every task has must_include, forbidden, and scoring_notes. For multi-group experiments, separate gold checklists into planning_gold (items all groups can achieve) and evidence_gold (items only augmented groups can access).
• Metric fairness annotation: Each evaluation metric must declare evidence_access_required (true/false). Score families with different access requirements must be reported separately, never mixed into a single total. Details in references/scoring-statistics.md.
• Artifact QA: Creating knowledge artifacts (cards, schemas, tasks) follows a scaffold→validate→commit cycle. Validators catch syntax errors, enum mismatches, and registry sync issues before they accumulate. Details in references/repo-architecture.md.
• Output contract: Agent output follows a strict schema (YAML/JSON). The scorer and analysis pipeline depend on this contract.
• Design justification (Discipline 6): Before implementing any non-trivial schema, scoring system, or experiment protocol, write a Design Justification Document. Template and grounding framework in references/methodology-grounding.md. This is required, not optional.

Phase 3 · Execute & Analyze

Goal: Run experiments, score, compute statistics, analyze errors.

Preflight gate: local validators must pass. Then:

1. Dry run — print prompt, no API call
2. Smoke run — 1 task × 2 groups, verify output parsing
3. Wave 1 — small set × all groups, minimum viable data
4. Scoring: Track A (rule-based) + Track B (semantic) cross-validation. Details in references/scoring-statistics.md.
5. Statistics: Cohen's d primary, 95% CI, paired t, Wilcoxon. --reproduce flag for one-click reproducibility.
6. Error analysis: hallucination, output depth, specificity, task appropriateness.

Phase 4 · Handoff & Writing

Goal: Package results for the next phase or agent.

• Claim-safe memo: After any closed loop that produces strong signals, write a memo separating: supported findings, positive signals, not-yet-supported claims, and required next evidence. This prevents overclaiming and directly drives the next phase. Never jump to expansion without calibrating the claim boundary first.
• Alignment doc: ~1 page with state, entry files, new surfaces, preflight commands, protected surfaces. Never pass chat history.
• Upstream proposals: Any insight affecting direction goes to sync/upstream_proposals/ first. Template in references/agent-collaboration.md.
• Writing markers: [REF-MISSING], [CRITICAL-CHECK], [TODO]. Never use AI numbers without verification.

Non-Negotiables

1. No unverified citation becomes a research fact
2. No debug result becomes a formal result
3. No agent changes baseline, rubric, or metric definitions without a proposal
4. No raw result is overwritten
5. No failed experiment is deleted
6. No phase gate passes before validators report zero FAIL
7. No closed loop expands to full scale before calibration: lock down claim boundaries, audit artifacts, and fix provenance gaps first
8. No non-trivial design enters execution without a Design Justification Document (Discipline 6)

Theoretical Alignment: ETCLOVG Mapping

The ETCLOVG seven-layer taxonomy from Agent Harness Engineering: A Survey (Li et al., 2026, CMU/Yale/JHU) provides an external validation of this skill's structure and reveals two coverage gaps:

Key Insights from ETCLOVG

1. Harness Coupling Problem: Changes to any ETCLOVG layer are not local optimizations — they propagate. Adding a tool changes context window usage, which changes verification reliability. Rule: treat any layer change as a system-level change; re-validate all layers.

2. Cost–Quality–Speed Trilemma: Stronger sandboxing, richer context, deeper evaluation improve quality but cost tokens, latency, and infrastructure. When a closed loop produces strong signals, resist the urge to add more harness. Instead, audit what can be removed without quality loss (Anthropic's finding: some context resets help old models but hurt newer ones).

3. Trace-Native Evaluation: Agent evaluation should not be "leaderboard-style" (who scores highest) but "quality control" (why did it fail, which layer caused it, is the evaluator itself trustworthy). Record full execution traces; judge path quality, not just outcomes.

Actionable Supplements (v1.4.0 candidates)

• Observability module: Extend manifest.csv to a traces/ directory per run, recording model calls, tool results, state snapshots, errors, retries, and cost/latency.
• Path evaluation: Add "was the execution path reasonable?" as a separate scoring dimension alongside outcome correctness.
• Evaluator trustworthiness: Before trusting an automated evaluator, ask: "Is the evaluator itself a potential failure point?" (Guerdan et al., NeurIPS 2025)
• Harness decoupling audit: When changing any ETCLOVG layer, run a cross-layer re-validation before trusting the results.

Reference: Li, J., Xiao, X., et al. (2026). Agent Harness Engineering: A Survey. OpenReview. Project page

References

• references/repo-architecture.md — three-layer repo, module contracts, manifest, validators
• references/experiment-design.md — progressive building, controlled groups, gold checklists
• references/scoring-statistics.md — dual-track validation, effect size, reproducibility
• references/scientific-thinking.md — cognitive disciplines for agent-led research
• references/agent-collaboration.md — governance, evidence status, alignment docs
• references/methodology-grounding.md — theoretical grounding for schema, scoring, and experiment design