Autoresearch Approach

Use this for the repo cloned at /home/tj/.openclaw/workspace/external/autoresearch or for close variants.

Core idea

Follow the upstream program.md discipline:

• keep scope tiny
• modify only the allowed file
• run one bounded experiment at a time
• log every run
• keep winners, reset losers
• do not pause for permission once the loop starts

Read references/upstream-program.md before autonomous experimentation. Read references/upstream-readme.md when you need repo context, setup, or platform constraints.

Repo contract

In upstream autoresearch:

• prepare.py is read-only, it owns data prep, tokenizer, dataloader, evaluation, and the 5 minute time budget
• train.py is the only file the agent edits during experiments
• program.md is the human-authored operating manual for the agent
• metric is val_bpb, lower is better
• results.tsv stays untracked and records every run

If adapting the method to another repo, identify the equivalent of these three roles before doing anything else.

Setup workflow

1. Confirm the repo and branch base.
2. Propose a fresh run tag.
3. Create autoresearch/<tag>.
4. Read the small set of in-scope files fully.
5. Verify cache/data exists before the first run.
6. Initialize results.tsv with the header only.
7. Run the untouched baseline first.

Use scripts/init_autoresearch_run.sh <repo-path> <tag> to do steps 2 to 6 for the upstream repo.

Experiment loop

After setup, repeat:

1. Note the current good commit.
2. Make one coherent change to the allowed file.
3. Commit the experiment.
4. Run the bounded training command with stdout/stderr redirected to run.log.
5. Extract val_bpb and peak_vram_mb.
6. If the run crashed, inspect the tail, decide whether to fix once or discard.
7. Append one TSV row to results.tsv.
8. Keep the commit only if it improved the main metric.
9. Otherwise hard reset to the prior good commit.

Default run command upstream:

uv run train.py > run.log 2>&1

Default metric extraction upstream:

grep "^val_bpb:\|^peak_vram_mb:" run.log

Decision rules

Prefer changes that are:

• simple
• measurable
• easy to revert
• comparable under the fixed budget

Keep these heuristics:

• baseline first, always
• one idea per experiment when possible
• complexity must justify itself
• equal result with simpler code is a win
• if a run exceeds 10 minutes total, kill and treat as failure
• do not silently change the evaluation harness

Logging format

Upstream results.tsv is tab-separated with this header:

commit	val_bpb	memory_gb	status	description

Statuses:

• keep
• discard
• crash

Do not git-commit results.tsv unless the human explicitly asks.

Practical notes

• Redirect noisy training output to files, not chat.
• Read the exact upstream wording in references/upstream-program.md before long autonomous runs.
• For new platforms or smaller hardware, check references/upstream-readme.md for Karpathy's downscaling guidance.
• If a user asks for "the Karpathy autoresearch approach", use this skill and the cloned repo at /home/tj/.openclaw/workspace/external/autoresearch.