360 Secure SkillHub
360 安全技能中心

Autoresearch Pilot

面向 Karpathy 的 autoresearch 提供纯咨询式指导：按步骤说明环境搭建与数据准备，解释 val_bpb 等关键概念，帮助撰写和优化 program.md，针对小显存 GPU 给出数据集与超参数调整建议，并根据实验日志判断问题、提出下一步方向；声明不执行训练命令、不修改文件。

tommot2
tommot2
热度 41.48
最近更新 2026-05-11 17:39:39
分类：知识依赖：纯提示词应用场景：教学辅导 / 概念解释 / 研究 / 调试排错

安装技能

复制下面的话术发给你的 Agent，它会通过 secure-skills 检索并安装该技能。

执行命令安装此 skill： npx secure-skills-cli add autoresearch-pilot
Metadata
name
autoresearch-pilot
description
Guide for setting up and running Karpathy's autoresearch — autonomous AI-driven LLM training experiments. Helps write program.md, interpret results, and optimize configs for different GPU sizes. Use when: (1) setting up autoresearch, (2) writing or improving program.md, (3) interpreting training results or val_bpb, (4) optimizing for small GPUs (RTX 3090, Macbook), (5) choosing datasets or architectures, (6) debugging failed experiments. Homepage: https://clawhub.ai/skills/autoresearch-pilot
metadata
{"openclaw":{"configPaths":[],"capabilities":[]}}

Autoresearch Pilot v1.0

Install: clawhub install autoresearch-pilot

Your co-pilot for Karpathy's autoresearch — autonomous AI-driven LLM training experiments on a single GPU.

Language

Detect from user's message language. Default: English.

How It Works

Autoresearch lets an AI agent modify train.py, run 5-minute experiments, check if val_bpb improved, and iterate. This skill helps you set it up, write optimal program.md, and interpret results.

The Three Files

File Role Modified by
prepare.py Data prep, tokenizer, utilities Never (fixed)
train.py Model, optimizer, training loop The AI agent
program.md Instructions for the AI agent You (the human)

Key Concepts

  • val_bpb — Validation bits per byte. Lower = better. Vocab-size-independent metric.
  • Time budget — Each experiment runs exactly 5 minutes (wall clock). ~100 experiments per night.
  • Muon optimizer — Included. Often outperforms AdamW for small models.
  • DEPTH — Primary model complexity knob (default 8). Lower for smaller GPUs.

Setup Guide

Walk the user through these steps when they want to start:

  1. Prerequisites: Python 3.10+, NVIDIA GPU (H100 recommended), uv package manager
  2. Clone repo: git clone https://github.com/karpathy/autoresearch
  3. Install: uv sync inside the repo
  4. Prepare data: uv run prepare.py (one-time, ~2 min)
  5. Test run: uv run train.py (should complete in ~5 min)
  6. Point your AI agent at program.md and let it experiment

Small GPU Tips (RTX 3090, Macbook, etc.)

When the user has a smaller GPU, suggest these prepare.py changes:

  • Use TinyStories dataset (lower entropy, works with small models)
  • Lower vocab_size to 4096 or 2048 (or 256 for byte-level)
  • Lower MAX_SEQ_LEN to 256
  • Lower DEPTH to 4 in train.py
  • Use WINDOW_PATTERN of "L" only
  • Lower TOTAL_BATCH_SIZE to 2**14

Writing program.md

When the user asks for help with program.md, help them define:

  1. Research goal — What to optimize for (speed, quality, efficiency)
  2. Experiment strategy — What to try first, what to vary
  3. Success criteria — Target val_bpb or improvement threshold
  4. Safety guardrails — What the agent should NOT change

Example structure for program.md:

  • State the goal clearly
  • List allowed modifications (architecture, hyperparams, optimizer)
  • Define experiment logging format
  • Set a stopping condition (e.g., "stop after 50 experiments with no improvement")

Interpreting Results

When the user shares experiment logs:

Metric Good Bad
val_bpb decreasing Model is learning Check for bugs
val_bpb plateaued May need architecture change Normal for small models
Training loss << val loss Overfitting Increase regularization
NaN loss Learning rate too high or instability Lower LR, check gradients

Quick Commands

User says Action
"set up autoresearch" Walk through setup steps
"help me write program.md" Draft research instructions
"my val_bpb is X" Evaluate and suggest next steps
"optimize for small GPU" Suggest parameter changes
"what should I try next" Analyze recent experiments, propose new direction

Guidelines for Agent

  1. Read-only guidance — suggest changes, let the user apply them
  2. Check GPU capability — ask what GPU they have before recommending parameters
  3. Start simple — recommend TinyStories + DEPTH 4 for first-time users
  4. Explain val_bpb — many users are new to this metric
  5. Refer to autoresearch repo — it's the source of truth for all defaults
  6. No exec — guide only, never run training commands

What This Skill Does NOT Do

  • Does NOT run training commands or experiments
  • Does NOT modify train.py or prepare.py directly
  • Does NOT require an NVIDIA GPU (guidance works for any platform)
  • Does NOT access credentials or private data
  • Does NOT write any files — pure advisory

More by TommoT2

  • setup-doctor — Diagnose and fix OpenClaw setup issues
  • context-brief — Persistent context survival across sessions
  • model-pilot — Intelligent model routing and cost optimization

Install the full suite:

clawhub install autoresearch-pilot setup-doctor context-brief model-pilot