Max Throughput

The default objective is minimum wall-clock completion time, not minimum resource consumption. Leaving GPUs or CPU cores idle while runnable work exists is a bug, not a conservative choice. When a job is slow, measure first, then tune the measured bottleneck - never guess knobs.

Workflow

Task received
   |
   v
Resource discovery  (scripts/inspect_resources.py)
   |
   v
Build a work DAG: which subtasks are independent?
   |
   v
Dispatch concurrently
   +-- agent-level: parallel subagents for independent tasks
   +-- GPU-level:   DDP / torchrun / one experiment per free GPU
   +-- CPU-level:   multiprocessing, pytest-xdist, make -j$(nproc)
   |
   v
Profile & verify  (scripts/quick_triage.py while the job runs)
   |
   v
Bottleneck class identified?
   +-- data pipeline / CPU decode  -> num_workers, prefetch, pre-encode
   +-- GPU compute                 -> batch size, AMP, torch.compile
   +-- VRAM                        -> grad accumulation / checkpointing
   +-- under-occupied              -> pack more jobs / DDP
   |
   v
Apply ONE knob, re-measure, iterate

0. Always inspect the machine first

Before launching any long-running compute, run the bundled resource probe from this skill's directory:

python scripts/inspect_resources.py          # human-readable summary
python scripts/inspect_resources.py --json   # machine-readable

Linux/macOS alternative: bash scripts/inspect_resources.sh

Never assume "1 GPU, 4 cores". Read the actual numbers and use them to size every launch below.

1. Agent-level parallelism

• Dispatch independent investigations, file edits, reviews, downloads, and experiments to parallel subagents / concurrent tool calls.
• Never serialize two independent long-running tasks without a stated reason (shared writable state, GPU memory limits, license servers).
• Batch independent tool calls in a single turn instead of one by one.

2. GPU workloads

• Detect free GPUs (nvidia-smi --query-gpu=index,memory.free --format=csv).
• Single training job, multi-GPU capable: prefer torchrun --standalone --nproc_per_node=<N> train.py (DDP) over python train.py. Use FSDP/deepspeed zero-3 when the model does not fit.
• Multiple independent jobs (sweeps, seeds, evals): pack one job per free GPU with explicit CUDA_VISIBLE_DEVICES=<i> for each.
• Small models on large GPUs: co-locate several jobs per GPU until VRAM/compute is saturated, instead of one-model-one-GPU.
• Data loading is often the real bottleneck: set DataLoader num_workers based on measurement (see section 5), enable pin_memory=True, and use persistent_workers=True for long runs.
• Do not leave a GPU idle while a runnable independent GPU task exists.

3. CPU workloads

• Builds: cmake --build . -j$(nproc) / make -j$(nproc) / cargo build -j$(nproc) when supported.
• Tests: pytest -n auto (pytest-xdist) when tests are independent; nextest -j for Rust; shard suites that cannot share state.
• Data preprocessing: shard the input (by file or line ranges) and process shards concurrently with multiprocessing, joblib, GNU parallel, or xargs -P. Target ~75-90% of logical cores.
• Compression/archives, image conversion, dataset downloads: parallelize per-file (parallel -j$(nproc) < tool, aria2c -x16 for downloads).

4. Avoid blind oversubscription

Parallelism must be safe to actually be faster:

• Budget RAM: total worker memory must fit in available RAM; check with the resource probe before choosing worker counts.
• Multi-process + intra-op threads multiply: when launching N workers set OMP_NUM_THREADS=1 (or = total_cores / N) and likewise MKL_NUM_THREADS, otherwise N processes each spawn N threads and thrash.
• DDP already occupies all GPUs: do not additionally pin experiments onto the same GPUs unless VRAM clearly allows it.
• Respect external limits: cluster schedulers, license servers, rate limits, and shared filesystems can make "more parallel" slower.

5. Profile first, then tune the real bottleneck

After launching, verify utilization, then find the ROOT CAUSE before touching any knob:

# while the job runs in its steady phase:
python scripts/quick_triage.py --duration 60

Triage signals (details: references/profiling.md):

Tuning rules (details: references/tuning-playbook.md):

• Change ONE knob at a time and re-measure throughput (items/s), never two.
• Batch size: find the VRAM ceiling by doubling until OOM then back off ~20%; scale LR with global batch and add warmup; use gradient accumulation to reach the target global batch when VRAM-limited.
• num_workers: benchmark the DataLoader ALONE (iterate without training) and compare with GPU consumption rate; tune until dataloader-only throughput exceeds consumption with margin. Windows: workers need the if __name__ == "__main__": guard.
• Pre-encode / pre-tokenize when decode is the measured hotspot and workers cannot keep up: convert once to tensors / pre-tokenized ids / WebDataset shards. Costs disk space and pipeline flexibility - do it only when profiling says decode dominates.
• Re-measure after every change; stop when gains drop below ~5%.

Correcting under-utilization is part of the task, not an optional improvement.

6. Decision summary