Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

TL;DR

SkyPilot took Karpathy’s autoresearch (an agent that edits train.py, runs a 5-minute training experiment, and keeps changes that lower val_bpb) and removed the one-GPU bottleneck by giving Claude Code a SkyPilot “skill” that teaches it to provision and manage Kubernetes GPU clusters on its own. Over 8 hours on 16 GPUs (13 H100s + 3 H200s on CoreWeave), the agent submitted ~910 experiments, ran factorial grids of 10–13 experiments per wave instead of serial hill-climbing, and drove val_bpb from 1.003 → 0.974 (a 2.87% improvement). The headline finding was qualitative: parallelism didn’t just speed up search, it changed what search the agent did — including an unprompted two-tier strategy where the agent screened ideas on cheap H100s and promoted winners to H200s for validation after noticing H200s scored ~0.005 better at the same config. Total cost: ~ $300 compute +$ 9 Claude API.

Key claims

One agent + SkyPilot skill drove 16 K8s GPUs to ~90 experiments/hour vs ~10/hour single-GPU baseline — a 9× throughput multiplier over an 8-hour overnight run, with ~700 valid results out of ~910 submissions [§Results].
Total optimization moved val_bpb from 1.003 (baseline) to 0.974 (2.87%), with the largest single jump coming from scaling model width (aspect ratio 48 → 96, model_dim 384 → 768) which outweighed every hyperparameter tweak combined [§Phase 2].
Five emergent phases of the search were not pre-planned by the human: hyperparameter sweep → architecture discovery → fine-tuning the wider model → optimizer tuning (muon_beta2 0.95 → 0.98 was the biggest late find) → diminishing returns [§Phase 1–5].
Parallelism shifted the agent’s strategy from greedy hill-climbing (single-GPU baseline) to factorial grids of 10–13 experiments per 5-minute wave, surfacing interaction effects between parameters that sequential search would miss [§How parallelism changed the agent’s research strategy, Table].
The agent discovered hardware heterogeneity on its own — without being told H100s and H200s differed — and developed an unprompted two-tier validation strategy: screen 10+ hypotheses on H100s, promote top 2–3 to H200 for confirmation (the H200’s 9% faster step time produced ~0.005 better val_bpb on identical configs at the 5-minute budget) [§Emergent research strategies].
Hardware-conditional ranking inversion was observed and caught by the emergent two-tier strategy: FINAL_LR_FRAC=0.03 sometimes beat 0.05 on H100 but consistently lost on H200, because more training steps reward keeping the LR higher at the end of the schedule [§Emergent research strategies].
The skill (instructions plus a SkyPilot YAML) is the entire interface: the agent reads instructions.md, fetches the SkyPilot skill, runs sky launch / sky exec -d (detached) to provision and pipeline experiments on each of 16 clusters, parses sky logs for EXPERIMENT_RESULT: lines, and commits winning changes to train.py [§Giving the agent cloud GPUs].
Total bill for the 8-hour run: ~ $200 (13 H100s @$ 2/h) + ~ $60 (3 H200s @$ 2.3/h) + ~ $9 Claude API ≈ under$ 300 [§Cost].
Returns flattened sharply in Phase 5: per-experiment Δval_bpb < 0.0001 once architecture scale + batch size + optimizer structure were dialed in, suggesting the 5-minute training budget caps what further search inside this train.py can extract without architectural novelty or budget extension [§Phase 5].
Best discovered config on the 5-minute nanochat-style budget: AR=96 (768 model_dim), depth 8, alternating Sliding+Local window pattern, batch size 2^18, Adam betas (0.70, 0.95), weight_decay 0.08, Muon beta2=0.98, matrix_lr=0.05, embedding_lr=0.6 [§Best configuration].

Method

The default autoresearch setup is three files: prepare.py (read-only data + eval), train.py (the only file the agent may edit), and program.md (instructions: a fixed 5-minute training budget; minimize val_bpb; everything in train.py is fair game so long as it runs). Out of the box, the loop is sequential: edit → wait 5 min → check → repeat, capped at ~12 experiments/hour by step 2.

SkyPilot’s contribution is an experiment.yaml that wraps one training run as a SkyPilot task (Kubernetes-backed, accelerators: {H100:1, H200:1} so SkyPilot picks what’s free), plus a SkyPilot “skill” — a self-contained markdown instruction file that Claude Code reads to learn the sky launch / sky exec -d / sky logs / sky status workflow. The setup block runs once per cluster (data prep, deps); subsequent sky exec calls on the same cluster skip straight to training. sky exec -d queues a job that auto-starts when the current one finishes, letting one agent pipeline experiments across 16 parallel clusters with zero idle time. Results are parsed from stdout (EXPERIMENT_RESULT: <id> val_bpb=<x> memory_gb=<y>). The human-side interface collapses to: claude "Read instructions.md and start running parallel experiments."

Results

Phase-level val_bpb trajectory across the 8-hour run [§Phase 1–5]:

Phase	Range (exp #)	val_bpb	Δ	Theme
0	baseline	1.003	—	—
1	0–200	0.981	0.022	hyperparameter sweep (batch, betas, weight decay, softcap)
2	200–420	0.977	0.004	architecture (AR=48→96 in one parallel wave of 6)
3	420–560	0.975	0.002	fine-tune wider model on H200
4	560–700	0.974	0.001	optimizer (Muon `beta2` 0.95→0.98)
5	700–910	~0.974	<0.0001	diminishing returns

Throughput: ~90 experiments/hour (16-GPU parallel) vs ~10/hour (single-GPU baseline). 16-GPU run information density per decision: 10–13 simultaneous experiments per wave vs 1 per decision in the serial baseline. Cost: ~ $300 GPU +$ 9 Claude API [§Cost].

Why it’s interesting

This is the first filed datapoint where an agent’s infrastructure — not its model, prompt, or training recipe — is the load-bearing change, and where the qualitative shift in search strategy (greedy hill-climbing → factorial grids → emergent two-tier hardware screening) matters more than the raw 9× speedup. It complements CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation from the opposite end: CUDA Agent trains a model to be good at kernel-optimization RL on a fixed 128-H20 sandbox, while this run uses an untrained general coding agent (Claude Code) and shifts the burden to the SkyPilot-skill substrate that lets the agent allocate its own compute — both are kernel/training optimization but the trained-policy vs scaffold split is sharp. It also sits adjacent to FARS: Fully Automated Research System (FARS) and the AI-Scientist lineage, but the unit of work is much smaller — a single train.py knob-tweak rather than a full paper — which is what makes the iteration count (910) and cost (~$300) workable. The emergent H100/H200 two-tier strategy is the most interesting bit: it’s a worked example of “agent given more axes of choice invents a workflow a human researcher would have to be told to use,” in the spirit of the closed-loop self-synthesis pattern that dominates Agentic Software Engineering but applied at the hardware-management layer.