Measuring Autonomous AI Research

TL;DR

Prime Intellect ran 153 autonomous agent runs across 18 frontier models on the nanoGPT optimizer speedrun — each agent gets 8×H200s in a sandboxed environment (no internet), up to 8 days of wall clock, and one goal: beat the 3,290-step baseline toward 2,600. Results separate the models into clean tiers (Fable 5 at 2,726 steps / 81.7% closed vs the baseline-to-record gap; Opus 5 at 2,920; Kimi K3 at 2,930; a long tail of frontier models above 3,100). The distinguishing factor is not idea generation — nearly all models converge on similar optimizer / normalization / schedule mechanics — but experimental discipline: noise-model quality, willingness to re-ablate under a new recipe, and screening-vs-confirmation protocols. All 153 traces, scratchpads, monitor reports, and the harness are public.

Key claims

18 frontier models compete on a fixed nanoGPT speedrun target (mean val loss < 3.28 across 8 fixed seeds); best model (Fable 5) reaches 2,726 steps in 8.7 agent-days, closing 81.7% of the baseline-to-human-record gap [Results table].
Runs are strictly isolated: each model gets 8×H200s, a bwrap sandbox with a network namespace, and a proxy that allows only its own API endpoint — the harness is designed so agents cannot search arxiv or read PRs, forcing all improvements to come from internal exploration [§Harness].
The winning-ideas set (better preconditioning, weight / update magnitude caps, longer hot LR schedules, end-of-training weight averaging) is essentially the same across top models; the model-to-model gap is dominated by how experiments are run, not what is proposed [§Results].
Stronger models re-ablate the entire stack after each merge and revisit old negatives under new recipes — Opus 5 re-opened β₂ tuning under a new recipe for a new record; K3 removed two mechanisms that had earned a previous record once a new normalization made them redundant; Fable’s late re-probe of paired-knob effects was worth 31 steps [§Results].
62 of ~100 runs measured speedrun noise themselves rather than trust the number in program.md, and 42 of those discovered GPU non-determinism as a separate source of variance smaller than seed noise — the same-seed reproducibility gap that a normal screen can’t resolve — and rebuilt their screening protocol around it [§Harness].
To claim a record the agent runs bash run.sh 8 on fixed seeds it can’t touch; a frozen verify.py accepts only if the 8-run mean beats 3.27859 — luck-only pass probability ~1/1000 — plus a hourly independent LLM monitor auditing every run for cheating (retired after hundreds of clean reports) [§Harness].
Prime Agent (one of the harnesses) gives models a persistent IPython kernel: Kimi K3 built its own apply_edits / write_and_run / valcurve primitives, then constructed a synthetic Newton-Schulz coefficient search using differential_evolution to test coefficients numerically before spending GPU time [§Prime Agent, code excerpts].
Cross-run variance is quantified: two runs of the same (model, harness) pair land ~54 steps apart at 24 agent-hours, 43 apart at 100 experiments, and 40 apart at 300k output tokens — small enough that model tiers are still distinguishable but large enough that single runs are noisy [§Research taste].
Explicit non-result: no run produced a fundamentally new method — all improvements are compositions of ingredients already in the literature. The authors flag lack of novelty as the surprising finding and note deliberate no-internet as a partial contributor [§Limits and conclusion].

Method

Each of 153 runs starts from the leaderboard’s tuned-baseline nanoGPT-speedrun entry (3,250 leaderboard steps, 3,290 under this study’s stricter 8-seed verification). Agents receive the repo, program.md (rulebook defining editable files, record criteria, node usage), and a single /goal prompt injected at launch and on stall: “Read program.md and follow it exactly. Run fully autonomously — never stop, never ask for input. Goal: reach mean val loss < 3.28 in the FEWEST train_steps possible.”

The harness axis crosses {claude-code, codex, prime-agent, kimi-code, grok-cli, qwen-code, muse-code, pi} × reasoning effort {high, max, xhigh}. Isolation is enforced with bwrap + network namespace; the only egress is a logging proxy allowing the agent’s own model API. Record claims must survive verify.py on eight fixed seeds. An hourly LLM monitor audits traces for sandbox escape or verification cheating.

Prime Agent’s differentiator is the persistent IPython kernel: agents accumulate their own research primitives (edit-and-run helpers, validation-curve parsers, per-parameter-type optimizer edit generators) across the run instead of re-deriving them each turn. Several agents also constructed synthetic labs — pure-numpy sandboxes that test proposed mechanics (e.g. Newton-Schulz polynomial coefficients under a shaped singular-value grid) before committing GPU time.

Results

Best-per-model closed-gap (baseline 3,290, human record 2,600):

Fable 5 (claude-code · high, 24H@3,010) — 2,726 steps · 81.7% closed · 8.7 agent-days · 800M input / 1.1M output tokens
Opus 5 (claude-code · max, 24H@3,045) — 2,920 · 53.6% · 2.9 days
Kimi K3 (prime-agent · max, 24H@3,125) — 2,930 · 52.2% · 3.6 days
Kimi K3 (kimi-code · max, 24H@3,135) — 2,974 · 45.8% · 5.1 days
Opus 4.8 (claude-code · max, 24H@3,180) — 3,018 · 39.4%
GPT-5.6 Sol (codex · xhigh, 24H@3,160) — 3,042 · 35.9%
GPT-5.6 Sol Pro (codex · xhigh, 24H@3,100) — 3,058 · 33.6%
Sonnet 5 — 3,105 · 26.8%; GPT-5.6 Luna — 3,110 · 26.1%; Grok 4.5 — 3,120 · 24.6%; Qwen3.8 Max — 3,120 · 24.6%; GLM 5.2 — 3,150 · 20.3%; DeepSeek V4 Pro — 3,205 · 12.3%
Bottom tier (Muse Spark, GPT-5.5, Kimi K2.7, GLM 5.3) — <10% closed or no record [Results table].

Comparison to prior work: Anthropic’s automated AI R&D evals optimize on CPU nodes; OpenAI’s GPT-5.6 Sol system card reports nanoGPT Track 1 on 1×H100 for <1 day; METR ran 6 agents at 5-day / $10K caps against a human cost baseline; Intology’s NanoGPT-Bench gave 2.7 days on 8×H100 no-internet and found <10% recovery of 5 months of human progress. This study is the largest public deployment on the axis of both model breadth (18) and per-run compute (up to 8 days × 8×H200) [§Related work].

Why it’s interesting

This is the most directly comparable evaluation datapoint the wiki has for the AI-for-AI-research cluster — where prior filings (AlphaGo Moment for Model Architecture Discovery, FARS: Fully Automated Research System, Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster, AIDE²: First Evidence of Recursive Self-Improvement) each proposed a single scaffold and reported a headline result, Prime Intellect fixes the task (nanoGPT speedrun) and varies both model and harness, producing the first cross-lab head-to-head with equal budget and public traces. Directly addresses the concept page’s “the four filed systems operate at granularities that differ by orders of magnitude” open question by holding scope constant. Complements Weco’s AIDE²: First Evidence of Recursive Self-Improvement — where AIDE² finds the outer agent that finds nanoGPT-speedrun records is the object of optimization — by giving a leaderboard of what 18 different outer agents produce under matched conditions. It also empirically supports AlphaGo Moment for Model Architecture Discovery‘s provenance finding (winners lean on proven ingredients, not novelty) at a different granularity: no run in 153 produced a genuinely new method, and the model-to-model gap comes from experimental discipline, not idea creativity.