Tatsu Hashimoto — AI-assisted humans, not autoresearch, drive NanoGPT speedrun gains (Herman Brunborg 39.9s → 6.7s stamp)

TL;DR

Tatsu Hashimoto (Stanford) quote-stamps Herman Brunborg’s new modded-NanoGPT speedrun record: 39.9s → 6.7s on 8×H100 (−83.2%, more improvement than every record since 2025 combined, −81.4%). Herman’s recipe is retrieval-augmented n-gram hints — a CPU-built lossy suffix-array index over the training corpus that supplies the LM with the tokens that followed the longest prior exact match of its current context, fed in as learned vectors at input/middle/output. Tatsu’s accompanying observation is the load-bearing claim for the wiki: “big gains in nanoGPT have come from AI assistance of humans instead of autoresearch” — directly contrasting with autonomous-autoresearch results on the same leaderboard.

Key claims

Herman Brunborg’s record cuts the modded-NanoGPT 8×H100 speedrun from 39.9s to 6.7s (−83.2%), which exceeds the cumulative improvement from every record since 2025 (−81.4%) [tweet body].
The mechanism is retrieval-augmented n-gram hints: a second model trained in parallel on CPU over the full 10.3B-token corpus holds (lossy) 6-, 10-, and 20-gram indexes; at each training step the longest earlier exact-match in the index is retrieved and the tokens that followed are provided as hints to the main LM [tweet body + external PR context].
Tatsu names a surprise finding: “big gains in nanoGPT have come from AI assistance of humans instead of autoresearch” — attributing the record to Herman’s optimization skill under AI assistance, not to autonomous agent search [tweet body].

Method

Tatsu’s post is a quote-stamp with no new experiments, so the method under discussion is Herman Brunborg’s: train a lightweight retrieval model on the CPU in parallel with the GPU-side LM, build 6/10/20-gram exact-match indexes over the full training corpus, and inject the retrieved continuation as learned embeddings at input / middle / output of the network. The CPU index is causal and consulted before any training-shard tokens are inserted; the retrieval parameters register with the base optimizer and the retrieval index is built from the training shards (validation targets never inserted). The compounding “AI assistance” claim refers to Herman using LLMs (not an autonomous agent harness) to accelerate hypothesis generation and implementation while a human directs search.

Results

NanoGPT speedrun: 39.9s → 6.7s on 8×H100, −83.2% [tweet body].
The 2025–2026 cumulative improvement before Herman’s record was −81.4%; Herman’s single contribution exceeds all prior records combined [tweet body].
Qualitative claim with no numbers: AI-assistance-of-humans is dominating autoresearch on this leaderboard [tweet body].

Why it’s interesting

This is the sharpest adversarial datapoint yet for AI-for-AI Research: on the exact same benchmark where Measuring Autonomous AI Research ran 153 autonomous-agent runs across 18 frontier models with no autonomous run producing a fundamentally new method (best was Fable 5 closing 81.7% of the baseline-to-record gap), a human-with-AI-assistance just dropped the record by more than every autonomous and human improvement combined — using a method (retrieval-augmented n-gram memory) that no autoresearch run in the Prime Intellect sweep proposed. It’s also a concrete instance of the Parametric memory vs retrieval trade-off at pretraining scale: a CPU-side lossy n-gram index effectively buys several seconds of GPU training by letting the main LM skip memorizing rare exact repeats, resonating with Qwen3.8-Flash-Next — Hybrid Attention with QSA, Gated Residual, N-gram Embedding‘s n-gram-embedding table and with the broader observation in Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization that fine-grained schedule/data tricks dominate exotic optimizer swaps in the modded-NanoGPT regime.