Herman Brunborg — 83% NanoGPT speedrun improvement via retrieval-augmented n-gram hints

TL;DR

Herman Brunborg claims a new modded-NanoGPT speedrun record: 39.9s → 6.7s (−83.2%), reportedly larger than every record since 2025 combined (−81.4%). The trick is retrieval-augmented n-gram hints — a corpus mapping each training n-gram to the tokens that follow it is built during the timed run on otherwise-idle CPUs and fed to the model as hint tokens during training. Scaling the hint corpus reportedly shows accelerating returns (each doubling cuts loss more than the last), motivating the extreme CPU-kernel engineering: a branch-free SIMD builder that fits in L2 cache, 500× faster than Python and 11× faster than the best auto-researched Rust code.

Key claims

Modded-NanoGPT speedrun improved from 39.9s to 6.7s, −83.2%, “more than every record since 2025 combined (−81.4%)” [tweet body].
Mechanism: build a hint corpus mapping each training n-gram to the tokens that follow it; the model receives these tokens as hints during the forward pass and learns to use them [tweet body].
Scaling the hint corpus has accelerating returns on a fixed model checkpoint — each doubling cuts loss more than the last; because the corpus is built during the timed run, a faster builder means a larger corpus [tweet body].
The hint corpus is built on CPUs in parallel with model training on GPUs; builder is branch-free SIMD code that fits in L2 cache with a custom data structure [tweet body].
CPU builder is reported 500× faster than Python and 11× faster than the best auto-researched Rust code [tweet body].

Method

The published tweet thread does not include code or a writeup, so only the mechanism and the engineering shape are known. During pretraining, for each training n-gram the builder records the tokens observed to follow it elsewhere in the training corpus. These are concatenated into the input stream as “hints” that accompany the normal next-token-prediction loss, letting the model attend to retrieved continuations rather than having to memorize rare, repeated web content before it ever sees it. The builder exploits the fact that NanoGPT’s speedrun on 8×H100 leaves the host CPUs idle, overlapping corpus construction with GPU training. The custom data structure sits in L2 cache (reportedly) and is written as branch-free SIMD so the n-gram lookups land in a predictable pipeline; the reported comparisons are 500× vs a Python baseline and 11× vs the best Rust implementation produced by auto-researched kernel search. No results table, loss curves, hint-corpus-size sweep, or ablation separating “retrieval helps” from “larger builder helps” are posted in-thread.

Results

Headline: 6.7s to target validation loss on the modded-NanoGPT speedrun, down from the previous 39.9s record [tweet body].
Claimed to exceed the sum of all speedrun improvements since 2025 (−81.4%) [tweet body].
No training curves, hint-corpus-size sweep, or compute-vs-quality Pareto plot published in-thread; the result is a benchmark-leaderboard claim, not a paper.
Reply from @JonasSchouwer: “Knowing Herman I bet he could have improved it way more, but just stopped at 6.7 for the memes” — flags that the headline number may not be the ceiling [reply body].

Why it’s interesting

Three threads it lands in. First, it is a strong-but-unverified datapoint for the “retrieval as a training-time crutch” position opposite the in-weights n-gram memory lineage: Qwen3.8-Flash-Next — Hybrid Attention with QSA, Gated Residual, N-gram Embedding Qwen3.8-Flash-Next’s 51B N-gram Embedding table and Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling Over-Tokenized Transformer both bake rare-n-gram knowledge into parameters as an orthogonal scaling axis to MoE; this tweet instead feeds the same n-gram continuations in as retrievable hints at the input so the model does not need to burn parameters memorizing them — the retrieval-side counterpart to the parametric-memory thread tracked under Parametric memory. Second, the CPU-side builder engineering (branch-free SIMD, L2-cache-resident data structure, 500× Python / 11× Rust) is the host-side analog of the GPU-kernel IO-aware-design template on IO-Aware Kernel Design: identify a bottleneck no one was treating as kernel work (here, the corpus-build loop idling on CPUs during the timed run), restructure it as a cache-aware streaming kernel, and win the benchmark. Third, as a speedrun-leaderboard result it complements MuonH (hyperball) sets new modded-NanoGPT speedrun record at 3325 steps — Keller Jordan announcing @wen_kaiyue's result (MuonH hyperball) and the broader “every record since 2025” context: optimizer-side gains (MuonH) and data-side gains (this) are now stacking on the same benchmark, suggesting the leaderboard is not yet near the compute-optimal frontier. Unverified at filing time — no writeup, no code, no ablations — but the magnitude of the claim is large enough to be worth watching for the full release.