Noise: Speedrun for pretraining without backpropagation

TL;DR

Noise is Q Labs’ public pretraining-without-backprop benchmark, with the Dust recipe (Dust: Pretraining Transformers Without Backpropagation) as the opening baseline at validation loss 4.167. Model, data, and scoring are fixed; the only rule is no backprop (no autograd, no hand-written backward). Two tracks: lowest validation loss after one FineWeb epoch, and shortest wall-clock to loss 4.2. The #update section documents the first ten days of community PRs: the speedrun dropped from 1.76 h to 1.6 s (a >4000× cut) and Track 1 fell from 4.167 to 3.676, past backprop + Adam (3.769) and now trailing only backprop + AdamW. Q Labs is already lifting the training-token budget from 10M to 30M to re-expose headroom.

Key claims

The Noise benchmark pins everything except credit assignment to a 37.7M-parameter GPT trained on 10M FineWeb tokens for ≤1 epoch; the only disallowed operations are autograd and hand-written backward passes [Setting §, Track table].
Opening baseline is Dust + AdamW at 4.167 validation loss in 1.76 h, reaching loss 4.2 in 1.76 h — these set the record targets for Tracks 1 and 2 [Track table].
In the benchmark’s first ten days the Track 2 speedrun was cut

4000×, from 1.76 h to 1.6 s; Track 1 fell from 4.167 to 3.676, passing backprop + Adam (3.769) while still trailing backprop + AdamW [Update §].
Record progression credits specific community PRs: ANVIL II + identity-path credit (@Lazarus-931), orthogonal draws + associative credit memory (@JaydenTeoh), Boost → Cut full-Hadamard population- free credit (@spicylemonade), attention-freeze winbacks via own queries / value path / future-token probes / first-hop credit (@hoo0681, @CharlesDDDD, @NotDesigned), end-merge weight averaging for the current Track 1 record (@CharlesDDDD), multi-GPU moment kernels (@RobertTLange), and structured starts + tail averaging + 8-row updates + overlapped comms (@erodola, @Euraba, @areu01or00) [Update § highlights].
Q Labs reads most PRs so far as refinements of the previous entry, tuned to the one setting, and expects scalable gains from optimizer–estimator co-design rather than further speedrun micro- optimization — their response is to raise the token budget from 10M to 30M, with “more exciting changes coming soon” [Update § closing].

Method

Noise is a benchmark harness, not a learning algorithm. The task is pretraining a specific 37.7M-parameter GPT on a 10M-token FineWeb subset for at most one epoch, with validation cross-entropy as the single metric. The only rule distinguishing Noise from standard pretraining speedruns is no backpropagation: autograd is disallowed in both forward and backward mode, and no hand-written backward pass is permitted. Initialization, batch size, optimizer, schedule, forward-pass budget, and all algorithmic choices are open. Submissions are PRs to the qlabs-eng/noise repo; records are read off two tracks (lowest final validation loss on Track 1; shortest wall-clock to loss 4.2 on Track 2). Dust supplies the opening baseline on both tracks (4.167 in 1.76 h; 1.76 h to loss 4.2), inheriting its activation-space node perturbation recipe directly from the earlier Q Labs paper.

Results

Opening baseline (Dust + AdamW): Track 1 4.167, Track 2 1.76 h to loss 4.2 [Track table].
Ten days of community PRs: Track 2 1.76 h → 1.6 s (>4000×); Track 1 4.167 → 3.676, past backprop + Adam (3.769) and still trailing backprop + AdamW [Update §].
Mechanism attribution: @spicylemonade’s full-Hadamard Boost → Cut substitution for Dust’s random draws is called out as “the jump behind both tracks” — one algorithmic change doing the load-bearing work rather than cumulative kernel tuning [Update § highlights].
No numerical Track 1 breakdown vs backprop + AdamW is given in the post; the only claim is “ahead of Adam, trailing AdamW”.
Scaling response: Q Labs is moving both tracks from 10M to 30M training tokens to re-open headroom; no numbers for the new baseline are reported yet [Update § closing].

Why it’s interesting

Noise is the first filed public benchmark the research team has seen whose explicit purpose is to retire backpropagation as the training primitive — the #update section is already evidence that an open leaderboard produces a very different search than the single-group dynamic that led to Dust. The 4000× speedrun compression in ten days is the strongest short-term signal yet that Dust: Pretraining Transformers Without Backpropagation‘s “another 1000× is possible” claim was conservative, and the mechanism Q Labs flags — @spicylemonade’s full-Hadamard Boost → Cut — is a direct continuation of the interpretability-feeds-training lever spicylemonade — 2.3× pretraining speedup without backprop, inspired by Cammarata's 'Neural Networks Are Already Clean' documented but did not technically disclose. This fits the wiki’s “stability lives in a new locus each year” cluster (Training stability at scale) — architecture (QK-Norm, mHC), optimizer (SSO, AdamH), parametrization (µP), precision (NVFP4), and now credit assignment itself — but with one twist nothing else on the page shares: the benchmark is explicitly designed so Q Labs can raise the bar as the community saturates each setting, closer in spirit to modded-NanoGPT’s speedrun ladder (MuonH (hyperball) sets new modded-NanoGPT speedrun record at 3325 steps — Keller Jordan announcing @wen_kaiyue's result) than to a one-shot pretraining paper. The open question Q Labs itself names — whether scalable gains come from refinement of the current recipe or from optimizer-estimator codesign — is the live research question this benchmark exists to answer.