Lucas Beyer — curb the enthusiasm on zeroth-order LM training; loss 4.2 is near the unigram floor

TL;DR

Lucas Beyer posts a skeptical read on the Noise / Dust / spicylemonade zeroth-order pretraining wave the wiki has been tracking. Two concrete objections: (1) genetic-algo and forward-diff methods are known to work at small parameter counts and fall off hard as the “intrinsic” parameter space grows — which is the entire motivation for backprop (few outputs, many params); and (2) validation loss 4.2 for a 30M-param LM is very close to the unigram-distribution floor, so “matching backprop at loss 4.2” does not demonstrate that the method does any interesting learning yet. Beyer frames this as the “old-pessimist voice in the room” counterweight to social-media hype, and offers to publicly admit he was wrong if the scaling story pans out.

Key claims

Zeroth-order genetic-algo and forward-differentiation methods are established to work at small parameter counts but fall off hard as parameter space grows — and what matters is “intrinsic” dimensionality, not raw param count [tweet body (1)].
The whole point of backpropagation is the asymmetry few outputs, many params — which is precisely the regime where forward-diff and genetic estimators lose signal [tweet body (1)].
Validation cross-entropy 4.2 at ~30M parameters is “very high” — a bit higher and you are at the token-unigram floor, so matching backprop at that loss does not require any difficult/interesting learning to have happened yet [tweet body (2)].
Rediscovering methods from first principles is the best way to learn; the pessimism is about communication — social media amplification + VC + journalist attention warrants an explicit old-pessimist voice in the room [tweet body closing paragraph].
Author’s self-stated disposition: “if i turn out to be wrong I’ll happily admit it and cheer on” [tweet body closing].

Method

Opinion tweet; no methodology. The quote-tweeted upstream is spicylemonade’s announcement that community zeroth-order methods on the Noise benchmark have cut Track 2 wall-clock by ~4000× in ten days and pushed Track 1 past backprop + Adam (to 3.676). Replies to Beyer include a correction from spicylemonade that Track 1 currently sits at 3.409 vs 3.419 for backprop + Adam at 30M tokens (not 10M), and a note that Dust separately found bigger models were more population-efficient up to 243M params. A reply from @ar0cket1 argues that at K3 scale (14× wider, 74,000× more params) one only needs ~14× more perturbations because what is being perturbed is activation width, not raw param count — a direct counter to Beyer’s “intrinsic size” objection.

Results

A take, not a result. The quantitative anchors Beyer uses:

~30M params as the current benchmark scale.
loss 4.2 as the “very high, near unigram” reference.
No lower bound, no upper bound on when he expects the fall-off to bite — he does not name a parameter count or token budget.

Why it’s interesting

This is the first filed outside-the-cluster skeptical read on the pretraining-without-backprop line that the wiki has been treating seriously (Noise: Speedrun for pretraining without backpropagation, Dust: Pretraining Transformers Without Backpropagation, spicylemonade — 2.3× pretraining speedup without backprop, inspired by Cammarata's 'Neural Networks Are Already Clean'). It is worth filing precisely because the Noise benchmark page’s “Why it’s interesting” and the Dust page both frame zeroth-order training as a potential displacement of backprop as the structural primitive — and Beyer’s two objections are concrete and testable rather than tribal.

Objection (1) is the one the wiki has not yet pressure-tested: Dust reports bigger models are more population-efficient up to 243M params (120× sweep) and reads this as inverting the textbook genetic-algo scaling story — but Beyer’s “intrinsic size matters, not raw N” is a sharper version of the same complaint and is not resolved by the 243M number (the activations being perturbed may themselves have small intrinsic dimension in a 243M overparametrized model). Objection (2) directly challenges the “matches / beats backprop + Adam” headline from both the Noise update log and spicylemonade’s 2.3× claim — if loss 4.2 is near unigram, then the real test is at much lower losses where frontier LLMs operate, and the whole leaderboard is currently in the regime Beyer calls uninteresting.

In the training-stability-at-scale frame on this wiki, Beyer’s take is the strongest filed “credit assignment is still the right stability locus, not a soft one” argument against the Noise/Dust lineage. It belongs on the record alongside the enthusiasm.