spicylemonade — 2.3× pretraining speedup without backprop, inspired by Cammarata's 'Neural Networks Are Already Clean'

ListenIgnore the Dead Neurons: Interpretability as a Speed Lever8:49

Show notes

A researcher going by spicylemonade reports a 2.3× pretraining speedup on a backprop-free recipe — by skipping gradient estimation for neurons that interpretability says are dead or redundant. We unpack the claim, its lineage from Cammarata's "Neural Networks Are Already Clean" and Q Labs' Dust, and why "cashing in a structural prior as compute savings" is the interesting part. Heavy caveat: it's one tweet with a short training curve, no code, no eval — so we read it as a provocation, not a result. **Sources** - spicylemonade — 2.3× pretraining speedup without backprop, inspired by Cammarata's 'Neural Networks Are Already Clean'

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

spicylemonade (research at Vals AI, ex-YC) reports a 2.3× speedup on pretraining-without-backprop by porting ideas from Nick Cammarata’s interpretability essay Neural Networks Are Already Clean into Q Labs’ Dust zeroth-order pretraining recipe. The author’s followup (“blackpill: just ignore the useless neurons”) suggests the speedup comes from skipping gradient estimation for neurons the interpretability story says are near-dead or redundant. Work is still in progress — the tweet is an announcement with a short training-curve video, no technical writeup or code yet. Posted in response to Samip’s (@industriaalist, Q Labs cofounder) earlier announcement of “Noise”, a follow-on to Dust that itself claims a 1000× improvement.

Key claims

A 2.3× pretraining-time speedup over Dust’s baseline zeroth-order recipe, demonstrated on a short training-curve video [tweet body].
The speedup is motivated by Cammarata’s claim that modern language models are ~90% monosemantic single-pole neurons under agentic interpretability tools — i.e. most neurons have one main job and a non-trivial fraction are dead or near-dead [tweet body, follow-up reply “blackpill: just ignore the useless neurons”].
Dust-style activation-space node perturbation is the base method being accelerated — the author’s phrasing (“speedups for pretraining without backpropagation”) directly inherits Dust’s framing [tweet body + quoted Q Labs context].

Method

Not disclosed. The tweet shows one training-curve video (~7s) and promises more. The follow-up reply (“blackpill: just ignore the useless neurons”) is the only mechanistic hint, suggesting the speedup preferentially skips perturbation / credit assignment for neurons that Cammarata’s interpretability framework identifies as unused or redundant. Dust in its base form perturbs activations at every token in every layer type; selectively masking perturbation to the ~90% monosemantic poles — or more aggressively to the ~2–4% not-yet-mapped ones — would reduce per-step cost without reducing the useful part of the population estimate, which matches the “2.3×” speedup magnitude.

Results

A single training-curve video. No numerical table, no eval set, no model scale disclosed in the tweet. The author posts this as “inspired by… turned a couple of its ideas into speedups… still in the works” — i.e. an in-progress announcement, not a finished result.

Why it’s interesting

This is the first filed example of interpretability directly feeding pretraining efficiency on the research team’s radar, and it closes a loop the two source documents left open. Neural Networks Are Already Clean argued that ~90% of neuron poles have one main job and the remaining polysemanticity is recursive generalization rather than knotted superposition — but Cammarata framed this as a diagnostic claim about model internals, not a training lever. Dust: Pretraining Transformers Without Backpropagation in turn established that activation-space node perturbation can match backprop on 20M-token pretraining and is 10³–10⁴× more population-efficient than EGGROLL — but Dust treats every neuron equally when drawing jitter. If spicylemonade’s claim holds, the speedup is the first evidence that interpretability-derived structural priors (monosemanticity → sparsity of useful credit assignment) can be cashed in as compute savings in a non-backprop training regime, which is the sort of lever that is impossible or very awkward under chain-rule backprop. The follow-up by @MrCatid (“Honestly this line of research is probably the most disruptive to the industry right now of everything people are toying with at home”) reads as community corroboration that the Dust-lineage of zeroth-order pretraining is being watched as a potential frontier displacement.

Caveat: this is a tweet with one training curve, no code, no eval details. The claim deserves a full writeup before being treated as a real result.