Dust: Pretraining Transformers Without Backpropagation

ListenDust: Pretraining Without the Backward Pass10:14

Show notes

A deep look at Dust from Q Labs, the first zeroth-order method reported competitive with backprop for pretraining transformer language models. Instead of perturbing weights, Dust jitters activations at every token, turning each token into a virtual population member evaluated in one forward pass. We cover the mechanism, the EGGROLL comparison, the surprising "bigger models are more population-efficient" result, and what a search-based training regime might mean for models that aren't end-to-end differentiable. **Sources** - Dust: Pretraining Transformers Without Backpropagation (Samip Dahal, Bishwas Mandal, Serdar Gülbahar, Akshay Vegesna)

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

Dust (Quest Labs) is the first zeroth-order algorithm reported competitive with backprop for pretraining transformer LMs. Instead of perturbing weights (as in weight-space ES like EGGROLL), it perturbs activations independently at every token, treating each token as a “virtual population” member evaluated in parallel in one forward pass. On GPT-style FineWeb pretraining up to 20M tokens and 243M parameters, Dust matches or exceeds backprop at large populations, is ~10³–10⁴× more population-efficient than EGGROLL, and — contrary to conventional ES wisdom — bigger models are more population-efficient. The authors pitch it as a candidate for a compute-rich search-based regime where architectures no longer need to be end-to-end differentiable.

Key claims

At 100k and 1M FineWeb tokens, Dust ends below backprop test loss starting from a few hundred draws (at 100k) and from a thousand draws (at 1M); at 10M tokens the fitted population limit lands just above backprop and at 20M the ladder is still falling at 16k draws with a power-law fit of 4.431 vs backprop’s 4.633 [§3.2, Table 1, Fig. 2].
EGGROLL (a transformer-adapted implementation of the SOTA weight-space ES method) with 256× Dust’s population still does not reach Dust at 64 draws; extrapolation of the EGGROLL ladder suggests it would need several thousand to ~10⁴× more population to match Dust’s smallest population [§3.2, Fig. 1, App. D].
Dust is tested unchanged under Adam: Dust and backprop both improve, while EGGROLL gains almost nothing from Adam — so Dust’s gradient estimate already composes with optimizers tuned for backprop [§3.3, Fig. 3].
Across a 120× parameter sweep (2M → 7M → 38M → 243M) at fixed 10M tokens, bigger models are more population-efficient at every population from 256 up; the 38M/243M models keep gaining ~30% more from population past 1k draws than the 2M/7M models, which saturate earlier [§4.1, Table in §4.1, Fig. 4].
Cosine between Dust’s gradient estimate and the true backprop gradient rises with population for every layer type, is well-fit by a two-parameter saturation law c(K) = c_max K / (K + c) with RMSE < 0.06, and the ceiling c_max stays flat across two orders of magnitude in training tokens (10M → 1B) — “useful gradients emerge from population alone” with no chain rule built in [§4.2, Fig. 5, App. G].
EGGROLL’s cosine with backprop at the same forward-pass budget is still below 0.05 at 128k forward passes for every layer type except the head — a mechanistic account of why its training barely moves [§4.2].

Method

Dust replaces backprop with activation-space node perturbation. For a linear layer y_t = W x_t the output is jittered at every token, y_t → y_t + σ a_t with a_t ~ N(0, I), a forward pass is run, and each token’s jitter is credited by the centered loss reduction at that token plus a decayed sum of the loss reductions at later tokens (the layer’s effect reaches them through attention). Averaged over K draws, the reward-weighted noise ĝ_t is the estimated output-side error; its outer product with the clean input x_t (summed over tokens) is the weight gradient — the same object backprop constructs, but the output error comes from the population rather than the chain rule.

Because perturbations happen in activation space, one sequence of a few thousand tokens evaluates a few-thousand-member population in a single forward pass — three orders of magnitude more members per forward than weight-space ES, and crucially orthogonal to the batch axis EGGROLL already uses. Interference is reduced by (a) jittering different layer types in separate forward passes (reusing the cached clean forward and only re-running blocks from l onward), (b) scoring attention-internals (Q, K, V, gates, value embeddings) via alignment with the estimated attention-output error rather than loss reductions directly, and (c) re-evaluating only a slab of the vocab for the LM head.

Dust vs backprop vs EGGROLL training curves

Results

Pretraining match-or-exceed: at 1M tokens Dust ends below backprop from ~1k draws; at 20M tokens the fitted population limit (4.431, 95% CI 3.89–4.58) is below backprop (4.633) with the ladder still falling [§3.2].
Population efficiency over EGGROLL: Dust at 64 draws beats EGGROLL at 16k draws at every budget ≥ 1M tokens by 0.4–0.6 test loss; needed-population extrapolation puts EGGROLL at 10³–10⁴× the population to match Dust’s smallest population [§3.2, Fig. 2, App. D].
Adam-compatibility: EGGROLL’s Adam ladder lands within 0.01 of its SGD ladder at every population; Dust’s and backprop’s both improve, Dust’s fitted limit sits below backprop [§3.3].
Overparameterization pays: at fixed 10M tokens and 256 draws, loss drops monotonically from 2M → 7M → 38M params (5.486 → 5.362 → 5.358), 243M only slightly worse (5.419); at 16k draws, 38M reaches 5.036 vs backprop’s 5.015 [§4.1 table].
Backprop-alignment at scale: layer-type mean cosine fits give c_max constant across 10M → 1B token checkpoints; EGGROLL stays below 0.05 cosine at matched compute [§4.2].

Why it’s interesting

Dust fits the research team’s “scaling laws + alternative training primitives” thread but attacks the credit assignment locus rather than architecture, optimizer, or precision. It is the strongest filed challenge to backprop-as-only-option in pretraining and complements Lost in Backpropagation: The LM Head is a Gradient Bottleneck which argues that backprop itself loses 95–99% of logit-gradient norm at the LM head — Lost-in-Backprop asks “is backprop the right chain rule?”, Dust asks “do we need the chain rule at all?”. The “bigger models are more population-efficient” finding is a surprising inversion of the textbook ES scaling story and reads as the ES-side analogue of Scaling Laws for Neural Language Models: there is a compute-overhang regime where the search-space geometry a larger model provides outweighs the dimensionality cost. If this holds, it suggests a class of models currently gated by non-differentiability (looped transformers trained BPTT-free, program-in-the-loop stacks) that this cluster — Looped Transformers, Recursive Language Models: the paradigm of 2026 — has been circling. The three-orders-of-magnitude gap over EGGROLL specifically disarms the common dismissal of zeroth-order methods as “doesn’t scale past CIFAR”.