Diffusion Beats Autoregressive in Data-Constrained Settings

TL;DR

A systematic 200-model scaling-law study (100 masked-diffusion LMs + 100 AR LMs, 7M-2.5B params, up to 800 epochs) of pretraining under data scarcity. Headline: masked diffusion language models extract value from repeated data up to ~100 epochs with little degradation, whereas AR models start overfitting after ~4 epochs and diverge under heavy repetition. The authors fit data-constrained scaling laws in the Muennighoff style, derive a closed-form critical-compute frontier C_crit(U) ∝ U^2.174 past which diffusion overtakes AR, and attribute the gap to implicit data augmentation from masked diffusion’s randomized token-ordering objective. Important caveat for the wiki: the result is empirically pinned to discrete masked diffusion over language tokens; extrapolation to continuous image/video diffusion is not tested in the paper.

Key claims

Masked diffusion LMs trained on repeated data behave roughly as if every repetition is fresh data up to ~100 epochs, while AR LMs hit that “repeated ≈ fresh” threshold at only ~4 epochs [project page Fig. 5; §3].
A data-constrained scaling-law fit yields a much higher half-life-of-reuse R*_D for diffusion (~500 epochs) than for AR (~15 epochs), quantifying diffusion’s slower decay-of-value under repetition [§3.2].
The AR-vs-diffusion crossover is governed by a closed-form critical compute C_crit(U) = 2.12 × 10^1.956 · U^2.174 in unique-token budget U; below threshold AR wins at fixed compute, above threshold diffusion wins [§3.3].
200 models trained from 7M to 2.5B parameters across varying unique-data sizes and up to 800 epochs (≤80B tokens total), using Muennighoff et al.’s hyperparameter recipe — which the authors flag as plausibly AR-favoring [§2, Appendix 8-9].
Compute-optimal allocation for fixed 100M unique tokens: diffusion’s best validation loss is reached at ~500 epochs, AR’s at ~50 epochs, with AR overfitting past that and diffusion showing no overfitting through the sweep [Fig. 2].
Downstream benchmark gains track the validation-loss gap: diffusion beats both standard and flop-matched-overfitted AR on 8/9 zero-shot reasoning/QA benchmarks at the compute-optimal selection [Results table].
Mechanism claim: adding explicit token-permutation data augmentation to AR training recovers part of diffusion’s advantage, supporting “exposure to diverse orderings as implicit augmentation” as the load-bearing explanation [§ “Why diffusion wins”].

Method

The setup is a Muennighoff-style data-constrained pretraining sweep. Each of two model families (decoder-only AR with causal attention; absorbing-state masked diffusion LM with bidirectional attention) is trained at a grid of (parameter count, unique-token budget, epoch count) settings, holding the optimizer/HP recipe fixed. The authors then fit closed-form scaling laws to validation loss as a function of (N, D, R) where R is the repetition count, extending Muennighoff et al.’s framework to fit a per-objective decay-of-value parameter R*_D. From the fitted surfaces they derive (a) the compute-optimal (N, epochs) allocation for any unique-token budget, and (b) the closed-form C_crit(U) crossover where the diffusion surface intersects the AR surface.

The mechanistic experiment is an ablation: train AR with explicit input-token permutations as data augmentation, and observe how much of the diffusion gap closes. This isolates token-ordering diversity as the variable of interest, vs. other differences between the objectives.

Results

Repetition tolerance: AR’s curves diverge upward past ~4 epochs; diffusion’s curves stay overlapping through 100 epochs and visibly through ~500 epochs of the sweep [Fig. 4].
Compute-optimal endpoint at 100M unique tokens: AR single-epoch ≈10.65 loss, AR best multi-epoch ≈3.71; diffusion single-epoch ≈much worse than AR, diffusion best multi-epoch ≈3.55. That’s a 67% loss reduction for diffusion vs. 48% for AR through training [§3.1, Fig. 2].
Downstream: diffusion wins on ARC-Easy (37.84 vs 35.63 / 35.35), COPA (59.00 vs 56.33 / 54.00), HellaSwag (30.24 vs 27.37 / 29.03), RACE (28.96 vs 25.28 / 24.88), WinoGrande XL (50.97 vs 48.87 / 49.41), SciQ (68.67 vs 58.05 / 50.50), and Lambada (15.19 vs 10.91 / 5.53), losing narrowly on PiQA and tying near AR-best on BoolQ [Results table].
Crossover frontier: C_crit(U) ∝ U^2.174 — a steep power-law dependence on unique-token budget, meaning the diffusion regime is unlocked relatively early once unique data is the bottleneck [§3.3, Fig. 6].

Why it’s interesting

Sharpens the Diffusion Language Models open question on scaling laws: this paper is the cleanest data-constrained scaling result for masked diffusion LMs the wiki has seen, complementing the inference-side findings already on the concept page (Mercury: Ultra-Fast Language Models Based on Diffusion, Introspective Diffusion Language Models, WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference). The 100-epoch repetition tolerance also gives a concrete mechanism for the “masked objective = implicit augmentation” intuition that The Design Space of Tri-Modal Masked Diffusion Models only assumes. Worth flagging explicitly: the result is for discrete-token masked diffusion over language; Terrance’s framing in the Slack note generalizes to “diffusion models” broadly — including image/video — but the paper doesn’t test continuous-state diffusion, and the “implicit augmentation via token-ordering” mechanism does not obviously port to fixed-resolution image latents. The wiki’s continuous-diffusion training-efficiency thread (Diffusion training efficiency) should treat this as suggestive, not established.