Understanding the Limitations of Diffusion LLMs through a Probabilistic Perspective

TL;DR

Sea AI Lab argues that viewing diffusion LLMs as any-order language models exposes a fundamental probabilistic-modeling weakness: natural language has strong structural biases that make some token orders (L2R, R2L) much easier to fit than random orders, but masked-diffusion training optimizes all orders uniformly. The result is that DLLMs become looser probabilistic models of language than same-budget AR LMs in either direction. The post frames the analysis around masked diffusion (the dominant DLLM training paradigm) and is positioned as a contrarian read against the prevailing “diffusion will replace AR” narrative.

Key claims

Diffusion LLMs are mathematically equivalent to any-order language models — the per-step training objective predicts a masked subset of tokens from the unmasked remainder, which is the marginal of generating in some permutation of positions [§ “Diffusion LLMs as Any-Order LMs”].
Natural language has strong directional structure: L2R and R2L orders are significantly easier to model than random orders [§ “Insight 1”].
Because masked diffusion optimizes all orders uniformly with equal weight, the model cannot concentrate capacity on the favorable L2R/R2L orders, producing a looser approximation to the underlying language distribution [§ “Insight 2”].
The blog focuses on the masked diffusion family (LLaDA, Dream, the dominant production design point) rather than continuous-state or score-based discrete diffusion [§ “Setup”].
The training-time formulation is analogized to BERT-style masked language modeling — predict a noised subset given the rest — but with the key difference that the masking ratio is drawn from a schedule rather than fixed [§ “Setup”].

Method

The post’s “method” is an analytical lens rather than a new training algorithm. The authors recast a masked-diffusion LM training step (predict the masked subset given the unmasked subset) as an unbiased estimator of the average log-likelihood across all token orderings of the sequence. Under this reformulation, a DLLM’s training loss is an average over the n! generation orders, while an AR LM’s loss is concentrated on a single order (L2R for standard LLMs, R2L for the right-to-left variant). The argument is then that if natural language has heavy directional structure — and the post claims it does, citing the empirical fact that L2R/R2L AR perplexities are significantly lower than random-order perplexities at matched capacity — then the uniform-over-orders objective is a strict regularizer: it forces capacity to be spent fitting hard orders that the model will rarely need at inference. The output is a probabilistic model with worse data fit per parameter than an AR LM specialized to a good order.

Results

The blog summary fetched is truncated; the artifact is a Notion long-form post with companion experiments that the authors gesture at but the body text excerpt does not enumerate quantitatively. The summary headline result — “DLLMs become worse probabilistic models than the L2R / R2L AR LMs in real language” — is asserted as the main empirical finding, with the structural-bias-of-language argument as the proposed mechanism. See the linked Notion for full plots and the order-conditioned perplexity measurements that back the claim.

Why it’s interesting

This is the first artifact on the wiki to argue against diffusion LMs from a fundamental probabilistic-modeling angle rather than an engineering one. The existing Diffusion Language Models thread documents the engineering case for DLLMs — Mercury’s throughput wins (Mercury: Ultra-Fast Language Models Based on Diffusion), the topological-reordering / introspective trick that makes them deployable on AR infra (WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference, Introspective Diffusion Language Models), the data-constrained scaling-law win (Diffusion Beats Autoregressive in Data-Constrained Settings) — and the open question of from-scratch vs continued-pretrain has been increasingly answered in favor of AR-initialization. This post sharpens that puzzle: if DLLMs are intrinsically worse density models because uniform-over-orders is the wrong prior for language, then the empirical reliance on AR continued-pretraining is not incidental — it is the only way to recover the directional bias the diffusion objective threw away. The argument also creates direct tension with Diffusion Beats Autoregressive in Data-Constrained Settings‘s claim that randomized token orderings help via implicit augmentation in the data-constrained regime: both can’t be globally right at the same scale.