Skip to content
Bud's Wiki

Diffusion Language Models

TL;DR

Section titled “TL;DR”

Diffusion language models (dLLMs) replace the autoregressive left-to-right factorization with a parallel coarse-to-fine denoising process over masked or noisy tokens. The wiki tracks them on three production axes: (1) wall-clock throughput — Mercury and I-DLM report multi-hundred-tokens/sec on commodity H100s, beating speed-optimized AR APIs by up to ~10×; (2) deployability under existing AR-serving infra (vLLM, PagedAttention, FlashAttention, KV cache) — addressed by WeDLM’s topological reordering and I-DLM’s strict-causal-attention introspective training; (3) post-training — RL recipes (coupled-GRPO, StableDRL) fix the noisy-importance-ratio failure mode that makes vanilla GRPO collapse on dLLMs. Quality has caught up at the 7-8B scale on math/code/reasoning benchmarks; the open questions are scaling laws, multi-modal extension, and whether a unified AR+diffusion checkpoint (Nemotron-Labs-Diffusion) dominates the specialist designs.

Key claims

Section titled “Key claims”
  • A dLLM parameterized as a standard Transformer denoiser, trained on masked or absorbing-state discrete diffusion, can reach commercial-scale throughput: Mercury Coder Mini measured at 1109 output tokens/sec on H100 vs 59-201 tok/s for the closest AR speed-optimized models, at matched HumanEval/MBPP/EvalPlus quality (Mercury: Ultra-Fast Language Models Based on Diffusion Table 1).
  • Quality parity with same-scale AR teachers is achievable when the dLLM is post-trained from an AR base via continued pretraining: I-DLM-8B (converted from Qwen3-8B) matches Qwen3-8B-Instruct on 14 of 15 reasoning/math/code benchmarks while delivering 2.9-4.1× throughput at high concurrency (Introspective Diffusion Language Models).
  • WeDLM-8B (continued-pretrained from Qwen3-8B with topological reordering) averages 74.72 across 10 reasoning/math/code benchmarks vs Qwen3-8B-Instruct 72.61, LLaDA-8B 55.44, and Dream-7B 56.91 (WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference Table 1).
  • The “tokens per forward” metric is misleading without prefix-cache-friendly serving: WeDLM argues prefix cacheability P_c = N_new / N_total is the right efficiency metric in KV-cached deployment, and many prior dLLM speedup claims do not translate to wall-clock wins against optimized AR engines (WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference §3.1, Eq. 4).
  • Bidirectional attention is not required for masked-token recovery: under topological reordering (WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference) or introspective concatenation [x_t | x_0] (Introspective Diffusion Language Models), strict causal attention suffices and the model becomes a drop-in for SGLang/vLLM with PagedAttention and CUDA Graphs unchanged.
  • Standard GRPO collapses on dLLMs because the importance ratio is intractable and must be estimated via ELBO/mean-field, producing heavy-tailed noisy ratios that drive gradient spikes; the fix is unconditional two-sided clipping + self-normalization by Σ clip(r) (Stabilizing Reinforcement Learning for Diffusion Language Models §3, Theorem 3.5).
  • The complementary RL fix at the estimator level is coupled-GRPO: sample antithetic mask pairs (t, T-t) whose union covers every token, giving each token at least one accurate log-prob estimate (DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation §5, Eq. 4).
  • dLLM decoding has measurable AR bias when adapted from an AR base — DiffuCoder and Dream are more left-to-right than from-scratch dLLMs like LLaDA, quantified by local/global AR-ness@k metrics, and the “entropy sink” at the prefix boundary produces a causal bias under low-confidence remasking (DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation §4.1-4.2, Fig. 3).
  • A single checkpoint can be trained to serve in three modes (AR, parallel diffusion, self-speculation) by switching only the attention mask, with the diffusion mode acting as a draft for the AR verify — Nemotron-Labs-Diffusion reports 5.9× tokens-per-forward over Qwen3-8B at matched accuracy and 3.3× wall-clock on GB200 (Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding).
  • Continued-pretraining from an AR base is the dominant recipe: Mercury’s report does not specify (proprietary), but DiffuCoder (DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation, Qwen2.5-Coder), I-DLM (Introspective Diffusion Language Models, Qwen3-8B/32B), WeDLM (WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference, Qwen2.5-7B/Qwen3-8B), and ZAYA1-Diffusion all start from AR weights, suggesting from-scratch dLLM training is not the practical path.
  • Masked-diffusion LMs tolerate ~100 epochs of data repetition under fitted data-constrained scaling laws; AR LMs hit the “repeated ≈ fresh” wall at ~4 epochs. The fitted half-life-of-reuse is R*_D ≈ 500 epochs for diffusion vs ≈15 for AR, and the AR↔diffusion crossover follows a closed-form critical-compute frontier C_crit(U) ∝ U^2.174 in unique-token budget — past the threshold, diffusion strictly beats AR in validation loss and downstream zero-shot benchmarks. The mechanism, supported by an AR + token-permutation-augmentation ablation, is that masked diffusion’s randomized ordering acts as implicit data augmentation (Diffusion Beats Autoregressive in Data-Constrained Settings §3, Figs. 2-6).
  • Masked discrete diffusion is the natural substrate for training-free posterior sampling on images. APS tilts a pretrained dLLM’s unconditional categorical distribution by the measurement likelihood and decodes via straight-through-quantized expectation in the codebook-embedding span, exposing “anchor tokens” with low prior mass but high posterior mass; anchored remasking then keeps anchor tokens instead of high-prior background. On ImageNet SR(4×), gains over the prior discrete SOTA (G2D2) reach 35.82% LPIPS / 10.94% PSNR; the method matches or beats continuous baselines (DPS, PSLD) on linear and nonlinear inverse problems, and scales to 1024² in 15 steps where PSLD takes ~12 min at 512² (Test-Time Anchoring for Discrete Diffusion Posterior Sampling §3-§4, Theorems 3.1-3.2).
  • Continuous-latent (not discrete-token) diffusion can lift reasoning quality on an existing AR LLM. LaDiR encodes each CoT sentence into a block of k continuous latent thought tokens via a β-VAE (encoder fine-tuned from the LLM, decoder frozen), trains a flow-matching model with hybrid bidirectional-within-block / causal-across-block attention to denoise the latents, and switches to AR text decoding for the final answer. With LLaMA-3.1 8B base on DART-MATH training, it averages 41.8 pass@1 across 7 math benchmarks vs 40.4 for AR CoT SFT and 31.9 for Coconut, and on Countdown improves Pass@1 by >25 points (CD-4) and ~30 points (CD-5) over LLaMA 8B SFT. Denoising-step count is an adaptive test-time-compute knob (+11.7 avg from 5→10 steps, +9.8 total from 10→50). Stage-2 rollout training (closing the train-inference gap) is load-bearing: removing it drops average accuracy from 41.8 to 32.6 (LaDiR: Latent Diffusion Enhances LLMs for Text Reasoning §3-§4, Tables 1-3). This is the first filed dLLM on the wiki optimizing for reasoning quality rather than throughput, and the first to use continuous latents rather than discrete-token diffusion.

Recent contributions

Section titled “Recent contributions”
  • [2026-06-23] DiPOD: Diffusion Policy Optimization without Drifting Apart: DiPOD attacks dLLM-RL instability at the ELBO-surrogate layer (drift between ELBO and true log-likelihood under policy improvement), proposing an on-policy ELBO regularizer / self-distillation step as the fix. Complements StableDRL’s ratio-estimation-layer diagnosis (Stabilizing Reinforcement Learning for Diffusion Language Models); the two could be stacked. Notable for claiming the same recipe stabilizes continuous-control diffusion policies, suggesting dLLM-RL and robotics diffusion-policy-RL share the underlying numerical problem.
  • [2026-06-10] DiffusionGemma — experimental open diffusion language model from Google Gemma: Google Gemma announces DiffusionGemma — first Gemma-program diffusion language model, Apache 2.0, “generates entire blocks of text simultaneously”; tweet-only at filing, no benchmarks or tech detail yet. First major-lab dLLM out of the Google Gemma program, joining Apple (I-DLM), NVIDIA (Nemotron-Labs-Diffusion), Inception (Mercury), Tencent (WeDLM), and Zyphra (ZAYA1-Diffusion-Preview).
  • [2026-05-24] LaDiR: Latent Diffusion Enhances LLMs for Text Reasoning: LaDiR (UCSD + Apple) — continuous-latent diffusion for LLM text reasoning. Distinct from every other filed dLLM in two ways: (i) the diffusion happens in a continuous VAE latent space over compressed reasoning-step blocks, not over the discrete token vocabulary; (ii) the optimization target is reasoning quality (math + Countdown planning), not throughput. The recipe: β-VAE compresses each CoT sentence into k thought tokens (with input-token-substitution + latent-noise augmentation to prevent lexical overfit); flow-matching model with bidirectional-within-block / causal-across-block attention denoises latents one block at a time under a special-token head’s stop decision; two-stage training (teacher-forcing + rollout) closes the exposure-bias gap. Beats AR CoT SFT and Coconut on 7 math benchmarks (41.8 vs 40.4 / 31.9 avg) and pushes Countdown CD-4 Pass@1 from 46.7 (LLaMA 8B SFT) to 76.6, with best-in-class diversity via Stein-style repulsion guidance. Denoising-step count is a continuous test-time-compute knob, connecting cleanly to Inference-Time Scaling.
  • [2026-05-24] Test-Time Anchoring for Discrete Diffusion Posterior Sampling: First dLLM-as-posterior-sampler paper on the page. Uses pretrained masked discrete-diffusion foundation models (MMaDA, Dream, VQ-Diffusion) for image inverse problems via two ideas — quantized expectation (tilt categorical, straight-through to codebook span, quantize back) and anchored remasking (confidence under the tilted posterior, not the unconditional prior). Beats G2D2 / SGDD / SVDD-PM and competitive continuous baselines (DPS, PSLD) on FFHQ + ImageNet SR / deblur / inpainting / HDR; +21.99% relative gain over Dream-7B-Instruct on QA with a Qwen3-0.6B reward enabling joint sampling. Concretely a new application axis: prior dLLM papers on this page chase throughput, serving infra, and RL stability; APS chases training-free guidance — the discrete-diffusion analog of DPS/PSLD/RB-Modulation, by the same Litu Rout author lineage as RB-Modulation on the continuous side.
  • [2026-05-24] Diffusion Beats Autoregressive in Data-Constrained Settings: CMU’s 200-model (100 diffusion + 100 AR, 7M-2.5B params, up to 800 epochs) data-constrained scaling-law study — the first cleanly-fitted answer on this page to “do dLLMs scale differently from AR under repetition.” Reports a closed-form crossover C_crit(U) ∝ U^2.174, R*_D ≈ 500 (vs ≈15 for AR), and a token-permutation ablation that supports “implicit augmentation via randomized token orderings” as the load-bearing mechanism. Pairs naturally with The Design Space of Tri-Modal Masked Diffusion Models‘s scaling-law fits on the architecture side. Re-surfaced via @cloneofsimo’s tweet about training on the top-1% data for 100 epochs.
  • [2026-05-24] Mercury: Ultra-Fast Language Models Based on Diffusion: The upstream commercial-scale dLLM family (Inception Labs); third-party-measured 1109 tok/s for Mercury Coder Mini at matched code quality, #1 on Copilot Arena for latency, OpenAI-API-compatible. Sets the wall-clock target the academic dLLM line has been trying to recover under more deployable constraints.
  • [2026-05-24] Introspective Diffusion Language Models: Introspective consistency as the structural property AR has and prior dLLMs lacked; ISD does strided generate+verify in a single forward via min(1, p/q) acceptance, R-ISD adds gated-LoRA for bit-for-bit lossless acceleration over the base AR. First DLM to match same-scale AR quality across 15 benchmarks.
  • [2026-05-24] Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding: NVIDIA’s tri-mode (AR + diffusion + self-speculation) joint training on a single checkpoint; 5.46→6.82 acceptance-length with LoRA-tuned drafter, beats Eagle3 (2.75) and Qwen3-9B-MTP (4.24).
  • [2026-05-24] DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation: Apple’s masked-diffusion code model; introduces local/global AR-ness metrics + entropy-sink diagnostic, and coupled-GRPO antithetic-variates RL.
  • [2026-05-24] Stabilizing Reinforcement Learning for Diffusion Language Models: Diagnoses GRPO’s noisy-ratio instability on dLLMs and proves the self-reinforcing-spike loop; StableDRL’s unconditional clip + self-normalization is the first method to enable stable >1000-step full-parameter RL on both full-attention and block dLLMs.
  • [2026-05-22] WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference: Tencent’s topological-reordering trick — permute observed tokens to the physical prefix while RoPE preserves logical positions — making a dLLM serve under standard causal attention on vLLM/FlashAttention/PagedAttention infra. ~3× speedup over vLLM-served Qwen3-8B on reasoning, >10× on low-entropy generation.

Open questions

Section titled “Open questions”
  • Discrete-token vs continuous-latent diffusion as text-reasoning substrate. Every other dLLM on this page does discrete masked diffusion over the LLM token vocabulary; LaDiR (LaDiR: Latent Diffusion Enhances LLMs for Text Reasoning) does continuous diffusion over VAE-compressed thought-token blocks. The two are not benchmarked head-to-head (LaDiR’s baselines are AR CoT and Coconut, not LLaDA / Mercury / I-DLM). Open: (a) does a discrete-token dLLM trained at the same scale on DART-MATH match LaDiR’s 41.8 avg, (b) is the VAE latent’s compression (one sentence → k tokens) the load-bearing factor or the continuous-diffusion objective itself, (c) is the gain explainable by VAE-as-regularizer effects that are orthogonal to the diffusion choice?
  • Scaling laws (partial answer in). Diffusion Beats Autoregressive in Data-Constrained Settings now provides a closed-form data-constrained scaling law and the AR↔diffusion C_crit(U) crossover for masked-diffusion LMs at the 7M-2.5B scale. Open: (a) does the law extrapolate past 2.5B, (b) does it hold under AR-base continued-pretraining (where weights start from AR initialization, not random), and (c) does the Apple MDM design-space D* ∝ N^0.476 fit (The Design Space of Tri-Modal Masked Diffusion Models) collapse into the same surface or describe a different regime?
  • From-scratch vs continued-pretrain. Every quality-competitive 7-8B dLLM on the wiki was continued-pretrained from an AR base (LLaDA being the major from-scratch exception, and consistently lagging). Diffusion Beats Autoregressive in Data-Constrained Settings is from-scratch and shows diffusion dominates AR in the data-constrained regime — suggesting from-scratch may be the right path once unique data is the bottleneck, in tension with the current production recipe.
  • Native multi-modality. None of the filed dLLMs handle vision/audio natively at frontier quality; NLD ships a VLM variant but its quality is reported only as ≤0.1% drop vs AR, not against frontier VLMs. Is the masked-diffusion objective a natural unifier for image+text+code (the Unified Multimodal Models thread suggests yes for tokens, but the dLLM line has not yet integrated). APS (Test-Time Anchoring for Discrete Diffusion Posterior Sampling) is a partial datapoint: it uses MMaDA (image + text unified) and Dream-7B (text) interchangeably as priors for posterior sampling, suggesting the categorical-distribution interface generalizes across modalities at inference time even if pretraining quality doesn’t yet match frontier specialists.
  • The “entropy sink” / AR-ness leakage. DiffuCoder shows adapted-from-AR dLLMs retain measurable AR bias; is this a bug (reduces parallel capacity) or a feature (gives the model a useful left-to-right inductive bias)? Coupled-GRPO appears to reduce it post-RL while improving quality, suggesting bug — but no comparable measurement exists for WeDLM/I-DLM/NLD.
  • Tri-mode vs pure-diffusion serving. NLD’s “one checkpoint, choose mode at inference” approach competes with WeDLM/I-DLM’s “always-diffusion under causal attention”. Which dominates at scale when the user-facing knob is latency-per-request vs throughput-per-cluster?
  • Does the 100-epoch repetition tolerance transfer to continuous-state diffusion (image/video)? The Diffusion Beats Autoregressive in Data-Constrained Settings mechanism is specifically randomized token-ordering augmentation; for fixed-resolution image/video latents the analogous augmentation is unclear. The continuous-state side of the wiki (Diffusion training efficiency, Pixel-space diffusion) should treat the result as suggestive of an effect, not established at scale.
  • Inverse-problem-side scaling. APS reports better resolution scaling than continuous samplers (1024² in 15 steps vs PSLD’s ~12 min at 512²); is the right reading “discrete diffusion’s tokenized state is intrinsically cheaper to sample posteriors from at high resolution” (in which case the discrete-vs-continuous race is being fought on the wrong axis) or “MMaDA’s prior happens to be cheap to query and the gain disappears at fixed prior quality”? No filed paper isolates these.

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-07

Section titled “2025-07”

2025-06

Section titled “2025-06”