The Diffusion Duality

TL;DR

Duo proves that a Uniform-state Discrete Diffusion process (USDM) on a vocabulary of size N emerges from an underlying Gaussian diffusion in R^N via an argmax projection, and uses that duality to port two Gaussian-diffusion staples — curriculum training and consistency distillation — to the discrete setting. The curriculum cuts training variance and doubles convergence speed, letting a 170M USDM beat the autoregressive transformer on 3/7 zero-shot perplexity benchmarks. Discrete Consistency Distillation (DCD) reduces sampling from 1024 → 8 NFE with minimal quality loss, the first discrete-diffusion analogue of Song et al.’s consistency models. For a research team thinking about non-AR text decoders or unified discrete-token stacks, this is the missing theoretical bridge that makes Gaussian-diffusion infrastructure (PF-ODEs, CD, EDM-style training) usable on text.

Key claims

Discretization of a Gaussian diffusion latent via argmax(z_t) produces a Markov chain whose marginals follow exactly a Uniform-state discrete diffusion process; the discrete diffusion parameter α_t is the pushforward of the Gaussian SNR through a closed-form “Diffusion Transformation operator” Ψ involving the standard-normal CDF [§3, Eq. 9-12].
The ELBO for the discrete process is provably tighter than that of the underlying Gaussian process, with equality only at the optimal denoiser — so training in the discrete space is preferable even though the duality is established in continuous space [Theorem 3.1, §3].
A Rao-Blackwellized reformulation of the USDM NELBO avoids materializing one-hot latents and reduces training variance [§3 ELBO, Suppl. A.4].
A curriculum that anneals a tempered-softmax relaxation of argmax from soft toward sharp, while sampling Gaussian diffusion times from a sub-interval, lowers gradient variance by an order of magnitude on the top 100 high-variance weights and roughly doubles training speed vs UDLM [§4.1.2, Fig. 2, Table 3].
After only 510K steps (curriculum + 10K fine-tune as discrete diffusion), Duo matches UDLM trained for the full 1M steps on LM1B PPL — empirically ≥2× convergence speedup [§5.1].
Discrete Consistency Distillation reuses the underlying Gaussian PF-ODE as a deterministic trajectory; the trajectory is then projected to discrete via argmax, sidestepping the absence of a discrete PF-ODE; teacher EMA is updated round-wise and the student is trained by KL between teacher and student denoising distributions [§4.2, Eq. 18-19, Algo. 1].
Duo surpasses prior USDMs on LM1B (PPL 33.7 vs UDLM 36.7, SEDD-Uniform 40.3) and OWT (25.2 vs 27.4 / 29.7), shrinking the gap to absorbing-state MDLM to under 2 PPL points [Table 1].
On zero-shot OWT-trained evaluation, Duo beats SEDD-Absorbing on 4/7 datasets, MDLM on 1/7, and the AR transformer on 3/7 (PTB 89.35 vs 82.05; Lambada 49.78 vs 51.28; AG News 67.81; Pubmed 44.48; Arxiv 40.39) [Table 2].
DCD compresses sampling from 1024 NFE → 8 NFE with minimal quality degradation; in the low-NFE regime Duo outperforms MDM-style absorbing models which lack the implicit (deterministic noise→data) property that consistency distillation requires [§5.2].
USDMs’ self-correcting property (already-predicted tokens can still evolve during reverse sampling) is structurally why they admit consistency distillation while MDMs need predictor-corrector add-ons; this answers the long-standing question of “why bother with uniform-state instead of absorbing-state” [§2.1, §3 Sampler].

Method

Duo’s centerpiece is a constructive bridge: define a Gaussian forward process z_t = α_t · onehot(x_0) + σ_t · ε on R^N, then project to discrete via x_t = argmax(z_t). The induced discrete process has marginals p(x_t|x_0) whose mixing coefficient α^disc_t = Ψ(α_t/σ_t) follows from a closed-form integral against the Gaussian density. Crucially, the discrete marginal evolves according to the linear ODE characterizing a Uniform-state discrete diffusion (Austin et al.’s D3PM Uniform). Training uses the discrete NELBO (tighter than the Gaussian one), reformulated via Rao-Blackwellization to avoid one-hot vectors.

The curriculum reparameterizes the loss to operate on continuous Gaussian latents read into the denoiser via a tempered softmax with temperature τ; τ is annealed from large (smooth, low-variance, biased) to small (sharp, unbiased, high-variance) over training. Empirically τ=10⁻³ matches the unbiased trajectory while cutting variance roughly 10× on the top 100 weights. The model is a 170M-parameter modified DiT (RoPE, AdaLN-time, bert-base-uncased tokenizer on LM1B / GPT-2 on OWT).

Discrete Consistency Distillation cannot define a PF-ODE in discrete space — the discrete denoiser only operates on argmax-ed inputs once the curriculum has cooled. Duo instead reverses the underlying Gaussian PF-ODE with an oracle clean-data denoiser (so the trajectory is the optimal one, not learned), projects each timestep to the discrete grid via argmax, and uses adjacent pairs (x_{t-Δt}, x_t) on this deterministic discrete trajectory to train the student by KL between teacher and student denoising distributions. Distillation proceeds in rounds; teacher weights are updated to the student’s at round boundaries, in the style of progressive consistency distillation.

Results

Training-time benchmarks on LM1B (with sequence packing) and OWT, all numbers are upper-bound PPL on diffusion-model bounds:

Duo (LM1B): 29.9 packed / 33.7 unpacked, vs UDLM 31.3 / 36.7 vs SEDD Uniform 40.3 vs PLAID 89.9 (100K steps, retrained) vs MDLM 27.0 / 31.8 vs SEDD Absorb 32.7 vs AR Transformer 22.3 / 22.8 [Table 1].
Duo (OWT): 25.2 vs UDLM 27.4 vs SEDD Uniform 29.7 vs MDLM 23.2 vs SEDD Absorb 24.1 vs AR 17.5 [Table 1].
Zero-shot OWT generalization (lower better): PTB 89.35, Wikitext 33.57, LM1B 73.86, Lambada 49.78, AG News 67.81, Pubmed 44.48, Arxiv 40.39. Beats SEDD Absorbing on PTB / Lambada / Pubmed / Arxiv, MDLM on Wikitext, and the AR transformer on PTB / Lambada / AG News [Table 2].
Ablation: Rao-Blackwellized ELBO contributes ~1.7 PPL points of the ~3-point UDLM→Duo lift; the curriculum contributes the remaining ~1.3 [Table 5].
Distillation: 1024 → 8 NFE with minimal sample-quality degradation per the paper’s reported regime; below ~16 NFE Duo’s outputs exceed MDM-style baselines that lack an implicit deterministic-trajectory property [§5.2].

Why it’s interesting

This paper is the structural prerequisite for any wiki thread that wants to apply Gaussian-diffusion machinery — particularly consistency distillation — to text. It also clarifies the uniform-state vs absorbing-state question that has hung over discrete diffusion since D3PM: USDMs’ self-correcting forward process is what enables a PF-ODE-like deterministic trajectory (via the underlying Gaussian) and therefore what makes consistency distillation possible, while absorbing-state MDMs structurally cannot. That diagnosis dovetails with The Design Space of Tri-Modal Masked Diffusion Models, which picks masked (absorbing) diffusion for Apple’s tri-modal stack and accepts the slower-sampling tradeoff for the data-efficiency-per-parameter scaling-law advantage; Duo offers the alternative recipe where USDM + DCD recovers few-step sampling. The consistency-distillation port is also a direct counterpart to the wiki’s Diffusion Distillation thread — every distillation paper there (ArcFlow: Unleashing 2-Step Text-to-Image Generation via High-Precision Non-Linear Flow Distillation, Transition Matching Distillation for Fast Video Generation, TDM-R1: Reinforcing Few-Step Diffusion Models with Non-Differentiable Reward, the Causal Forcing family) leans on the existence of a deterministic Gaussian PF-ODE; Duo is what gives a non-trivial discrete model that same lever. Compared to Introspective Diffusion Language Models and WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference — both serving-focused DLM recipes — Duo lives upstream: it’s the training/distillation theory that those serving recipes could be retargeted against, if the team ever wants USDM-flavored rather than MDM-flavored decoders.