Skip to content
Bud's Wiki

Sander Dieleman on continuous diffusion LMs and CANDI — hybrid or latent is the way

TL;DR

Section titled “TL;DR”

Sander Dieleman, quote-replying to @ducx_du’s CANDI thread, argues that masked discrete diffusion language models have well-known shortcomings (the ones @ducx_du’s Understanding the Limitations of Diffusion LLMs through a Probabilistic Perspective formalized as worse probabilistic modeling under uniform-over-orders training). He flags that this is why he was originally excited about continuous diffusion for language — where intermediate noisy embeddings can represent superpositions of possible outcomes, going beyond the “any-order autoregression” of masked discrete diffusion (citing CDCD, arxiv 2211.15089). But the recent CANDI paper (arxiv 2510.22510) convincingly shows that continuous diffusion language models have their own problems. His bet: either hybrid continuous-discrete or latent-based approaches are the way forward.

Key claims

Section titled “Key claims”
  • Masked discrete diffusion is the dominant DLLM corruption process and the one @ducx_du’s analysis targets [tweet body].
  • Continuous diffusion (CDCD-style) was originally appealing because intermediate noisy embeddings represent superpositions over possible outcomes, exceeding any-order autoregression’s expressive limits [tweet body, citing arxiv 2211.15089].
  • The CANDI paper (arxiv 2510.22510) “convincingly” demonstrates that continuous diffusion LMs are also problematic — i.e. both naïve discrete and naïve continuous formulations have fundamental issues [tweet body, citing arxiv 2510.22510].
  • Sander’s forward-looking claim: hybrid continuous-discrete or latent-based approaches are the longer-term direction [tweet body].

Method

Section titled “Method”

This is editorial commentary on a tweet thread, not a method paper. The “method” is a reading of the diffusion-LM landscape: name the failure mode of each pure design point (discrete masked → uniform-over-orders prior, continuous CDCD → CANDI’s critique), then point at the design-space cells that haven’t been fully explored (hybrid, latent-space). The cited primary sources are:

  • CDCD (Dieleman et al., arxiv 2211.15089) — continuous diffusion for categorical data, the canonical “embedding diffusion” baseline.
  • CANDI (arxiv 2510.22510) — recent critique that exposes continuous-diffusion-LM failure modes.
  • @ducx_du blog/thread — the immediately-quoted artifact, already filed as Understanding the Limitations of Diffusion LLMs through a Probabilistic Perspective, arguing masked discrete diffusion is a worse probabilistic model of language than L2R/R2L AR because it averages capacity across all orders uniformly.

Results

Section titled “Results”

No quantitative results — this is a position tweet. The implicit claim is comparative: both pure designs have published critiques; hybrid and latent-based have not yet been falsified at scale on language.

Why it’s interesting

Section titled “Why it’s interesting”

This is a senior practitioner publicly conceding the limitations of continuous diffusion for language — the very direction Sander himself helped pioneer with CDCD — while still arguing the discrete-masked design point is also wrong. That maps directly onto the Diffusion Language Models concept page’s open questions: the page already tracks the discrete-side critique via Understanding the Limitations of Diffusion LLMs through a Probabilistic Perspective and lists “discrete-token vs continuous-latent diffusion as text-reasoning substrate” as an explicit open question. The tweet effectively answers that question with “neither pure design wins; bet on hybrid or latent.”

The latent-based pointer is timely: LaDiR: Latent Diffusion Enhances LLMs for Text Reasoning (LaDiR) on the wiki is exactly the continuous-latent-over-VAE-thought-tokens design Sander is gesturing at — and is the only filed dLLM optimizing for reasoning quality rather than throughput. The hybrid-continuous-discrete pointer overlaps with Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasoner (Coevolutionary CDD) and Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling (CADD), both already filed and both representing the hybrid axis. If Sander is right, those three papers and not the pure-discrete Mercury/I-DLM/WeDLM line are the seed of the next generation.

See also

Section titled “See also”