Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasoner

TL;DR

CCDD argues that discrete diffusion LMs and looped transformers underperform their theoretical ceiling because continuous representations are strictly more expressive but harder to train end-to-end into the discrete output space. The fix: define a joint diffusion process over (continuous-representation, discrete-token) pairs and denoise both modalities with a single model that uses the continuous trajectory as an intermediate-supervision channel and the discrete trajectory as a decoder grounding. The paper claims continuous diffusion has provably stronger expressivity than discrete diffusion and looped transformers, while the joint formulation recovers the trainability and sample quality of discrete approaches. Positioned by Sander Dieleman as the third member of a “why not both?” trio with The Diffusion Duality and Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling.

Key claims

Continuous diffusion models are provably more expressive than (i) discrete diffusion models and (ii) looped transformers / continuous chain-of-thought for language modeling — closing a known theory–practice gap, since in practice continuous diffusion LMs typically underperform discrete ones [Abstract, §1].
The expressivity–trainability contradiction is attributed to decoding difficulty: continuous diffusion provides intermediate supervision (the denoising trajectory) that looped transformers lack, but pays for it with the burden of decoding tokens out of the continuous representation space at the end [Abstract, §1].
CCDD defines a joint multimodal diffusion process on the union of a continuous representation space and a discrete token space, with a single model denoising in the joint space — combining rich latent semantics (continuous side) with clean trainability and decoding (discrete side) [Abstract, §1].
The paper proposes effective architectures and training/sampling techniques for the joint process; concrete recipes are detailed in the body (per the abstract; full architectural details not retrievable from the abs-page fetch at filing time) [Abstract].
Empirical results on real-world language-modeling tasks are reported as “strong” relative to discrete-only and continuous-only baselines, framed as evidence that the joint design recovers the trainability the continuous-only approach lacks [Abstract].

Method

The fetched arXiv abstract page describes the framework at a high level: a single denoising model operates on a joint state (z_t, x_t) where z_t lives in a continuous representation space (acting as the latent reasoner) and x_t lives in the standard discrete-token space (anchoring decoding). The two trajectories evolve coevolutionarily — neither is purely a function of the other, and the model exploits the continuous trajectory as intermediate supervision that pure discrete diffusion does not have, while leveraging the discrete trajectory to avoid the rounding/decoding failures that have historically dogged continuous-diffusion LMs (Plaid, CDCD).

Architecturally this is contrasted with CADD (Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling), where the continuous path is a triggered Gaussian diffusion that only starts when the corresponding token is masked, and with Duo (The Diffusion Duality), where there is one underlying Gaussian process and the discrete chain is its argmax projection. CCDD instead carries both modalities in parallel throughout the entire trajectory and lets the model exploit both as conditioning inputs at every denoising step. Detailed architectures, training losses, and sampling algorithms are in the full PDF (not extracted in the fetched abs page).

Results

Extracted from the fetched abs page only:

The paper reports “strong empirical performance in extensive language modeling experiments on real-world tasks” — no specific numbers were retrievable from the abstract-only fetch at filing time [Abstract].
The expressivity claim — continuous diffusion > discrete diffusion ≈ looped transformers — is stated as a proof in the paper but the proof’s regime and assumptions are not extractable from the abs page [Abstract].
Submission history: v1 Oct 3 2025; v2 May 12 2026 (this version), 552KB; iterated on by the authors over ~7 months between submissions [arXiv metadata].

Note: detailed result tables and method figures are in the PDF body and were not part of the fetched content at filing time. A /bud refresh against the full PDF would tighten this section.

Why it’s interesting

CCDD is the most theoretically ambitious of @sedielem’s “continuous + discrete” trio. Where The Diffusion Duality (Duo) establishes a structural equivalence between continuous and uniform-state discrete diffusion and uses it to port distillation techniques, and Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling (CADD) augments masked discrete diffusion with a parallel Gaussian path as a soft hint, CCDD makes a stronger claim: continuous diffusion is strictly more expressive than both standard discrete diffusion and the looped-transformer / continuous-CoT family — which connects this directly to the wiki’s Looped Transformers cluster (HRM-Text, ELT, Hyperloop, Generative Recursive Reasoning) and to the broader question of where intermediate supervision in latent reasoning should come from. If the expressivity proof holds at the regimes the wiki cares about, it would suggest that the dLLM design space ought to be retargeted from “discrete with serving tricks” (WeDLM, I-DLM, NLD’s tri-mode) toward “continuous-latent reasoning + discrete decoding head.” That would also reframe LaDiR: Latent Diffusion Enhances LLMs for Text Reasoning (LaDiR) — which already does continuous-latent diffusion over VAE-compressed thought-token blocks — as a partial instantiation of the CCDD principle. The cluster of three papers @sedielem highlighted is the strongest signal so far that the field is converging on hybrid continuous+discrete diffusion as the next regime for LM training, not just a curiosity.