Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling

TL;DR

CADD (Apple) augments masked discrete diffusion with a paired continuous-latent diffusion so that masked positions carry a noisy-but-informative embedding instead of a featureless [MASK] “information void”. The discrete path stays a standard absorbing chain; the continuous path is a Gaussian diffusion in the token-embedding space, triggered the moment a token is masked. A single network conditions on both and predicts the original token. Architecture is identical to a standard MDM (same parameter count, same backbone) and the loss is a plain cross-entropy on token logits, so any existing MDM checkpoint can be fine-tuned into CADD. The continuous hint also lets sampling trade off mode-seeking (single-sample expectation) vs mode-coverage (multi-sample average) at inference.

Key claims

Masked discrete diffusion’s absorbing [MASK] state collapses all unobserved possibilities into one symbol and erases graded semantic proximity to the true token, while continuous-relaxation methods (CDCD / Plaid) suffer “rounding error” because they denoise only in embedding space and discretize at the end [§1, Fig. 1].
The CADD joint forward factorizes cleanly: at step t, the discrete component follows the standard absorbing chain and acts as a trigger for the continuous component — the embedding stays constant while its token is unmasked, then begins a Gaussian diffusion the moment the token is masked [§4.1, Eq. 4.3-4.4].
The joint marginal q(x_t, h_t | x_0) factorizes into a discrete part q(x_t | x_0) and a continuous part q(h_t | x_t, x_0) in closed form, both tractable [Prop. 1, Eq. 4.5-4.7].
The KL on unmasked positions is identically zero; the KL on masked positions splits into a token-prediction term and a reweighted-MSE continuous term, with a closed-form mixing ratio that depends on the SNR schedules of the two paths [Lemma 1, Eq. 4.14-4.16].
Empirically, dropping the continuous-MSE auxiliary loss and training with only the token cross-entropy is sufficient and is what the authors actually run for most experiments [§4.3, Eq. 4.17].
At inference, the choice of h_t estimator gives a controlled trade-off: a hard single-sample estimator favors mode-seeking (precise local prediction), while averaging multiple draws of h_t per discrete state favors mode-coverage (diversity) [§4.3, Eq. 4.18; App. C.3].
On OWT unconditional text generation (DiT backbone, ~168M params, same training budget as MDLM), CADD’s MAUVE and generative perplexity scale better with sampling steps than SEDD-Absorb and MDLM, with the gap widening to 4096 NFE; MDLM’s quality stagnates or degrades past 4096 NFE while CADD continues to improve, with generative PPL falling from 44.6 to 35.3 between 1024 and 16384 NFE [§5.1, Fig. 4].
On CIFAR-10 unconditional image generation, CADD reaches FID 2.88 / IS 10.04 at 512 NFE, beating MDM (4.66), MDM-Mixture (4.80), MDM-Prime (3.26), D3PM-Absorb (30.97), and continuous baselines including Continuous FM (6.35), Bit Diffusion (3.48), StyleGAN+ADA (3.26), and DDPM (3.17) [Table 2 left].
On ImageNet-32 unconditional generation, CADD reaches FID 3.74 at 1024 NFE, beating all reported discrete baselines (MDM 7.91, MDM-Mixture 8.08, MDM-Prime 6.98) and continuous baselines (DDPM 16.18, ScoreFlow-VP 8.34, Continuous FM 5.02) [Table 2 right].
On code generation (Qwen2.5-Coder backbone, same 65B-token budget as DiffuCoder), CADD trained from scratch reaches HumanEval pass@1 72.0 / MBPP 75.7 / BigCodeBench-Hard 17.6 (avg 55.7), beating DiffuCoder (67.1 / 74.2 / 12.8, avg 52.6) and matching/beating Qwen2.5-Coder AR (avg 52.2) and OpenCoder AR (avg 55.0). Initializing from a DiffuCoder checkpoint and continuing with the CADD loss pushes HumanEval to 73.8 [Table 3].
Trainable parameter count is identical to the MDM baseline; per-step training compute is comparable (only the forward + element-wise fusion of e_t and h_t differ), and inference time for 5000 samples on 8× H100 matches MDLM (0.5h) at K=1 [§5.1 Computation].

Method

CADD pairs each discrete token with a continuous embedding h_t of the same dimension as the model’s token-embedding table. The forward joint transition factorizes: the discrete part is the usual absorbing-state Markov chain (q(x_t | x_{t-1}) with mask-probability schedule α^disc_t), and the continuous part is a Gaussian diffusion q(h_t | h_{t-1}, x_t, x_{t-1}) that only starts once the corresponding token becomes masked. Concretely, while the token is unmasked, h_t holds its clean embedding e_{x_0} exactly; the first time the token is masked, the continuous trajectory begins and h_t then follows a standard Gaussian forward marginal N(α_t · e_{x_0}, σ_t^2 I). As t → T, all tokens are masked and h_T → N(0, I).

The denoising network takes both the partially-masked sequence and its paired continuous latents: clean embeddings e_{x_t} are read out of the token table for unmasked positions, replaced by noisy h_t at masked positions (with corruption drawn from q(h_t | x_t, x_0)), then summed element-wise into the per-position input. The backbone is unchanged from a standard masked DiT / Discrete DiT. Training minimizes a single cross-entropy over predicted token logits at masked positions; the paper notes the auxiliary continuous-MSE term can be added for a tighter ELBO but is empirically not needed.

At inference, sampling starts from fully-masked x_T and standard-normal h_T. At each step, the network produces categorical logits conditioned on (x_t, h_t). For each masked position, with probability proportional to the decreasing mask schedule the model samples a clean token (and freezes its h_t to the clean embedding); otherwise the token stays masked and h_t advances one step along its Gaussian diffusion trajectory. The “estimator choice” for h_t — hard single sample vs averaged multi-sample expectation — sets the mode-seeking vs mode-coverage knob.

Results

Text (OWT, 168M DiT, MDLM hparams): CADD strictly Pareto-dominates SEDD-Absorb and MDLM in (MAUVE, GenPPL, NFE) across NFE ∈ {128, 256, 512, 1024, 4096}, with the gap widening at higher NFE; CADD’s GenPPL drops monotonically from ~50 at 256 NFE to 35.3 at 16384 NFE, while MDLM saturates around 40 and slightly worsens past 4096 [Fig. 4].
CIFAR-10 (ADM-style backbone): FID 2.88 / IS 10.04 at 512 NFE — best result among all listed discrete and continuous baselines including StyleGAN+ADA, DDPM, and MDM-Prime [Table 2 left].
ImageNet-32: FID 3.74 at 1024 NFE, vs MDM-Prime 6.98 and Continuous FM 5.02 [Table 2 right].
Code (Qwen2.5-Coder backbone, 65B tokens): HumanEval / MBPP / BigCodeBench-Hard pass@1 = 72.0 / 75.7 / 17.6 (avg 55.7) beats DiffuCoder (67.1 / 74.2 / 12.8, avg 52.6) and matches OpenCoder AR (avg 55.0); fine-tuning a DiffuCoder checkpoint with the CADD loss further lifts HumanEval to 73.8 [Table 3].
Compute: same trainable parameter count as MDLM (168M); 5000-sample inference on 8× H100 is 0.5h for both at K=1; multi-sample K>1 scales linearly [§5.1].

Why it’s interesting

CADD is the cleanest of the three “continuous + discrete” diffusion-LM hybrids in the Diffusion Language Models cluster: it is a strict superset of standard MDM (same parameters, same loss type, same backbone, same training budget) yet recovers a meaningful slice of the continuous-diffusion advantages — graded semantic hints at masked positions, NFE scaling that does not stagnate, and a mode-seeking↔mode-coverage knob via the h_t estimator. This is closely complementary to The Diffusion Duality (Duo), which goes the opposite direction — proves the equivalence between continuous and uniform-state discrete diffusion and exploits it to port consistency distillation into the discrete space; CADD instead keeps the two paths separate and pairs them so the discrete model gets a continuous “soft hint” channel. It also sits next to Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasoner (CCDD), the third paper in @sedielem’s trio, which makes a stronger expressivity claim — continuous diffusion is provably more expressive than discrete and looped-transformer reasoning — and proposes a fully-joint multimodal denoising rather than CADD’s discrete-triggered Gaussian. The trend Sander Dieleman flagged (“why not both?”) is well-supported: CADD is the most production-friendly point on this design space, since any existing MDM checkpoint (DiffuCoder, MDLM, Dream, LLaDA) can be CADD-fine-tuned without re-architecting. The CIFAR-10 result also suggests this isn’t just a text trick — discrete-token image diffusion can beat continuous DDPM/StyleGAN+ADA once the masked-state semantic gap is filled.