Compositional Discrete Latent Code for High Fidelity, Productive Diffusion Models

TL;DR

DLC (Discrete Latent Code) conditions a DiT on a sequence of discrete tokens derived from Simplicial Embeddings over frozen DINOv2-L features, rather than continuous SSL embeddings or class labels. The discrete factorization decomposes the image distribution into p(c) (easy to model with a small autoregressive prior) and p(x|c) (tractable conditional diffusion), reaching SOTA unconditional ImageNet-256 FID (1.59) while reducing reliance on CFG. Because DLCs are compositional tokens, two codes can be mixed at inference to generate out-of-distribution images that combine semantics of multiple classes (e.g. “jellyfish + mushroom”). The DLC vocabulary is also pluggable into a pretrained text diffusion LM (LLaDA), giving a text-to-DLC-to-image pipeline by fine-tuning the LM on (caption, DLC) pairs.

Key claims

DLC tokens are produced by passing frozen DINOv2-L features through a randomly-initialized linear projection into Simplicial Embeddings (sequences of distributions over a token vocabulary) trained with a self-supervised distillation objective [§3, §5].
DiT-XL/2 conditioned on DLC512 sets SOTA on unconditional ImageNet-256 generation, with reported FID 1.59 [Table 1, Fig. 1a].
Generative FID improves monotonically with DLC sequence length (L), trading prior-modeling cost against conditioning informativeness [Fig. 5].
DINOv2-conditioned DiT achieves strong FID when given ground-truth continuous embeddings but degrades sharply when those embeddings must be sampled, because the continuous DINOv2 distribution is hard to model; DLC’s discrete formulation avoids that failure mode [Table 2; §6 discussion of DINOv2 vs RCG baselines].
Composing two DLCs (e.g. mushroom + jellyfish) and conditioning the diffusion model on the mixed code produces coherent OOD samples that combine the semantics of both source classes [Fig. 1b, Fig. 6].
A pretrained text diffusion LM (LLaDA) can be fine-tuned to generate DLCs from text captions, yielding a text-to-DLC-to-image pipeline that produces novel samples outside the image generator’s training distribution [§7].
DLC reduces reliance on CFG relative to continuous-embedding baselines at matched ImageNet FID [Table 1 caption].

Method

A frozen DINOv2-L ViT extracts per-patch features; a randomly-initialized linear projection maps each feature to a Simplicial Embedding — a distribution over a vocabulary of V image tokens. The argmax of each SEM gives a discrete token, yielding a length-L DLC sequence (e.g. L=512, V=256). The diffusion model p(x|c) is a standard DiT-XL/2 conditioned on the DLC tokens via cross-attention. A separate text-discrete-diffusion model p(c) (an autoregressive / masked-diffusion text LM) models the DLC distribution itself, so at inference the pipeline is: sample c ~ p(c), then sample x ~ p(x|c) via DiT. For text-to-image, the authors fine-tune LLaDA (a masked diffusion LM) on (caption, DLC) pairs so it emits DLCs conditioned on text; the same DiT then renders the DLC.

Results

Unconditional ImageNet-256 FID 1.59 with DLC512 + DiT-XL/2 — reported as SOTA on this setting [Table 1].
DLC-conditioned DiT outperforms continuous-SSL-conditioned DiT and DINOv2-conditioned RCG-style baselines under matched sampling [Table 2].
Compositional OOD samples (e.g. “jellyfish + mushroom”, “komodo dragon + carbonara”, “tabby cat + golden retriever”) are coherent and combine recognizable features of both source classes [Fig. 1b].
Generative FID continues improving with sequence length L, indicating the bottleneck is conditioning bitrate, not model capacity [Fig. 5].
LLaDA fine-tuned on (caption, DLC) pairs produces text-conditioned samples that fall outside the image generator’s training distribution — a productivity claim distinct from improved FID [§7, Fig. 1 right column].

Why it’s interesting

DLC sits in the same design space as Diffusion training efficiency‘s “supervise diffusion with strong external features” recipes — RAE / REPA / P-DINO all use DINOv2 as the substrate, and DLC adds a fourth use: as the input space of the diffusion conditional, discretized so a separate small prior can model it. It also complements Classifier-Free Guidance variants by reducing reliance on CFG via richer conditioning rather than redesigning the guidance mechanism (Ostris, SoFlow, CAFM) or the unconditional branch (SparkVSR). The text-to-DLC-to-image route is an unusual recipe — it uses a diffusion LM (LLaDA) rather than an autoregressive LM as the text-side prior, putting it in conversation with Mercury: Ultra-Fast Language Models Based on Diffusion and Diffusion Language Models; the compositional OOD claim (Fig. 1b) is the strongest evidence in the cluster that discrete SSL-token vocabularies meaningfully extrapolate beyond the training set.