DC-Gen: Post-Training Diffusion Acceleration with Deeply Compressed Latent Space

TL;DR

DC-Gen is a post-training pipeline from MIT-Han-Lab / NVIDIA that takes an already-trained text-to-image diffusion model and moves it onto a much more deeply compressed latent space without training from scratch. The core trick is a lightweight embedding-alignment phase that bridges the representation gap between the base model’s latent and the deep-compression latent, after which the base diffusion model can be fine-tuned cheaply on top. The framework targets the inefficiency of running high-resolution (e.g. 4K) diffusion in the original loosely-compressed latent space, recovering near-base quality while exploiting the smaller token grid of a deep-compression AE. It is the post-training counterpart to the same group’s DC-AE work — instead of training a new diffusion model from scratch in a deep-compression latent, DC-Gen migrates an existing one.

Key claims

Existing text-to-image diffusion models hit significant efficiency bottlenecks at high resolution (e.g. 4K) because their latent spaces carry substantial redundancy [Abstract].
DC-Gen replaces training-from-scratch with a post-training pipeline that preserves the quality of the base model while migrating it into a deeply compressed latent space [Abstract].
The principal obstacle to direct fine-tuning in the new latent is a representation gap between the base model’s latent and the deep-compression latent, which destabilizes naive fine-tuning [Abstract].
A lightweight embedding-alignment training stage bridges this representation gap before any diffusion-side fine-tuning happens [Abstract].

Method

DC-Gen is a two-stage post-training framework applied on top of a pre-trained text-to-image diffusion model. Stage 1 — embedding alignment: a lightweight training pass aligns the base model’s latent space to the target deep-compression latent space, closing the representation gap that would otherwise destabilize direct fine-tuning. Stage 2 — diffusion adaptation: the base diffusion model is fine-tuned to operate in the new, more deeply compressed latent. Because the alignment stage handles the change of latent representation, the diffusion side only needs to adapt to a smaller token grid rather than relearn the data distribution from scratch. The framework is presented as generally applicable to existing text-to-image diffusion models rather than tied to a specific backbone.

Results

Headline numbers are not extractable from the abstract excerpt available at filing time (arxiv API was returning 429s on fetch). The framework is positioned as a quality-preserving accelerator for high-resolution generation (up to 4K) without the cost of training from scratch in the new latent space. A full results pass should be added on /bud refresh once arxiv fetch recovers.

Why it’s interesting

DC-Gen is the post-training pair to DC-AE 1.5: Accelerating Diffusion Model Convergence with Structured Latent Space — same lab, same lever (deep-compression latents like f32/f64 with high channel counts), but a different question: instead of “train a new diffusion model from scratch in a deep-compression latent,” DC-Gen asks “take an existing diffusion model and migrate it.” That migration is exactly the failure mode the DC-AE 1.5 paper documents: direct fine-tuning a DiT into a deep-compression latent is unstable because the latent’s structure is wrong for the base model, which is why DC-AE 1.5 had to redesign the latent (channel-wise ordering) to make the diffusion side learn it. DC-Gen takes the dual approach — keep the latent design, fix the diffusion side via an embedding-alignment phase. This complements L2P: Unlocking Latent Potential for Pixel Generation (L2P), which transfers a frozen LDM into pixel space by training only its shallow layers on synthetic source-model outputs — both are post-training latent-migration recipes but at opposite ends of the compression axis (L2P removes the latent, DC-Gen compresses it further). It also slots into Diffusion serving optimization as a complement to the algorithmic step-reduction frontier (TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Times, First Block Caching, NVFP4): DC-Gen attacks the spatial token count of the latent itself, orthogonal to step count and per-step kernel latency.