Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation

TL;DR

The paper (ICML 2025) argues that standard diffusion distillation is ill-posed: a multi-step teacher and a one-step student differ in both step size and effective compute (and often parameter count), so they converge to different local minima, and forcing the student to instance-match the teacher (L2 / consistency) is the wrong objective. The proposed alternative — D2O (Diffusion to One-Step) — drops the distillation loss entirely and finetunes the pretrained diffusion U-Net with only a standalone GAN objective (Projected GAN multi-scale discriminator + r1 regularization). With this, the diffusion training is reinterpreted as generative pre-training: most of the capability is already in the weights, and lightweight GAN finetuning “unlocks” it for one-step generation. D2O-F demonstrates this directly by freezing ~85.8% of parameters (only normalization layers, QKV projections, and skip connections stay trainable), beating distillation baselines while using ≤5M training images.

Key claims

Standard distillation is structurally suboptimal because teacher (multi-step solver) and student (1–4 step model) differ in step size and parameter count, so they converge to different local minima; direct instance-level imitation (L2, consistency) constrains the student to the teacher’s suboptimal-for-one-step minimum [§1, §3].
A standalone GAN objective — without any distillation loss — is sufficient to convert a pretrained diffusion U-Net into a one-step generator that matches or beats prior distillation methods [§1, §4].
The right GAN recipe is a Projected GAN multi-scale discriminator over features from a frozen pretrained network (VGG16-BN, EfficientNet-lite0) with non-saturating loss + r1 regularization; differentiable augmentation (diffAUG) hurts in this regime [§4.2].
Diffusion pre-training equips the U-Net with general generative capability; GAN finetuning is an “unlocking” step rather than from-scratch training, evidenced by competitive results at 0.2M images and near-SOTA at 5M images on CIFAR-10 — orders of magnitude less than typical distillation [§1, §5.1].
D2O-F: freezing ~85.8% of parameters (the convolutional layers) while leaving only the normalization layers (ADA-GN / GN), QKV projections, and skip connections trainable yields the best one-step results, sometimes outperforming full-D2O [§5.2, Table 2].
The freeze-most-of-the-network recipe only works with the GAN objective: applying the same parameter freezing under a Consistency Distillation objective fails, supporting the claim that distillation needs to change the core conv weights (because the teacher’s optimum is in a different basin) while the GAN objective can find a one-step minimum within the pre-trained feature space [§5.3].
Stacking a Consistency Distillation loss on top of D2O / D2O-F slows training and does not improve final performance — distillation does not compose with the GAN-only recipe [§5.3].
Frequency-domain analysis: across multi-step inference, the diffusion U-Net suppresses noise across all frequencies (more at high-noise/early steps), and refines low frequencies early then high frequencies late; deeper (low-res) U-Net blocks specialize on lower spatial frequencies and shallower blocks on higher ones — proposed as the mechanism behind diffusion-as-pre-training [§6].
Headline numbers, all one-step:
- CIFAR-10 unconditional: D2O-F FID 1.54 at 5M images [Table 1].
- ImageNet 64×64 conditional: D2O-F FID 1.16 at 5M images, vs SiD 1.52 at 930M images and DMD2 1.26 with longer training [Table 1].
- AFHQv2 64×64, FFHQ 64×64: near-SOTA at ≤5M images [Table 1].

Method

D2O takes a pretrained diffusion U-Net $G_\theta$ (EDM-style) and treats it as a generator: a noise sample $z \sim P_z$ (shaped to match the model’s input at the target timestep) is passed once through $G_\theta$ to produce a sample. A non-saturating Projected GAN objective is applied:

$\max_{\{D_l\}_{l \in L}} \mathbb{E}_x[\log D_l(P_l(x))] + \mathbb{E}_z[1 - \log D_l(P_l(G_\theta(z)))]$ $\min_{G_\theta} \sum_{l \in L} \mathbb{E}_z[-\log D_l(P_l(G_\theta(z)))]$

where $\{P_l\}_{l \in L}$ are feature extractors from multiple layers of frozen pretrained networks (VGG16-BN, EfficientNet-lite0). r1 gradient penalty on the discriminator is critical; differentiable augmentation is harmful. No distillation loss is used.

D2O-F runs the same GAN objective but freezes the convolutional layers of the pretrained U-Net (≈85.8% of parameters), leaving only normalization layers (ADA-GN / GN), QKV projections in attention, and skip-connection 1×1 convs trainable. This is the “unlock the pre-trained generator” mode, and it converges faster and to better FID than full D2O on every reported dataset.

The frequency-domain analysis is a separate diagnostic, not part of the training objective: log-FFT differences $\mathrm{DFFT}(x_1, x_2) = \log(|FFT(x_1)|+1) - \log(|FFT(x_2)|+1)$ are computed between (a) U-Net input vs output at each timestep and (b) consecutive U-Net outputs across timesteps, then aggregated per-block to characterize what each level of the U-Net hierarchy does during the multi-step trajectory.

Results

CIFAR-10 unconditional, one-step FID: D2O-F 1.54 at 5M training images, D2O 1.62; baselines include DMD 2.62, CTM 1.98, SiD 1.92. CIFAR-10 conditional, one-step FID: D2O-F 1.45.

ImageNet 64×64 conditional, one-step FID: D2O-F 1.16 at 5M images, vs SiD 1.52 at 930M images, DMD2 1.26 with longer training, CTM 1.92, ECM 2.49. At 0.2M images, D2O-F already gives “promising” results well below from-scratch GAN baselines, which the authors take as evidence that the U-Net is not learning generation from scratch but adapting pre-trained capability [Table 1, §5.1].

AFHQv2 64×64 and FFHQ 64×64: D2O-F is near-SOTA at ≤5M images. Visual samples are reported qualitatively comparable to multi-step EDM.

Ablations: removing r1 regularization or adding differentiable augmentation both degrade FID; multi-scale Projected GAN beats single-scale and StyleGAN-style discriminators; freezing convolutions improves over freezing nothing in the GAN setting but destroys training under a Consistency Distillation loss, isolating the recipe to GAN-only [§5.2, §5.3]. Training is reported to converge much faster than CD- or DMD-style distillation under the same compute.

Why it’s interesting

D2O is the missing predecessor for the wiki’s Continuous Adversarial Flow Models (CAFM, April 2026): CAFM explicitly argues that adversarial post-training is not distillation and helps multi-step generation by improving distribution matching; D2O is the same separation pushed to the one-step extreme — GAN alone, no distillation, on a pretrained diffusion model. Both papers credit the learned discriminator (vs Euclidean / consistency losses) as the load-bearing piece. D2O additionally provides the strongest evidence yet that the diffusion U-Net is overcomplete for one-step generation: 14% of the parameters (normalization + QKV + skip) carry the entire adaptation.

This sharpens the conflict the wiki’s Diffusion Distillation page already tracks. ArcFlow (ArcFlow: Unleashing 2-Step Text-to-Image Generation via High-Precision Non-Linear Flow Distillation) argues the linear shortcut in distillation underfits the teacher’s tangent variation and fixes it with a non-linear flow integral. DP-DMD (Diversity-Preserved Distribution Matching Distillation for Fast Visual Synthesis) argues reverse-KL is mode-seeking and fixes it with v-prediction at step 1. D2O argues the entire teacher-tracking premise is the wrong basin and fixes it by deleting the distillation loss. The three claims are not contradictory but they push in opposite directions on “how much of the teacher should the student copy” — D2O sits at the “none” end.

Operationally, the D2O-F freezing recipe (normalization layers + QKV projections + skip connections) is a near-twin of the recipes Parameter-Efficient Finetuning tracks for domain-shift adaptation — DiffHDR: Re-Exposing LDR Videos with Video Diffusion Models (rank-32 LoRA on DiT attention + FF), HDR Video Generation via Latent Alignment with Logarithmic Encoding (LumiVid) (<1% params), SPDA-SAM: A Self-prompted Depth-Aware Segment Anything Model for Instance Segmentation (LoRA on frozen SAM). D2O-F is the same template applied to the generation-mode shift (multi-step → one-step) rather than the domain shift (LDR → HDR; RGB → RGB+D). That two unrelated transfers converge on “freeze the conv backbone, train norms / projections / skips” is the kind of structural agreement that suggests this is the right primitive for adapting large pretrained diffusion U-Nets across a wide class of distribution shifts.