Is Noise Conditioning Necessary for Denoising Generative Models?

TL;DR

Sun, Jiang, Zhao & He (Kaiming He’s group) challenge the prevailing assumption that timestep / noise-level conditioning is indispensable in denoising diffusion models. They reformulate the training/sampling loop to expose the regression target as the posterior expectation over (data, noise, t) triplets producing a given noisy sample x_t, then show that high-dimensional natural images concentrate this posterior so tightly that a t-free network can approximate the t-conditional one. Most diffusion variants (iDDPM, EDM, FM, Consistency Models) degrade only gracefully when t-conditioning is removed; some flow-matching variants improve slightly. A tailored “uEDM” reaches FID 2.23 on CIFAR-10 unconditional vs 1.99 for the t-conditional baseline. Only DDIM with ODE sampler fails catastrophically (FID 40.90), and a derived error bound — computable from schedules alone, no training — explains why.

Key claims

The training loss in Eq. 2 has a unique effective regression target Ḡ(x_t, t) that is the expectation of the raw target G(x_0, ε, t) over all (x_0, ε, t) triplets producing the same x_t; the noise-unconditional variant minimizes the further-marginalized Ḡ(x_t) where t is also integrated out [§4.1, Eq. 5-8].
Under high-dimensional data (d ≥ Õ(1), d̃ « d), the conditional distribution p(t | x_t) is sharply concentrated — variance O(1/d) — so the t-marginalized target is close to the t-conditional one and a t-free network can fit it [§4.2, Statement 1, Fig. 3].
The per-step regression error ε_eff between Ḡ(x_t) and Ḡ(x_t, t) is bounded by O(1/√d) × σ_data, i.e. small enough that the approximation is reliable for natural images [§4.3, Statement 2, Eq. 11].
An iterative-sampler accumulated-error bound (Statement 3, Eq. 12) can be computed from the noise schedule and dataset alone — no training required — and correlates strongly with empirical FID degradation across iDDPM, DDIM, EDM, FM, uEDM [§4.4, Fig. 4].
The bound predicts and the experiments confirm that DDIM with ODE sampler is the only catastrophic failure case: bound is ~3 × 10^6 vs ~10^2-10^3 for stable methods, and DDIM-blind FID is 40.90 vs ≤ 4.55 for everything else [§6.2, Table 4].
Flow-Matching variants (1-RF) are insensitive or improve when t-conditioning is removed: FID 3.01 → 2.61 with Euler 100, 2.87 → 2.63 with Heun 99 — consistent with FM’s regression target being t-independent in their reformulation [§6.1, Table 2].
Consistency Models (iCT, ECM) — believed to be highly sensitive to t-embedding details — still produce coherent samples without t-conditioning (FID 2.59 → 3.57 for iCT) [§6.1, Table 2].
The proposed uEDM, a simple schedule modification of EDM (c_in(σ) = 1/σ_data instead of σ-dependent), reaches FID 2.23 on CIFAR-10 unconditional without any t-input, narrowing the gap to t-conditional EDM (1.99) and beating Heun-EDM-blind (3.36) [§5, §6.1, Table 2].
Increasing sampler stochasticity (DDIM η-interpolation between ODE and SDE) monotonically improves blind-FID — the SDE’s independent noise injections enable per-step error cancellation that the ODE’s deterministic dynamics cannot [§6.2, Fig. 6].
Replacing the true t with a predicted t from an auxiliary network (supervised or unsupervised) does not recover the t-conditional FID — predicted-t variants behave similarly to the t-free variant, because even with ground-truth supervision the predictor inherits the residual uncertainty in p(t|x_t) [§6.2, Fig. 7, Table 5].

Method

The framework starts with a unified affine schedule x_t = α_t x_0

σ_t ε that subsumes iDDPM, DDIM, EDM, and Flow Matching as choices of (α_t, σ_t) and the prediction target G(x_0, ε, t) ∈ {ε-pred, x_0-pred, v-pred}. The neural net f̂_θ is then either f̂_θ(x_t, t) (standard) or f̂_θ(x_t) (the t-free variant under study). The unified sampler in Eq. 4 encapsulates Euler, Heun, DDPM-SDE, DDIM-ODE and others.

The theoretical core is the Effective Target analysis: by construction, MSE regression in Eq. 2 against a stochastic target G collapses to fitting its conditional expectation Ḡ given the network’s inputs. The t-conditional case yields Ḡ(x_t, t); the t-free case yields Ḡ(x_t) — the t-marginalized version. The authors then bound the L2 distance between Ḡ(x_t) and Ḡ(x_t, t) via the variance of p(t|x_t), which they prove decays as O(1/d) for high-dimensional data lying on a low-dimensional manifold (Statement 1). The single-step error bound (Statement 2) propagates through the iterative sampler under a Lipschitz assumption on Ḡ to give the accumulated bound (Statement 3, Eq. 12).

uEDM is a minimal modification of EDM motivated by the bound: setting c_in(σ) = 1/σ_data (a constant w.r.t. σ_t) instead of EDM’s σ-dependent preconditioner removes one source of σ-dependence the network would otherwise have to track without seeing t. All other EDM schedule choices are preserved.

Results

CIFAR-10 unconditional, FID (w/ t → w/o t): iDDPM SDE 500 NFE 3.13 → 5.51; DDIM ODE 100 NFE 3.99 → 40.90 (catastrophic); DDIM SDE 1000 3.18 → 5.41; ADM SDE 250 2.70 → 5.27; EDM Heun 35 1.99 → 3.36; FM (1-RF) Euler 100 3.01 → 2.61 (improves); FM RK45 127 2.53 → 2.63; iCT 2-step 2.59 → 3.57; ECM 2.57 → 3.27; uEDM Heun 35: 2.04 → 2.23 (best blind result) [Table 2].
Accumulated-error-bound vs empirical FID: DDIM 3×10⁶ ↔ FID 40.90; EDM 1×10³ ↔ 3.80; FM(1-RF) 1×10² ↔ 2.61; uEDM 1×10² ↔ 2.66. Three orders-of-magnitude gap between catastrophic and stable methods, predicted before training [Table 4].
Stochasticity sweep: DDIM η ∈ [0,1] interpolates from ODE to SDE; blind-FID monotonically improves from ~41 (η=0) to ~5 (η=1), matching the iDDPM-blind level [§6.2, Fig. 6].
Predicted-t variants (b: pretrained noise predictor; c: jointly-trained noise predictor; d: t-free baseline) all give near-identical FID: e.g. EDM gives 1.99 / 3.27 / 3.39 / 3.36 across (a,b,c,d) — supervising the predictor with ground-truth t does not recover the t-conditional gap, because the residual uncertainty in p(t|x_t) is the limiting factor [Fig. 7].
Generalization to other datasets: ImageNet 32² FM 1-RF 5.15 → 4.85 (improves); FFHQ 64² EDM 2.64 → 3.59; CIFAR-10 class-conditional EDM 1.76 → 3.11, FM 2.72 → 2.55 — same pattern as unconditional [Table 3].

Why it’s interesting

This is the paper that opened the noise-conditioning-free thread in diffusion modeling — and it’s exactly the t-removal baseline Dejia’s Slack thread was probing for the in-house DiT. The result matters because timestep conditioning is one of the few “always-on” inputs to a diffusion DiT, plumbed through every AdaLN-zero block; removing it would dramatically simplify the architecture and free parameters for other modulation. This paper says it can be done on CIFAR-10-class problems, with a caveat that the parameterization matters far more than people realized — and that’s the wedge The Geometry of Noise: Why Diffusion Models Don't Need Noise Conditioning later expanded into a full theoretical account, proving that velocity-prediction (FM, EqM) is structurally stable under blind generation while noise-prediction (DDPM/DDIM) is structurally unstable due to Jensen Gap amplification. Read together, these two papers explain why v-pred + FM is the right primitive for any architectural simplification of the t-conditioning pathway.

The in-context timestep token alternative Dejia is comparing against is captured in Improved Mean Flows: On the Challenges of Fastforward Generative Models (iMF), which retains t but moves it into multi-token in-context conditioning that replaces adaLN-zero entirely (cutting the B-size model from 133M to 89M parameters at FID 4.57 → 4.09). Dejia’s in-house finding — that 4–16 in-context timestep tokens suffice at 256px but degrade at 512px and 1024px — is a useful counterpoint to iMF’s CIFAR/ImageNet-256² results. The pair (this paper + iMF) brackets the design space: remove t entirely (Sun et al.) vs keep t but as in-context tokens, not AdaLN (iMF), with AdaLN modulation as the surprisingly hard-to-beat default at higher resolutions.