Spectral Progressive Diffusion for Efficient Image and Video Generation

TL;DR

Spectral Progressive Diffusion (SPD; project page acronym “SPEED”) is a training-free plug-and-play sampler for pretrained DiT image and video generators that grows the spatial resolution along the denoising trajectory rather than running every step at full resolution. The starting observation is the spectral autoregression of diffusion — low-frequency components emerge from noise first, high-frequency details only in late timesteps — so high-resolution computation at early timesteps is largely operating on noise-dominated frequencies and is redundant. The method derives an optimal resolution schedule from each model’s measured power spectrum and Nyquist limit, uses a spectral noise-expansion mechanism at each resolution transition, and adds an optional fine-tuning recipe for further quality. Reported speedups are 1.5–2× on SDXL / Lumina / CogVideoX with visual quality preserved.

Key claims

Pretrained diffusion models implicitly generate visual content autoregressively in the frequency domain — low-frequency content earlier in denoising, high-frequency content only in later timesteps — so high-resolution computation at early/noise-dominated timesteps is largely redundant [Abstract].
A general framework for progressively growing resolution along the denoising trajectory of any pretrained diffusion model can exploit this structure: at high noise, denoise at lower resolution; at low noise, denoise at full resolution [Abstract].
A spectral noise expansion mechanism handles the resolution-transition step — when the sampler upgrades from resolution $r_i$ to $r_{i+1}$ , the noise spectrum at the newly-added higher-frequency bands has to be initialized so the trajectory remains on-manifold for the pretrained model [Abstract].
The optimal resolution schedule is derived from the model’s measured power spectrum: by matching the noise level at which each high-frequency band begins to emerge from noise against the spatial resolution that can represent it (each resolution’s Nyquist limit), the schedule decides when to grow resolution rather than using a hand-chosen heuristic [project page §Method].
The framework is training-free by default — applies to a pretrained DiT with no fine-tuning — and an optional fine-tuning recipe further improves efficiency and quality [Abstract].
Reported speedups: 1.5–2× on SDXL, Lumina, and CogVideoX with visual quality preserved [§Results, per alanhou.org review].

Method

The pipeline is conceptually a resolution schedule layered over a standard PF-ODE sampler. Given a pretrained diffusion model and its measured power spectrum on a calibration set, the authors derive a schedule of $(r_i, t_i)$ pairs: at noise level $t_i$ , switch the denoising resolution to $r_i$ . The schedule is derived by, for each candidate resolution $r$ , computing the highest spatial frequency that resolution can represent (the Nyquist limit), and finding the noise level at which the model’s power spectrum at that band crosses out of the noise floor. Below that noise level the higher-frequency band is signal-dominated and needs higher resolution; above it, the band is noise-dominated and lower resolution is sufficient.

At each transition, the lower-resolution latent at $t_i$ is upsampled to the new resolution and the newly-added high-frequency components of the noise are sampled from the appropriate spectral distribution (this is the spectral noise expansion mechanism — it is what keeps the upsampled latent statistically consistent with what the pretrained model expects to see at that noise level). The PF-ODE solver then continues at the higher resolution. The base sampler can be Euler or a higher-order solver; multistep solvers reset to first-order at each transition (the unofficial ComfyUI port explicitly does this), since solver state across a resolution change is not valid [community implementation note]. An optional fine-tuning recipe further improves quality at aggressive schedules.

Results

Headline efficiency claim: 1.5–2× speedup on SDXL, Lumina, and CogVideoX while preserving visual quality [§Results, per alanhou.org review]. The community ComfyUI port reports a representative configuration start_scale=0.50, mid_scale=0.75, transition_1=0.80, transition_2=0.60 reaching 14.55s with taper=8 for crossfading the DCT seam to reduce ringing at transitions — a 1.82× speedup at qualitative parity on a tested anima model — though the unofficial implementation notes that visible artifacts can appear on some outputs and that results vary by model and prompt [ruwwww/ComfyUI-SPEED README].

The paper’s positioning relative to prior work: TokenFlow and FreeNoise exploited the same frequency-structure observation, but only for specific tasks (video consistency, editing); SPD is the first general framework that systematically reduces computation by matching resolution to the frequency content actually being generated at each timestep [project page §Related work, per alanhou.org review].

Why it’s interesting

SPD is the direct companion / follow-up paper to Foveated Diffusion: Efficient Spatially Adaptive Image and Video Generation from the same Stanford group (Chao, Yariv, Xiao, Wetzstein — same four authors, with first/last shuffled). Foveated Diffusion reduces tokens spatially by exploiting human perceptual eccentricity; SPD reduces tokens temporally-along-the-denoising-trajectory by exploiting the model’s own spectral autoregression. Both papers frame the headline as “compute-per-token reduction without quality loss” but on orthogonal axes — and both are plug-and-play on pretrained DiTs (FLUX.2 Klein / Wan2.1 for Foveation, SDXL / Lumina / CogVideoX here). The Wetzstein-group is methodically tiling out the efficiency-via-non-uniform-compute design space, and the two papers should be read together. Notable distinction: Foveated Diffusion needs post-training (LoRA rank 32) to avoid duplicated-object artifacts, while SPD is training-free by default and uses spectral noise expansion to avoid the analogous artifact at resolution transitions.

The paper also matters for the broader [[diffusion-training-efficiency]] discussion the wiki has been building. SPD adds a seventh axis to that thread: progressive-resolution sampling. The page currently tracks (1) external-encoder supervision (RAE/REPA/P-DINO), (2) internal self-distillation (SRA/Self-Flow), (3) latent design (UL/L2P), (4) noise-conditioning-free training (Geometry of Noise), (5) CFG-free training (SoFlow, Ostris), (6) spatial token-count reduction (Foveated Diffusion). SPD is the first axis that is inference-only and training-free — it doesn’t change what the model knows, only when/where it spends compute given the model’s own spectral structure. That makes it operationally close to caching/serving recipes like Run FLUX.1-dev three times faster and FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU, but mechanistically very different: those recipes exploit redundancy across timesteps (cache reuse), while SPD exploits redundancy across spatial frequencies at a given timestep.

For Luma specifically: this is a drop-in 1.5–2× speedup on real-time video generation backbones with no retraining, and it stacks cleanly with foveated post-training (since the foveation axis is spatial and SPD’s is spectral) and with step-reduction / distillation (since SPD operates within a sampling step, not across them). Sander Dieleman’s quote-RT framing — “at high noise, only low frequencies need to be represented accurately” — is the cleanest statement of why this is a free lunch for any DiT with a measurable spectral autoregression structure. The unofficial ComfyUI port already exists [github.com/ruwwww/ComfyUI-SPEED], so the practical engineering cost to evaluate on internal Luma video backbones is small.