Alias-Free Latent Diffusion Models: Improving Fractional Shift Equivariance of Diffusion Latent Space

TL;DR

AF-LDM is a redesign of Stable Diffusion that makes both the VAE and the U-Net (approximately) fractional-shift-equivariant, so small input shifts produce correspondingly small output shifts rather than wildly different images. It ports StyleGAN3’s continuous-signal interpretation of features to the LDM stack and adds two LDM-specific pieces the StyleGAN3 recipe alone can’t handle: (1) an explicit equivariance loss that suppresses aliasing through training rather than only through alias-free modules, and (2) Equivariant Attention — re-using a fixed reference frame’s keys/values across shifted queries, which they show is mathematically the same operator as Cross-Frame Attention used in video editing. The result enables a much simpler warping-equivariant video-editing pipeline and consistent super-resolution / normal estimation across input shifts.

Key claims

Existing anti-aliasing modules (ideal sampling, filtered nonlinearities) alone are insufficient for LDMs because aliasing amplifies during VAE training even from a properly initialized alias-free model — the network learns to exploit leaked high-frequencies to improve reconstruction [§3.2; Table 2 rows 2-5].
The iterative denoising process compounds aliasing across U-Net steps: in standard LDM, shift PSNR degrades progressively through DDIM steps, while AF-LDM holds it nearly constant [Fig. 5].
Self-attention is not shift-equivariant under cropped shifts or warping because softmax + matrix multiplication couples all keys/values; equivariance under any relative deformation requires using a fixed reference frame’s K and V while shifting only Q [§3.4, Eq. 8–9].
Equivariant Attention has the same algebraic form as Cross-Frame Attention from video editing literature, but the paper argues it must be used during training, not just inference, to suppress aliasing — earlier video-editing work used CFA only at test time [§3.4].
An equivariance loss — penalizing the L2 distance between (shift then encode) and (encode then shift) on cropped valid regions — regularizes the VAE and U-Net to act as band-limited continuous signal operators; circular shift is used inside the U-Net loss (with cropped masking) to avoid boundary-pad artifacts during iterative denoising [§3.3, Eq. 4–6].
On ImageNet, AF-VAE reaches encoder SPSNR 45.10 / decoder SPSNR 33.88 versus 25.07 / 22.31 for the SD VAE baseline, at the cost of modest rFID degradation (1.00 vs 0.69) [Table 2].
On FFHQ unconditional LDM, AF-LDM reaches latent SPSNR 40.94 / image SPSNR 28.06 versus 27.47 / 12.38 for LDM+SD-VAE — order-of-magnitude better shift consistency, at FID 19.05 vs 14.91 [Table 3].
The model is robust to irregular warping (flow), not only rigid shifts — enabling a video-editing pipeline that just DDIM-inverts each frame, swaps the prompt, and regenerates, without explicit temporal modules or latent warping [§4.2, Table 4, Fig. 8].
Naive transfer of StyleGAN3’s full alias-free recipe is incompatible with LDMs: Fourier-feature latents and boundary-pixel cropping can’t be applied because the LDM’s latent is produced by an encoder operating on real images, not a free parameter [§3.2, Table 1].

Method

AF-LDM is built on top of Stable Diffusion’s VAE + text-conditioned U-Net and modifies both. On the architecture side, it replaces bilinear up/downsampling with FFT-based resampling (a closer approximation to ideal Shannon resampling than StyleGAN3’s Kaiser filters), wraps nonlinearities in up-then-downsample pairs to band-limit the high frequencies they introduce, and rewires self-attention into Equivariant Attention (= Cross-Frame Attention) where the K/V are computed once on a reference frame and reused for every shifted query.

On the training side, the loss adds an equivariance term that penalizes mismatch between (Fourier-shift then forward) and (forward then Fourier-shift) under random fractional offsets. The VAE encoder and decoder get this term separately. The U-Net’s term uses circular padding during the shift (to avoid the iterative denoising amplifying boundary errors) but masks to the cropped valid region when computing the loss. The U-Net is trained with Equivariant Attention enabled — the paper notes that without EA, the equivariance loss would just try to repair the attention layers instead of attacking the actual aliasing.

Both the VAE and the U-Net are initialized from Stable Diffusion weights and fine-tuned; the VAE is trained on ImageNet, the unconditional LDM on FFHQ, and a text-conditional AF-SD is retrained on LAION Aesthetic 6.5+. Downstream applications (super-resolution via latent I2SB, normal estimation via YOSO) reuse the same AF-VAE and AF-SD.

Results

VAE equivariance: Encoder SPSNR 25.07 → 45.10, Decoder SPSNR 22.31 → 33.88 (a +20 dB / +11.5 dB jump). rFID worsens 0.69 → 1.00. Reconstruction PSNR drops 26.94 → 25.53 [Table 2].
LDM equivariance on FFHQ: Latent SPSNR 27.47 → 40.94, Image SPSNR 12.38 → 28.06 — image-space SPSNR more than doubles. FID 14.91 → 19.05 — quality degrades modestly but consistency improves dramatically [Table 3].
Fractional shifts: standard LDM shows the characteristic “bouncing” — high SPSNR only at integer shifts, valleys at half-integer — while AF-LDM averages higher SPSNR with much lower variance across the fractional shift continuum [Fig. 6].
Denoising step trajectory: in vanilla LDM, SPSNR decays monotonically through the DDIM schedule; AF-LDM holds it roughly flat from t=0 to t=T [Fig. 5].
Warping-equivariant video editing: simple DDIM-invert-then-regenerate produces qualitatively coherent edited frames where the SD baseline flickers and breaks identity [Fig. 8, Table 4].
Downstream super-resolution (I2SB) and normal estimation (YOSO): AF-LDM-based variants produce visibly stable outputs across input shifts where the SD-based baselines flicker around high-frequency texture [Fig. 9–10].

Why it’s interesting

This paper occupies a niche on the wiki that is currently underweight: consistency / stability of LDMs as a first-class design target, distinct from quality (FID), efficiency (training compute), or capacity (bitrate). The wiki has several latent-design papers oriented around training efficiency or downstream gFID — What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusion (manifold properties), Unified Latents (UL): How to train your latents (bitrate as a hyperparameter), Autoencoders for Diffusion: A Deep Dive (engineering-side diffusability) — but none target shift-equivariance explicitly, and the “weird shifting and grid artifacts” Terrance flagged in the Slack post are exactly the failure mode this paper attacks at its source. Two specific connections:

The Equivariant-Attention-as-Cross-Frame-Attention identity is a clean theoretical rationale for why CFA helps in inversion-based video editors. The wiki’s video-editing papers (and the Layered Image/Video Decomposition cluster’s editing-focused entries) use CFA as an empirical trick; AF-LDM frames it as the only attention form that satisfies shift-equivariance under cropped shifts, and shows it has to be in the training loop, not just inference, to do real work.
The signal-processing diagnosis (aliasing amplifies during training in the absence of an explicit equivariance term) sits adjacent to The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks‘s analysis of how training dynamics amplify outlier features — both are stories about networks exploiting non-band-limited frequencies to fit data, with consequences that show up only at inference under perturbations the training distribution didn’t contain.