Making Reconstruction FID Predictive of Diffusion Generation FID

TL;DR

The “reconstruction–generation dilemma” — VAEs with better rFID often produce worse latent-diffusion gFID — has been the central empirical headache motivating RAE / VA-VAE / EQ-VAE / UL. This paper proposes interpolated FID (iFID): for each validation image, find its nearest neighbour in latent space, spherically interpolate the two latents, decode, and compute FID against the source set. Across 13 publicly available VAEs (SD-VAE, FLUX-VAE, QwenImage-VAE, SD3-VAE, EQ-VAE, VA-VAE, SOFT-VQ, MAE-TOK, DE-TOK, DM-VAE, REPAE-VAE, RAE, IN-VAE) and SiT-B/SiT-XL diffusion models, iFID is the first VAE-side metric to show a strong correlation with gFID — Pearson 0.85–0.89, Spearman 0.86–0.92 — while rFID’s correlation is near zero and PSNR/SSIM/LPIPS are strongly negatively correlated. The paper also gives a mechanism: diffusion models generate unseen samples by interpolating training-set latents, so iFID directly measures the validity of those interpolations.

Key claims

rFID’s correlation with SiT-XL gFID is near-zero (PCC −0.06 to −0.15) while iFID achieves PCC 0.85–0.89 / SRCC 0.86–0.92 across the 13-VAE benchmark, with and without CFG [§4.2, Table 3].
PSNR, SSIM, LPIPS are strongly negatively correlated with gFID (PCC −0.73 to −0.83), empirically confirming the reconstruction–generation dilemma rather than dismissing rFID’s poor correlation as noise [§4.2, Table 3].
Refinement-phase quality is what rFID actually measures: rFID correlates strongly with gFID(t) at small t (0.37 at t=0.1) and decays to ~0 by t=0.4; iFID has the opposite profile, weak at small t and strong at large t (0.85 at t=0.8) [§3.3, Table 1].
iFID’s strong correlation rests on a known result: a diffusion score estimator that exactly matched the empirical score would only replicate training data, so the model’s generalization comes from underfitting, and the unseen samples it produces are local interpolations of training latents (Kamb & Ganguli 2024; Aithal et al. 2024) [§3.4].
Reconstruction losses prefer an isolated, disconnected latent space because the decoder finds it easier to distinguish modes, but diffusion sampling needs a connected interpolable latent — this is presented as the mechanism of the reconstruction–generation dilemma, with a 2-mode Gaussian toy example [§3.5, Fig. 4].
Spherical latent interpolation gives the best correlation (PCC 0.84 vs linear 0.78 / mask 0.74) for Gaussian VAEs, consistent with preserving the Gaussian prior; results are robust to reference-dataset size (50k–1000k) and to top-K (K=1 vs K=10) NN [§4.3, Table 4].
Interpolation strength α=0.5 (true midpoint) is the most diagnostic; correlation with gFID rises monotonically with α from 0.22 at α=0.1 to 0.89 at α=0.5; smaller α values stay closer to rFID-like behaviour [§4.3, Table 5].
Diffusion-loss as a VAE-quality metric is weak — PCC ≈ 0.21–0.34 with gFID — and so are EQ-Loss / SE-Loss / VF-Loss / GMM-Loss; iFID is the only VAE-side metric in the comparison that crosses 0.8 [§4.2, Table 3].

Method

For a validation set $X=\{x_i\}$ , encode each $x_i$ to its latent $z_i$ , find the nearest-neighbour latent $\mathrm{NN}(z_i)$ in the latent space (using a 50k–1000k reference set drawn from the ImageNet training split), and form the interpolated latent $z_i' = \mathrm{slerp}(z_i, \mathrm{NN}(z_i), 0.5)$ . Decode each $z_i'$ through the VAE decoder $g_\phi$ to get a “fake” image, then compute FID between $\{g_\phi(z_i')\}$ and the original $X$ . This is iFID. Total cost is one NN lookup per validation sample plus a standard FID — no diffusion model is trained or evaluated.

The paper’s secondary methodological contribution is gFID(t): instead of evaluating end-to-end gFID, the diffusion model is asked to denoise samples that have been forward-noised to intermediate timestep $t$ , and gFID is computed against the source image. Sweeping $t \in [0, 1]$ separates “refinement-phase” sample quality ( $t$ small, semantics already determined) from “navigation-phase” sample quality ( $t$ large, semantics still in flux). This is the diagnostic that lets the paper show rFID and iFID cover non-overlapping regimes.

Results

Headline: iFID achieves PCC 0.89 / SRCC 0.91 with SiT-XL gFID-without-CFG, and PCC 0.88 / SRCC 0.92 with CFG, across the 13-VAE benchmark; no other reconstruction or VAE-regularization metric exceeds 0.85, and most are below 0.4 in absolute value [Table 3]. Per-phase decomposition: rFID correlates 0.37 with gFID(t=0.1) and 0 with gFID(t≥0.4); iFID correlates 0.26 at t=0.1 and 0.85–0.89 at t≥0.6 [Table 1]. Visualization: for reconstruction-only VAEs (SD-VAE, FLUX-VAE), nearest-neighbour latents decode to semantically irrelevant images and the midpoint interpolation is broken; for diffusion-tuned VAEs (VA-VAE, RAE), the NN is semantically similar and the midpoint is photorealistic [§4.2]. Robustness sweeps in Table 4 show that dropping the reference set to 50k or using K=10 NN barely moves the correlation (PCC 0.78–0.89). Code is released at github.com/tongdaxu/Making-rFID-Predictive-of-Diffusion-gFID.

Why it’s interesting

This is the missing diagnostic for the cluster of late-2025 / early-2026 papers that have been redesigning the VAE to make diffusion train faster — Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders (RAE), Unified Latents (UL): How to train your latents (UL), PixelGen: Improving Pixel Diffusion with Perceptual Supervision (PixelGen), DINO-SAE: DINO Spherical Autoencoder for High-Fidelity Image Reconstruction and Generation (DINO-SAE), and the RAEv2 follow-up — all of which are implicitly optimizing for something that rFID does not capture. iFID gives a single, training-free number to evaluate any of those latent designs before committing to a full DiT training run, and lets the team distinguish “good for reconstruction” tokenizers from “good for diffusion” tokenizers in minutes rather than weeks. The mechanistic claim — that reconstruction losses favour disconnected latents while diffusion needs connected ones — is also the cleanest single-sentence statement of why RAE-style frozen-encoder latents work; it sharpens the Diffusion training efficiency page’s open question about whether DINO/RAE/UL/PixelGen are converging on a common substrate (yes — they are all making the latent more interpolable). The negative result that diffusion loss itself only correlates ~0.3 with gFID is a useful caution for anyone using validation diffusion loss as a checkpoint-selection signal.