PV-VAE: Video Generation with Predictive Latents

TL;DR

PV-VAE is a video VAE for latent diffusion that adds a predictive reconstruction objective on top of standard reconstruction losses: during training, future frames are randomly discarded, the encoder sees only the observed past, and the decoder must reconstruct the entire (past + future) video. The thesis is that this forces latents to encode temporal dynamics and motion priors rather than just pixel-accurate appearance, which improves downstream diffusion generation without complicating the training pipeline. At t4s16c64 high compression, PV-VAE reports UCF101 FVD 146.37 / IS 69.72 and RE10K FVD 72.50, beating Wan2.2 VAE (180.79 / 87.15) at matched compression with comparable train throughput.

Key claims

Better reconstruction does not imply better latent diffusion generation; PV-VAE’s central premise is that diffusability of latents is a distinct property from reconstruction fidelity [Introduction].
A predictive reconstruction objective — discard future frames at random, encode only observed past, pad the latent sequence to the original length, decode the complete video — makes latents encode motion priors and smoother temporal trajectories [Training objective].
At t4s16c64 (4× temporal, 16× spatial, 64 channels), PV-VAE achieves UCF101 FVD 146.37 / KVD 14.52 / IS 69.72, beating Wan2.2 VAE (180.79 / 17.80 / 67.32) and SSVAE (168.68 / 19.71 / 66.39) at the same latent config [Table 1, “Generation performance”].
The same model reaches RE10K FVD 72.50 / KVD 4.06, the strongest reported among 8 baselines including CogX-VAE, IV-VAE, WF-VAE-L, Hunyuan-VAE, Wan2.1 VAE, Wan2.2 VAE, and SSVAE [Table 1, “Generation performance”].
Train speed (4.40) and train memory (33.34) are competitive with Wan2.2 VAE (4.96 / 30.90) at higher channel count — the predictive objective does not impose a meaningful throughput tax [Table 1].
After predictive training, the encoder is frozen and the decoder is fine-tuned for standard reconstruction; this two-stage recipe is what closes the train/inference gap between predictive and standard reconstruction modes [Training objective].
Reconstruction quality is competitive but not SOTA: at 256² Kinetics-400, rFVD 3.45 / PSNR 32.26 / LPIPS 0.020 trails Hunyuan-VAE (2.96 / 34.30 / 0.016) and IV-VAE (2.78 / 34.08 / 0.019), explicitly accepting reconstruction cost for generation gain [Table 2, “Reconstruction performance”].
PCA visualizations and a temporal-distance probe support the mechanism claim: PV-VAE latents allocate more representational capacity to dynamic foregrounds, and latent temporal distance is smoother for adjacent frames while more monotonic across longer intervals [Visual evidence section].
LVDM features trained on PV-VAE latents transfer better than on baseline VAE latents to optical flow, next-frame prediction, and tracking probes — i.e. the motion-aware latent improves downstream understanding, not only generation [Visual evidence section].

Method

PV-VAE keeps a standard video-VAE architecture — 3D causal convolutions, 16× spatial / 4× temporal compression, 64 latent channels — and modifies only the training objective. In each step, a random suffix of the input clip’s frames is masked out. The encoder consumes only the observed (un-masked) prefix and produces a shorter latent sequence, which is then padded back to the original latent length before being decoded. The decoder is asked to reconstruct both the observed frames (a reconstruction task) and the masked future frames (a prediction task), under the standard video-VAE loss stack: MSE, temporal-difference reconstruction, LPIPS, GAN, KL.

After this predictive pretraining, the encoder is frozen and the decoder is fine-tuned on the standard “see-everything → reconstruct-everything” objective. This second stage closes the train/inference gap: at inference the encoder sees the full clip (no masking), and the fine-tuned decoder is calibrated for that input distribution. The recipe deliberately adds no auxiliary networks, no external feature distillation, and no separate prediction head — the predictive supervision is folded into the existing decoder.

Results

Headline generation numbers on Latte-style class-conditional 17-frame video generation at 256²:

UCF101: PV-VAE FVD 146.37 / KVD 14.52 / IS 69.72 vs Wan2.1 VAE 167.10 / 11.54 / 66.04 (t4s8c16, lower compression), Wan2.2 VAE 180.79 / 17.80 / 67.32 (t4s16c48, matched compression), SSVAE 168.68 / 19.71 / 66.39 (t4s16c48). PV-VAE is best on FVD and IS among same-compression baselines [Table 1].
RE10K: PV-VAE FVD 72.50 / KVD 4.06 vs Hunyuan-VAE 83.45 / 13.23, Wan2.1 VAE 83.84 / 10.64, Wan2.2 VAE 87.15 / 10.11 — best across both compression families [Table 1].

Reconstruction at 17×256²: rFVD 3.45 / PSNR 32.26 / SSIM 0.95 / LPIPS 0.020, with infer speed 0.69 / infer memory 7.97. Hunyuan-VAE leads at 2.96 / 34.30 / 0.97 / 0.016 but with infer memory 22.00 [Table 2]. At 512², rFVD 1.88 / PSNR 35.03, again trailing Hunyuan-VAE 0.90 / 37.13 [Table 2].

The trade is explicit: PV-VAE gives up some reconstruction quality (because the encoder is trained to retain only what’s needed to predict the future, not perfectly reconstruct the past) and recovers it via decoder fine-tuning, while keeping a latent space that lights up downstream diffusion FVD by 20–35 points at matched compression.

Code release is blocked by confidentiality; the page notes the recipe is light enough to reproduce on top of existing video-VAE pipelines.

Why it’s interesting

PV-VAE is a clean datapoint for the wiki’s open question about how to design a video latent space that is good for downstream diffusion, not just reconstruction — the central thesis of the Diffusion training efficiency page. The cluster of recent fixes there has been on the image side (RAE, UL, PixelGen, SRA, Self-Flow) and largely about what features the tokenizer aligns to or what loss the diffusion gets. PV-VAE proposes an orthogonal lever: shape the encoder via a self-supervised predictive objective on video itself, with no external feature target and no diffusion-side change. This complements Making Reconstruction FID Predictive of Diffusion Generation FID, which provides the diagnostic (iFID) for why reconstruction-only VAEs hurt diffusion — PV-VAE is one concrete recipe that could plausibly improve iFID by forcing more interpolable, temporally-smooth latents (though the page does not report iFID).

The motion-aware-latent angle also connects to Adaptive 1D Video Diffusion Autoencoder (One-DVA), which routes detail/motion into a separate 1D query latent next to a 2D structural latent — PV-VAE makes the single 2D-style latent itself motion-aware via training objective rather than via architectural split. Contrasts with TC-AE: Unlocking Token Capacity for Deep Compression Autoencoders (TC-AE), which fixes the tokenizer reconstruction-vs-semantics tradeoff on the image side via staged compression + internal iBOT — both papers share the design pattern of internal self-supervision at the tokenizer instead of an external pretrained backbone. Worth flagging that PV-VAE is the first filed paper here where the self-supervision specifically targets temporal structure, which is the modality where external supervision (V-JEPA 2, Depth Anything 3) was reported to hurt in Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis.