VideoVAE+: Large Motion Video Autoencoding with Cross-modal Video VAE

TL;DR

VideoVAE+ is an ICCV 2025 video VAE from HKUST that targets large-motion reconstruction — the regime where prior video VAEs (Open-Sora, CV-VAE, Cosmos, CogVideoX) consistently exhibit motion ghosting, edge blur, and color drift. The recipe has three parts: (1) a two-stage spatiotemporal architecture that combines simultaneous ST compression (good at detail) and sequential ST compression (good at motion) instead of choosing one; (2) cross-modal training that injects the text caption from the source text-to-video dataset via cross-attention at every ResNet block of the encoder/decoder; (3) joint image-video training enabled by a fully spatiotemporally-factorized network where image batches skip all temporal modules. The 16-channel model achieves PSNR 34.15 / SSIM 0.94 / LPIPS 0.027 on WebVid, beating CogVideoX-VAE by ~1.3 dB at matched compression (4×8×8).

Key claims

Inflating an SD VAE to 3D via simultaneous spatiotemporal compression yields good detail recovery but poor motion recovery; sequential compression (frame-by-frame SD VAE → lightweight temporal autoencoder) yields good motion recovery but poor detail — combining both into a two-stage temporal-aware spatial autoencoder + temporal autoencoder beats either alone on large-motion test sets [§3.2, Table 3].
On the large-motion ablation, the two-stage solution reaches PSNR 24.67 / SSIM 0.72 / LPIPS 0.116 vs. simultaneous 24.06 / 0.73 / 0.129 and sequential 23.37 / 0.69 / 0.148 [Table 3].
Cross-modal training is the first reported use of text as a conditioning signal inside a video VAE: visual feature patches act as queries and values, Flan-T5 text embeddings act as keys via cross-attention after each ResNet block, with patch sizes 8/4/2/1 across the four encoder layers [§3.3].
The model uses standard Conv3D instead of the CausalConv3D of MagViT-V2, OD-VAE, and OPS-VAE [§3.4].
Joint image-video training (8:2 video:image batch ratio) is critical: without it, the 16-channel model collapses to PSNR 9.26 / LPIPS 0.68 on image reconstruction; with it, the model reaches PSNR 35.34 / LPIPS 0.017 — slightly behind SD-3.5 (36.52 / 0.012) but a usable image encoder [Table 2].
On WebVid at 4×8×8 / 4 channels, VideoVAE+ reports PSNR 30.31 / SSIM 0.868 / LPIPS 0.054, beating Open-Sora-Plan (29.16 / 0.833 / 0.079), Open-Sora (29.38 / 0.828 / 0.124), and CV-VAE (28.68 / 0.815 / 0.107) [Table 1].
On WebVid at 4×8×8 / 16 channels, the cross-modal VAE reaches PSNR 34.50 / SSIM 0.937 / LPIPS 0.032, beating CogVideoX-VAE (32.89 / 0.921 / 0.050), EasyAnimate-VAE (32.12 / 0.909 / 0.041), Cosmos-Tokenizer (31.25 / 0.886 / 0.103), and CV-VAE-16 (32.28 / 0.908 / 0.055) [Table 1].
On the Large-Motion test set at 16 channels, VideoVAE+ reaches PSNR 32.24 / SSIM 0.912 / LPIPS 0.048, the strongest among all baselines [Table 1].
A TemporalConv kernel-size sweep shows that a temporal kernel of 7 with 3×3 spatial extent (i.e. (7,3,3)) outperforms (3,1,1) / (5,1,1) / (7,1,1) / (5,3,3) — wider temporal context and explicit spatial coupling in the temporal conv matter [Table 4].
Replacing the image GAN loss with a 3D video GAN loss adds ~0.1 dB PSNR and slight SSIM/LPIPS gains [Table 4].

Method

The video autoencoder is built in two stages, both inflated from a pretrained 2D image VAE (SD-1.4 for the 4-channel variant, SD-3.5 for the 16-channel variant). Stage 1 is the temporal-aware spatial autoencoder: 2D conv layers from the source VAE are inflated to 3D convs of kernel (1,3,3) (initialized from the 2D weights), with added temporal conv layers of kernel (7,3,3); the middle block inflates 2D attention to 3D + adds a temporal attention. Critically, stage 1 only compresses spatially — the temporal dimension is preserved. Stage 2 is a lightweight temporal autoencoder: 3D ResNet blocks with convolutional temporal downsampling, with an asymmetric decoder design (two 3D ResNet blocks per upsampling layer in the decoder, one per downsampling layer in the encoder) so the decoder gains some “hallucination headroom” over the encoder. The two stages are trained jointly end-to-end.

Cross-modal conditioning is bolted in via patch-based cross-attention: after every ResNet block in both stages, the feature map is patched (sizes 8/4/2/1 across the temporal-aware spatial autoencoder, 1 in the temporal autoencoder), visual patches act as Q/V, Flan-T5 text tokens act as K, LayerNorm is applied, a projection conv is applied to the cross-attention output, and the result is added back via a residual connection.

For joint image-video training the network is fully spatiotemporally factorized: every temporal conv / temporal attention / temporal autoencoder layer can be skipped at the batch level. Image batches bypass everything temporal; video batches forward through everything. 8:2 video:image batch ratio. Losses: pixel reconstruction + LPIPS perceptual + KL regularization (on the innermost latent only, with shape (C, T, H, W)) + 3D video GAN loss (enabled after 50K warmup). The 4-channel model is trained 230K steps + 100K joint-image-video steps; the 16-channel model 310K + 185K steps; cross-modal variants add 160K text-pair steps initialized from the trained weights.

Results

Headline reconstruction numbers across three test sets (PSNR ↑ / SSIM ↑ / LPIPS ↓), all 4×8×8 spatial+temporal compression:

Test set / channels	Best baseline	VideoVAE+ (ours)
WebVid / 4ch	Open-Sora 29.38 / 0.828 / 0.124	30.31 / 0.868 / 0.054
Inter4K / 4ch	Open-Sora 29.27 / 0.843 / 0.132	29.04 / 0.857 / 0.068 (Cross-Modal: 29.04 / 0.851 / 0.068)
Large-Motion / 4ch	Open-Sora 27.76 / 0.803 / 0.154	27.62 / 0.814 / 0.084
WebVid / 16ch	CogVideoX 32.89 / 0.921 / 0.050	34.50 / 0.937 / 0.032 (Cross-Modal)
Inter4K / 16ch	CogVideoX 32.51 / 0.923 / 0.053	33.57 / 0.935 / 0.038 (Cross-Modal)
Large-Motion / 16ch	CogVideoX 31.09 / 0.898 / 0.069	32.24 / 0.912 / 0.048 (Cross-Modal)

Image reconstruction (480×864, 500 UHD-4K frames): the 4-channel joint-trained model edges out SD-1.4 (30.87 / 0.904 / 0.040 vs 30.22 / 0.897 / 0.044); the 16-channel joint-trained model trails SD-3.5 on PSNR (35.34 vs 36.52) but is close on perceptual metrics. Without joint training the 16-channel model is catastrophically broken on images (PSNR 9.26) — joint training is doing real work, not just a regularizer [Table 2].

The cross-modal variant beats the non-cross-modal variant by a fraction of a dB on most settings but does not always win on every metric — its main advantage is detail preservation in scenes where text disambiguates structure (faces, text rendering, edges).

Why it’s interesting

VideoVAE+ is now the third filed paper in the wiki making a focused argument that the video VAE itself is a first-class lever for downstream video diffusion quality, not just a swappable preprocessor. It pairs naturally with PV-VAE: Video Generation with Predictive Latents (PV-VAE), which uses a predictive-reconstruction objective on the same architecture class to bias latents toward motion-awareness, and with Adaptive 1D Video Diffusion Autoencoder (One-DVA), which routes motion into a separate 1D latent. VideoVAE+ is the architectural-design point in this cluster: rather than changing the objective (PV-VAE) or splitting the latent (One-DVA), it argues that the combination of simultaneous and sequential spatiotemporal compression — both already explored in isolation — is what large-motion video reconstruction needs.

The cross-modal-attention-in-the-VAE idea is the more contrarian contribution. It is conceptually adjacent to Unified Latents (UL): How to train your latents (UL), which redesigns the encoder/prior coupling so the latent’s information density can be tuned to the diffusion budget — VideoVAE+ is instead adding information from another modality to a fixed compression. If text conditioning at the VAE layer is real, it weakens the assumption (implicit in TC-AE: Unlocking Token Capacity for Deep Compression Autoencoders and the RAE lineage) that a tokenizer should be modality-pure. The downside is that text-conditioning at decode time imposes a coupling between the VAE and the upstream T2V generator’s prompt distribution, which is a serious constraint for production pipelines like the Wan/LTX tokenizer stacks Luma cares about.

This is also a useful baseline anchor: every “next-gen” video VAE paper filed in 2025–2026 (PV-VAE, TC-AE, UL, the Wan2.x VAEs) is implicitly competing against this 16-channel cross-modal number, even though VideoVAE+ predates most of them by a year. The pairing with TC-AE: Unlocking Token Capacity for Deep Compression Autoencoders is particularly sharp: TC-AE’s staged-token-compression diagnosis (“single-bottleneck token-to-latent compression destroys structure”) is the image analog of VideoVAE+‘s “single-stage spatiotemporal compression destroys motion or detail” — both papers identify a single-stage compression as the failure mode and propose two-stage fixes.