Skip to content
Bud's Wiki

VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders

TL;DR

Section titled “TL;DR”

VideoRAE ports the Representation Autoencoder recipe from images to video by using a frozen Video Foundation Model (V-JEPA 2, VideoMAEv2) as the encoder and pairing it with a lightweight 1D self-attention projector to compress multi-scale hierarchical features into a compact latent. It supports both continuous latents (for Diffusion Transformers) and discrete tokens (for autoregressive models) via multi-codebook high-dimensional quantization, and adds a local-and-global representation-alignment objective to the decoder so training does not need KL regularization. On UCF-101 class-to-video, it reports SOTA gFVDs of 40 (AR) and 93 (DiT), while converging roughly 5× faster than competing autoencoder baselines; a controlled 2B-scale text-to-video study shows replacing LTX-VAE with VideoRAE speeds convergence at comparable settings.

Key claims

Section titled “Key claims”
  • Frozen Video Foundation Model encoders (V-JEPA 2, VideoMAEv2) can be turned into compact, reconstruction-capable, generation-friendly video latents when paired with a lightweight 1D self-attention projector that consumes multi-scale hierarchical features [Abstract, §1].
  • The same VideoRAE latent supports two decoding regimes with a single encoder: continuous latents for Diffusion Transformers, and discrete tokens for autoregressive models via multi-codebook high-dimensional quantization [Abstract].
  • A local-and-global representation-alignment objective against the frozen VFM teacher during decoding preserves semantics and removes the need for KL regularization on the latent [Abstract].
  • On UCF-101 class-to-video, VideoRAE achieves state-of-the-art gFVD of 40 with an AR generator and 93 with a DiT generator [Abstract].
  • VideoRAE converges roughly 5× faster than competing autoencoder baselines under the same generator settings [Abstract].
  • In a controlled 2B-parameter text-to-video study, swapping LTX-VAE for VideoRAE yields faster convergence at comparable settings [Abstract].

Method

Section titled “Method”

VideoRAE is a video representation autoencoder. The encoder side is a frozen Video Foundation Model — V-JEPA 2 or VideoMAEv2 — whose multi-scale hierarchical features are read out from several layers and then compressed by a small learned 1D self-attention projector into a compact latent tensor. Two heads sit on top: a continuous latent (for DiT-style generators) and a discrete-token latent produced by multi-codebook high-dimensional quantization (for autoregressive video generators). The decoder is trained under a local-and-global representation-alignment loss that supervises the reconstructed features against the frozen VFM’s features at both patch and global scales, which the authors report is sufficient to omit KL regularization from the training objective. Downstream, class-to-video generators are trained on UCF-101 (AR and DiT variants) and a 2B-parameter text-to-video model is trained under a matched setting against an LTX-VAE baseline.

Results

Section titled “Results”

Headline numbers, class-to-video on UCF-101: gFVD 40 with an autoregressive generator and gFVD 93 with a DiT generator, both reported as state of the art at filing [Abstract]. Convergence: roughly 5× faster than competing autoencoder baselines under matched generator settings [Abstract]. Text-to-video: at 2B backbone scale, replacing LTX-VAE with VideoRAE improves convergence speed under comparable training settings [Abstract]. Reconstruction is reported to be strong in both the continuous and discrete regimes [Abstract]. Section-level details and full ablations are not available from the abstract-only fetch; concrete downstream deltas beyond convergence-speed and UCF-101 gFVD will be worth revisiting once the paper HTML/PDF is parsed.

Why it’s interesting

Section titled “Why it’s interesting”

This is the first entry in the wiki’s representation-autoencoder line that operates on video, which was one of the concept page’s explicit open questions — “where does the ‘compress vs preserve’ tradeoff land at video scale?” (Representation Autoencoders). It sits closest to One Layer Is Enough: Adapting Pretrained Visual Encoders for Image Generation (FAE) on the design axis: instead of keeping the encoder’s high-dimensional features and modifying the generator (the Diffusion Transformers with Representation Autoencoders recipe), VideoRAE compresses VFM features through a minimal projector — here a 1D self-attention block — so standard DiT and AR backbones can be plugged in unchanged. It also composes with the JEPA-line evidence: V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning argues V-JEPA-family encoders now produce dense, temporally-consistent features suitable for downstream tasks, and Back to the Features: DINO as a Foundation for Video World Models (DINO-world) already showed that predicting in a frozen video encoder’s latent space is a viable world-modeling substrate; VideoRAE is the corresponding generative decoder side of that thesis, closing the loop between latent-predictive world models and pixel-level video generation. The dual-head design (continuous + multi-codebook discrete) also sidesteps a live open question in Autoregressive Video Generation — whether AR and DiT video generators need different tokenizers — by demonstrating a single frozen encoder + projector serving both.

See also

Section titled “See also”