Skip to content
Bud's Wiki

Representation Autoencoders

TL;DR

Section titled “TL;DR”

Representation Autoencoders (RAE) is the class of latent-diffusion designs that replace a from-scratch VAE encoder with a frozen pretrained self-supervised visual encoder (DINOv2, SigLIP-2, MAE) and pair it with a learned decoder. The original recipe (RAE, Oct 2025) operates the diffusion model directly on the encoder’s high-dimensional features and pays for it with architectural changes — wider DiT, an auxiliary DDT head, dimension-aware noise schedule, noise-augmented decoder. Subsequent work has forked into three engineering positions: scale the same recipe to T2I and drop the optional pieces (RAE T2I scaling); push the recipe further with multi-layer aggregation + REPA-complementarity for 10× speedup (RAEv2); or reject the high-dimensional latent and re-compress it back to SD-VAE shape with a minimal single-attention encoder (FAE). The shared thesis: diffusion is faster and better when it doesn’t have to relearn visual structure that DINO already knows.

Key claims

Section titled “Key claims”
  • Frozen self-supervised encoders (DINOv2, SigLIP-2, MAE) paired with a trained ViT decoder beat SD-VAE on reconstruction rFID at ImageNet-256 — MAE-B 0.16, DINOv2-B 0.49, SigLIP2-B 0.53 vs SD-VAE 0.62 (Diffusion Transformers with Representation Autoencoders Table 1a).
  • High-dimensional pretrained latents do not work out of the box for diffusion. Standard DiT-S on RAE blows up to gFID 215.76 vs 51.74 on SD-VAE, and DiT-XL underperforms SD-VAE 23.08 vs 7.13 without the load-bearing fixes (Diffusion Transformers with Representation Autoencoders Table 2).
  • The original RAE recipe required four ingredients to converge on the high-dimensional latent: (i) DiT hidden dimension ≥ latent token dimension, (ii) dimension-dependent noise-schedule shift, (iii) noise-augmented decoder training, (iv) shallow-wide DDT head for FLOP-efficient scaling (Diffusion Transformers with Representation Autoencoders §4-5).
  • Two of those four ingredients drop out at T2I scale — the wide DDT head’s advantage saturates at 2.4B+ backbones where width already exceeds the latent dim, and noise-augmented decoding becomes negligible after ~15k steps. Only the dimension-aware noise shift remains essential at scale (Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders §3.2-3.3, Fig. 4).
  • RAE consistently beats VAE across DiT scales 0.5B → 9.8B and LLM scales 1.5B → 7B for T2I, with the gap widening with model size up to ~6B DiT params (Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders Fig. 5).
  • RAE and REPA are complementary, not competitive. RAE uses the pretrained representation as the encoder; REPA distills the same representation to intermediate decoder layers. Combining both improves performance and simplifies guidance by reusing the REPA head — eliminating the need for separate guidance models (AutoGuidance) or extra forward passes (CFG) (Improved Baselines with Representation Autoencoders §2.1).
  • Multi-layer encoder aggregation (sum the last k DINO layers rather than just the final layer) further unlocks RAE’s power: with multi-layer aggregation stronger encoders like DINOv3-L improve generation, whereas they performed worse under the original RAE’s final-layer-only recipe (Improved Baselines with Representation Autoencoders §2.1).
  • FAE inverts the engineering tradeoff: instead of keeping the encoder’s high-dimensional features and modifying the generator, FAE compresses pretrained embeddings down to a low-dimensional latent (16×16×32, SD-VAE-shape) using a deliberately minimal single self-attention layer + linear projection encoder, and keeps the SiT/LightningDiT backbone unchanged (One Layer Is Enough: Adapting Pretrained Visual Encoders for Image Generation §4.1, §4.3).
  • The compression-side recipe (FAE) reaches FID 1.29 with CFG on a 675M backbone at ImageNet-256, competitive with the original RAE’s 839M DiTDH-XL (1.13), supporting the position that RAE’s architectural exotica is substitutable by latent-side engineering (One Layer Is Enough: Adapting Pretrained Visual Encoders for Image Generation Table 1).
  • Deeper encoders hurt in the FAE/compression regime: a deep Transformer encoder overfits the (easier) feature-reconstruction objective and discards information not directly supervised, while a pure linear projection cannot de-redundantize patch-wise global signal. A single attention layer is the sweet spot (One Layer Is Enough: Adapting Pretrained Visual Encoders for Image Generation §4.1, Table 5).
  • RAE-family latents preserve the encoder’s discriminative properties enough to share one latent space for understanding and generation. RAE T2I keeps SigLIP-2 features usable for understanding-task heads; FAE retains 86.17% ImageNet linear probe vs DINOv2-g’s 87.00% (Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders §5, Tab. 6; One Layer Is Enough: Adapting Pretrained Visual Encoders for Image Generation §4.4).
  • Generation and reconstruction quality decouple under the original RAE recipe: MAE-B gives the best rFID (0.16) but DINOv2-B gives the strongest generation, so DINOv2-B is the default choice (Diffusion Transformers with Representation Autoencoders §4 preamble).
  • Latent test-time scaling becomes natural when generation and understanding share one encoder: an LLM can verify its own generated latents without a pixel decode/re-encode round-trip. Answer-Logits best-of-32 lifts a 7B LLM + 5.5B DiT from GenEval 55.5 → 67.8 (Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders §5, Tab. 5).
  • From-scratch joint pre-training is a third design point. VTP trains a ViT autoencoder from scratch under CLIP + DINO-style SSL + reconstruction + (stage-2 GAN) and shows the tokenizer’s gFID scales 65.8% over a 10× compute sweep, while a matched reconstruction-only AE plateaus immediately (Towards Scalable Pre-training of Visual Tokenizers for Generation §4.3-4.5). Generation also scales with tokenizer parameters (20M → 300M: gFID 31.28 → 26.12) and tokenizer data (100K → 100M: gFID 47.59 → 27.45) with downstream DiT FLOPs held constant.
  • VTP-L (78.2% zero-shot, 85.7% LP, 0.36 rFID, gFID 2.81 @ 80ep no-CFG with LightningDiT-B) beats VA-VAE-d32 (gFID 4.29) and RAE-d768 (gFID 4.28, under the same generation model) at matched downstream training budget — and converges 4.1× faster than VA-VAE (Towards Scalable Pre-training of Visual Tokenizers for Generation Table 2, Fig. 9). Conflict: the 800ep+AutoGuidance RAE result (gFID 1.13) is not directly compared, so the “from-scratch beats frozen-encoder” claim at the absolute ceiling remains open.
  • A single pixel-diffusion decoder works for both VAE and semantic (SigLIP / DINOv2) latents. PiD’s pixel-space diffusion decoder accepts a conventional VAE latent or a frozen-encoder semantic latent as a sigma-aware conditioning signal, and produces 4× / 8× upscaled images in ~210 ms on GB200, ~1 s on RTX 5090 (PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion §3, Abstract). This contests the implicit assumption (RAE, RAE T2I, RAEv2, FAE) that an RAE-style pipeline needs a bespoke learned decoder co-designed with the encoder; for inference at megapixel scale, a generic pixel-diffusion decoder is now an alternative.

Recent contributions

Section titled “Recent contributions”
  • [2026-07-20] Flow Matching in Feature Space for Stochastic World Modeling: FlowWM (FAIR + Inria) is the first RAE-family entry that targets stochastic world modeling rather than image or video generation — flow matching directly in frozen DINOv3 features for the perception-side task of forecasting future features for downstream detection/depth. Confirms two RAE ingredients transfer outside image generation: (i) shallow-wide projection head with d_head ≫ d_latent is load-bearing (depth ablation saturates at 2 layers, width kills performance if too narrow), (ii) dimension-aware noise-schedule shift (SD3-style) is essential when scaling either latent dimension or prediction horizon. Adds a new efficiency primitive the RAE line has not previously needed: a differentiable one-step endpoint projection ẑ_1 ≈ z_t + (1 − t)·v_θ that lets temporal-consistency and task-driven losses supervise the sample endpoint without ODE-through-time backprop — a technique that generalizes to any flow-matching model needing downstream-task supervision.
  • [2026-07-16] VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders: VideoRAE (Xie/Wu/Hu/Huang/Jiang) is the first RAE-family entry that operates on video, directly addressing the concept’s open “compress vs preserve at video scale” question. It sits on the FAE side of the design axis — frozen V-JEPA 2 / VideoMAEv2 encoder + a lightweight 1D self-attention projector that compresses multi-scale hierarchical features into a compact latent — and adds a dual-head design (continuous latent for DiT + multi-codebook high-dim quantization for AR) that lets a single encoder serve both video-generation paradigms. Local-and-global feature-alignment loss against the frozen VFM teacher removes KL regularization from training. Reports SOTA UCF-101 class-to-video gFVDs of 40 (AR) and 93 (DiT) with ~5× faster convergence than baselines, and faster convergence than LTX-VAE at 2B T2V scale.
  • [2026-07-10] LingBot-VA 2.0: Native Video-Action Pretraining for Generalizable Robot Control: Extends the RAE-family recipe (frozen foundation-model encoder + trained decoder + latent aligned to foundation-model features) into video-plus-action pretraining via the RepWAM instantiation: a ViT autoencoder with full spatial attention per frame and causal temporal attention across frames is aligned to a frozen Perception Encoder via temporally-pooled MSE loss, and a second self-supervised objective learns latent-action tokens via an inverse-dynamics + forward-dynamics pair with a bottleneck. Concrete datapoint that RAE-style semantic-alignment losses transfer beyond image generation into embodied-AI pretraining substrates, and that the “align to a frozen foundation model” recipe composes cleanly with an action-supervision head on the same latent space.
  • [2026-07-04] From SRA to Self-Flow: Attention Separation Reveals the Real Mechanism Is Data Augmentation: Revisits Self-Flow’s dual-timestep-scheduler explanation: blocking cross-noise attention doesn’t hurt (and can help), so the improvement over SRA is data augmentation rather than cross-noise reasoning — a cheaper mechanism than Self-Flow’s EMA-teacher recipe.
  • [2026-06-24] Krea 2 Technical Report: Krea 2 benchmarks FLUX.1-dev VAE, Qwen Image VAE, DC-AE, FLUX 2 VAE, and an internal REPA-E-style autoencoder (DINOv3 semantic alignment + light diffusion loss) for a production T2I run. Reports DC-AE’s 32× spatial compression imposes a hard upper bound on fine-detail generation from reconstruction error; the internal REPA-E-style AE is competitive with Qwen Image but Krea ships the validated-at-scale Qwen Image VAE and FLUX 2 VAE for production.
  • [2026-05-25] PixelDiT: Pixel Diffusion Transformers for Image Generation: PixelDiT is the explicit encoder-free counterpoint to the RAE / RAE-XL direction: same NVIDIA team that produced PiD, trained end-to-end without any autoencoder, hits gFID 1.61 on ImageNet 256² vs RAE-XL’s 1.13. The ~0.5 FID gap (in RAE’s favor) is the current empirical price of keeping an encoder out of the loop entirely on this benchmark. The frame here is different from RAE: RAE freezes a pretrained DINOv2 encoder and keeps a latent space; PixelDiT removes the encoder side completely and uses DINOv2 only as a REPA-style alignment target for mid-patch-pathway tokens. Both approaches land near each other on the FID-vs-compute Pareto, suggesting the DINOv2 inductive bias matters more than whether it’s applied as an encoder or as a feature-alignment objective.
  • [2026-05-25] PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion (PiD, NVIDIA — Lu/Wu/Wu/Wang/Ling/Fidler/Ren): contests an implicit RAE-cluster assumption — that each RAE variant needs a bespoke decoder co-trained with its frozen encoder. PiD’s pixel-diffusion decoder is encoder-agnostic: it works on classic VAE latents and on SigLIP/DINOv2 semantic latents with the same backbone, exposing the latent only through a lightweight sigma-aware adapter. Adds a fourth design point alongside RAE (frozen high-dim + bespoke wide decoder), FAE (frozen + compress + standard decoder), and VTP (from-scratch joint pre-training): “frozen encoder + generic pixel-diffusion decoder”, with the upstream LDM optionally exiting early because the decoder can run on partially-denoised latents.
  • [2026-05-25] Towards Scalable Pre-training of Visual Tokenizers for Generation (VTP, MiniMax — Yao/Song/Zhou/Wang): from-scratch joint pre-training of a ViT autoencoder with CLIP + SSL (DINOv2-style) + reconstruction + decoder-only GAN fine-tune. First visual tokenizer with documented scaling laws for generation: gFID +65.8% over 10× compute, and clear scaling with parameters and data while a reconstruction-only baseline plateaus. d=64 latent stays SD-VAE-shape-compatible with LightningDiT-B. 4.1× faster downstream DiT convergence than VA-VAE. Third major design point in the cluster alongside RAE (frozen high-dim) and FAE (frozen + compress).
  • [2026-05-25] One Layer Is Enough: Adapting Pretrained Visual Encoders for Image Generation (FAE, Apple — Gao, Chen, Chen, Gu): compresses DINOv2/SigLIP features to a low-dim SD-VAE-shaped latent via a single self-attention layer, decouples feature reconstruction from pixel synthesis with a double-decoder. SiT/LightningDiT and STARFlow plug in unchanged. FID 1.29 with CFG / 1.48 no-CFG at 800ep on ImageNet-256, 1.70 / 2.08 at 80ep. Inverts the engineering tradeoff against RAE: instead of modifying the generator to fit the encoder’s latent, modify the latent to fit the generator.
  • [2026-05-22] Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders (RAE T2I scaling, Tong/Zheng/Wang et al.): same group, scales the RAE recipe from class-conditional ImageNet to T2I with SigLIP-2 + 0.5B–9.8B DiTs. 4–4.6× speedup over FLUX-VAE at 1.5B LLM + 2.4B DiT. Dim-aware noise schedule stays essential; wide DDT head and noise-augmented decoder become optional. Demonstrates latent test-time scaling using the shared latent space.
  • [2026-05-22] Improved Baselines with Representation Autoencoders (RAEv2, Singh/Zheng/Wu/Zhang/Shechtman/Xie): improves the original RAE recipe via (a) multi-layer encoder aggregation (sum last k layers), (b) treating RAE and REPA as complementary and combining them. >10× convergence speedup; SOTA gFID 1.06 on ImageNet-256 in 80 epochs (vs 800 for original RAE).
  • 2025-10-13 Diffusion Transformers with Representation Autoencoders (RAE, Zheng/Ma/Tong/Xie — the original): foundational paper. Frozen DINOv2/SigLIP-2/MAE encoder + trained ViT decoder, DiT operates on encoder features. Establishes the four architectural ingredients required to make high-dim latents work. SOTA at filing: gFID 1.13 with AutoGuidance on ImageNet-256 / 512 (DiTDH-XL, 839M, 800ep).

Open questions

Section titled “Open questions”
  • High-dim vs low-dim latent — which engineering bet wins? RAE+DiTDH-XL (839M, high-dim) hits FID 1.13 w/ CFG; FAE+SiT-XL (675M, low-dim) hits 1.29 w/ CFG. RAE wins on FID but pays in backbone size and architectural exotica; FAE is shape-compatible with existing pipelines. The crossover regime (model scale × dataset × modality) hasn’t been mapped.
  • From-scratch joint pre-training vs frozen-encoder + decoder retraining — which wins at the ceiling? VTP beats both at 80ep matched budget but has not been compared at RAE’s 800ep + AutoGuidance regime. The convergence-speed argument favors VTP for compute-constrained runs; the absolute-FID argument is unresolved.
  • Does multi-layer aggregation (RAEv2) help FAE or VTP? Both papers point at the same observation — final-layer-only is sub-optimal — but FAE only uses the final layer of DINOv2/SigLIP and VTP’s latent has no “layers” to aggregate over (it’s bottleneck-d64). The straightforward ablation hasn’t been run.
  • Where does the “compress vs preserve” tradeoff land at video scale? All five filed papers are images; video adds temporal redundancy that may make the high-dim path more wasteful and the low-dim path more attractive — or may need a different recipe entirely.
  • Self-Flow’s challenge. Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis argues external-DINO supervision (RAE, REPA, P-DINO) is a transitional crutch: REPA degrades with stronger DINO encoders past 625M params, while internal self-distillation (Self-Flow / SRA) recovers expected scaling laws and works across image / video / audio. RAEv2 reports the opposite result on the encoder front — stronger encoders (DINOv3) improve RAEv2 generation — but the experiments aren’t apples-to-apples on backbone scale. VTP adds a new wrinkle: it does its own SSL during tokenizer pre-training rather than distilling from a frozen teacher, which is a different way of avoiding the external-DINO bottleneck. The fork remains open.
  • iREPA’s spatial-structure thesis applied to VTP’s latent. What Matters for Representation Alignment: Global Information or Spatial Structure? shows linear-probe accuracy is not what makes representations useful for generation — spatial structure is. VTP’s success metric is linear-probe accuracy + reconstruction. Whether VTP’s d=64 bottleneck actually preserves spatial structure (vs collapsing it into a CLS-token-like global summary) is the relevant audit.
  • Encoder-agnostic decoder. FAE’s pixel decoder is reusable across encoder choices because it lives entirely in embedding space. Does this property hold for RAE / RAEv2 / VTP if their decoders are retrained with embedding-space inputs only, or does the wide-DiT requirement of the high-dim recipe entangle the decoder with the generator? PiD answers this empirically on the inference side: a generic pixel-diffusion decoder works for both VAE and SigLIP/DINOv2 latents. Whether end-to-end generation quality matches a bespoke RAE/FAE decoder at parity compute is not measured.
  • Latent TTS for video. RAE T2I demonstrates LLM-as-verifier on generated latents without pixel decoding for images. The analog for video would lift autoregressive video generation efficiency substantially — but the shared-latent property has to be preserved under temporal compression.
  • Is rFID predictive at all in this regime? Across the five papers, lower rFID does not consistently predict lower gFID — MAE has the best rFID under RAE but DINOv2 wins generation; SigLIP-2 wins T2I; FAE’s latent has higher rFID than FLUX-VAE on YFCC/Text but yields better generation; VTP shows rFID improving while gFID worsens in the reconstruction-only baseline. The rFID-gFID relationship in RAE-family latents is non-monotone and not understood.

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-03

Section titled “2026-03”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”