Adaptive 1D Video Diffusion Autoencoder

TL;DR

One-DVA is a video autoencoder built from a query-based ViT encoder and a pixel-space diffusion-transformer decoder. The encoder produces a hybrid latent — a 2D “structural” latent sampled from spatiotemporal ViT features plus a 1D query-derived latent whose length is dynamically truncated per-video by a motion-score-driven dropout. At standard compression ratios it matches HunyuanVideo / Wan2.x 3D-CNN VAEs on reconstruction metrics; the diffusion decoder buys back quality at aggressive compression and allows the latent length to be chosen per clip. The work is interesting to Luma because it threads three trends at once — 1D variable-length video tokenization, a generative (diffusion) decoder, and a hybrid 1D+2D latent that’s still tractable for a downstream LDM.

Key claims

One-DVA at a standard compression ratio (latent length 1024, channel dim 64) achieves the best PSNR (36.48) and SSIM (0.95) among CogVideoX / HunyuanVideo / Wan2.x / Magi1 on the Open-Sora Plan evaluation set, and the second-lowest rFVD (56.96) [Table 1].
A motion-score-driven scoring mechanism that picks 1D latent length per video beats a constant-length baseline at matched total token usage (rFVD 70.28 vs 72.42) [Table 1, “Avg 1D” vs “Con 1D”].
Structural-latent-only reconstruction (no 1D latents) still works but degrades sharply: rFVD 149.97, PSNR 32.80, LPIPS 0.057 — i.e. the 2D structural latent carries low-frequency content and the 1D queries carry detail/motion [Table 1, last row].
Two-stage training (deterministic pretraining of the encoder with pure-noise decoder input, then stochastic diffusion post-training with variable-length dropout) strictly beats end-to-end joint training at matched 217K iters: rFVD 115.64 vs 230.06, PSNR 34.20 vs 31.03 [Table 3].
Diffusion-based decoder training improves reconstruction over deterministic one-step decoding once given enough post-training: rFVD 67.36 → 65.19 and PSNR 35.85 → 36.26 over an extra 85K iters [Table 2]; the gain from more sampling steps is largest when the 1D condition is short (i.e. when the decoder has to do more generation) [Fig. 3].
For downstream LDM use, two additional steps are needed: a self-alignment regularizer that pulls each 1D latent toward its nearest structural-latent vector (so the 1D space inherits ViT locality priors), and decoder fine-tuning on LDM-sampled latents to suppress patch-like artifacts caused by train/inference distribution drift [§3.5].
On class-conditional 17×256×256 video generation, the full pipeline (One-DVA + LDM + decoder fine-tuning) reaches gFVD 210.9, matching Hi-VAE; omitting decoder fine-tuning degrades gFVD noticeably; LDM on structural-only latents is reasonable but capped by the structural-only reconstruction ceiling [§4.4, Table 4].

Method

The encoder is a ViT: patchify the input video into spatiotemporal embeddings, concatenate a fixed pool of learnable 1D queries and a few resolution/aspect special tokens, then run a stack of transformer blocks. After channel compression, the autoencoder emits two latent components — (a) a 2D structural latent obtained by spatially downsampling the channel-compressed spatiotemporal feature grid (a (T/4, H/16, W/16) tensor in the paper’s setup, mirroring advanced 3D-CNN VAE shapes) and (b) a 1D latent formed by selecting the first M query outputs. A variable-length dropout module then trims the tail of the 1D sequence; the truncation ratio is sampled from a distribution whose mean is set by a per-video motion score (pixel-difference based), in the spirit of matryoshka representation learning.

The decoder is a pixel-space Diffusion Transformer (DiT). The latent (structural + truncated 1D, with dropped positions filled by padding tokens) is concatenated with the noisy video and fed through transformer blocks; an unpatchifier with a long skip connection, linear projection, pixel-shuffle, and a final conv reconstructs RGB. Training uses flow-matching loss + LPIPS perceptual loss + KL regularization + REPA. Crucially, the recipe is two-stage: stage 1 uses pure noise as the decoder input (no diffusion timesteps) so the encoder is forced to carry all reconstruction-relevant info; stage 2 unlocks stochastic timesteps and variable-length dropout. To make the latent space usable for LDM training, the authors then (i) add a self-alignment loss pulling each 1D latent toward its top-1 structural-latent neighbor (injecting locality prior into queries) and (ii) fine-tune the decoder on LDM-predicted latents to bridge the train/inference gap.

Results

Reconstruction at standard compression on 17×256×256 video clips: rFVD 56.96 (second only to HunyuanVideo’s 51.47), PSNR 36.48 (best), SSIM 0.95, LPIPS 0.025 vs CogVideoX 68.17 / 34.97 / 0.94 / 0.033, Wan2.2 60.18 / 35.23 / 0.94 / 0.023, Magi1 70.07 / 36.25 / 0.95 / 0.035 [Table 1].
Variable-length: with motion-score-estimated 1D length (avg), rFVD 70.28 / PSNR 35.42 vs constant-length at same token count rFVD 72.42 / 35.40 — scoring helps [Table 1].
Two-stage > end-to-end at 217K iters: rFVD 115.64 vs 230.06 [Table 3].
Diffusion post-training over deterministic stage-1: rFVD 67.56 → 65.19 with +85K iters of diffusion vs 67.56 → 67.36 with +85K iters of more deterministic training [Table 2].
Generation: gFVD 210.9 on class-conditional UCF-style benchmark, matching Hi-VAE; decoder fine-tuning is the load-bearing step for closing the train/inference gap [§4.4, Table 4].
Training cost (for context): 415K iters at stage 1 ≈ 12 days on 32 × 80GB GPUs at 1.0B params, then ~800K more iters in stage 2; FSDP throughout [§4.1].

Why it’s interesting

This sits at the intersection of three things the team has been tracking: pixel-space diffusion as the decoder side of a generative pipeline, content-adaptive video token counts, and hybrid 1D+2D latents that don’t force a downstream LDM to fight an unstructured query space. The structural+1D split is a pragmatic answer to the worry that pure 1D query latents are too unstructured for diffusion learning — the alignment loss explicitly grafts ViT-style locality onto the 1D side. The decoder fine-tuning step is also worth noting: it’s a concrete instance of the “fix the train/inference distribution drift by training on the sampler’s own outputs” pattern that’s been popping up elsewhere (e.g. consistency-style self-distillation). Compared to the team’s filed Pixel-space diffusion work — PixelGen (perceptual losses on a pixel-native DiT) and L2P (transfer-learning an LDM into pixel space) — One-DVA uses pixel-space diffusion only at the decoder, leaving the LDM in latent space; it’s a different point in the design space and a useful data point on whether pixel-space diffusion needs to be end-to-end to win.