VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization

TL;DR

VideoFlexTok is a video tokenizer that resamples a video’s 3D VAE latents into a 2D token sequence — one axis is time, the other is a coarse-to-fine importance ranking. Nested dropout during encoding plus a DINOv2-feature REPA loss on the decoder force the first tokens of each (latent) frame to capture semantic + motion + geometry information, while later tokens add detail. A rectified-flow DiT decoder reconstructs realistic video from any token count. The payoff: a downstream AR text-to-video model can match a 3D-grid tokenizer’s quality with ~10× less compute, or produce a 10-second 81-frame video from only 672 tokens (8× fewer than a 3D grid baseline at the same resolution).

Key claims

Standard 3D-grid video tokenizers force the downstream generator to predict all low-level details “pixel-by-pixel” irrespective of content complexity, which inflates learning cost [§1, §2].
A nested-dropout encoder with register tokens reorganized along a (time × coarse-to-fine) 2D structure induces a coarse-to-fine hierarchy where the first tokens emergently capture object class, scene geometry, and motion [§3, §4.2, Fig. 1, Fig. 4].
The semantic-information emergence requires a REPA-style DINOv2 distillation loss on an early decoder layer; without it, reconstruction-focused training prioritizes low-level details and the few-token regime collapses [§3.2, §C.1, Fig. 9].
A rectified-flow DiT decoder can produce realistic videos from any number of input tokens, enabling adaptive compactness at inference without retraining [§3, §3.2].
Time-causal attention in both encoder and decoder (preserving the original signal’s causal structure) yields better downstream generative performance than full self-attention, even though full attention gives better raw reconstruction (gFVD 175.1 vs. 211.5 under time-causal vs. full decoder attention) [§4.5, Table 3, §C.2].
2D time-causal register structure beats LARP’s flat-1D registers on downstream generation (gFVD 287.6 vs. 352.1) despite worse reconstruction fidelity — confirming that easier-to-predict tokens beat better-to-reconstruct tokens for AR generative modeling [§4.5, Table 2].
Class-to-video on Kinetics-600 with a 2.2B AR model trained for 164B tokens: VideoFlexTok at 160 tokens/clip matches or beats baselines using 1024–1280 tokens — rFVD 48.7, gFVD 80.0, Cls. Score 0.833 vs. LARP 42.1 / 87.5 / 0.739 and Omnitokenizer 63.6 / 102.6 / 0.858 [§4.5, Table 1].
A 5–10× smaller AR model or 5–10× fewer training tokens achieves comparable or better C2V performance than the 3D-grid baseline, because shorter sequences are easier to model [§4.3, Fig. 6].
On text-to-video (Panda70M subset), under compute-optimal scaling from 1e19 to 5e21 FLOPs (400M to 5.2B params), VideoFlexTok matches the 3D-grid baseline with an order of magnitude less compute and overtakes it in the largest tested regime [§4.3].
Long-video synthesis: training a 3.2B AR model on 10-second 81-frame videos with only 672 tokens (8× fewer than the equivalent 3D-grid tokenizer’s 5376) produces coherent, text-aligned 10-second generations within the same total FLOPs budget as the shorter-video runs [§4.4, Fig. 1, Fig. 8].
Long-video decoding chunks overlap and condition the flow decoder on the last generated frames of the previous chunk, preserving temporal consistency even when reconstructing from few tokens [§3.4].
Probing the first 1–2 tokens by swapping in an edited first frame and conditioning the decoder reveals the first tokens primarily encode motion: a rolling orange becomes a rolling apple while keeping the source motion [§4.2, Fig. 4].

Method

VideoFlexTok is an autoencoder built on top of a frozen VidTok 3D VAE (8×8×4 latent compression, 16 channels). The encoder receives the flattened VAE latents per latent frame interleaved with 256 learnable register tokens (DINO-register style), and applies a time-causal self-attention mask so registers for latent frame t only see frames ≤ t. The output is a (T × N) grid of FSQ-quantized register tokens (codebook 64²) — the time-first dimension preserves causal structure, the per-frame dimension is the coarse-to-fine axis.

The coarse-to-fine ordering is induced by nested dropout (Kusupati-style Matryoshka dropout): during training, a random number of register tokens are masked from the end of each per-frame slice. The encoder is thus forced to pack the most reconstructible information into the earliest tokens. To prevent reconstruction-focused training from filling those early tokens with low-level pixel statistics, a REPA-style auxiliary loss reads features from an early decoder layer and matches them (cosine similarity) to DINOv2 features — biasing early tokens toward semantically meaningful content.

The decoder is a DiT trained with a rectified-flow objective in VAE-latent space. It receives the masked registers interleaved with the noised VAE latents (time-causal attention again) and the REPA-head supervision sits on an early layer. The final objective is L_flow + λ · L_REPA.

Downstream AR generation is a LLaMA-style causal Transformer with cross-attention to T5 text (T2V) or a [SOS] class embedding (C2V). Token order is time-first: predict token 1 across all frames, then token 2 across all frames, etc. — the natural reading order for the coarse-to-fine 2D structure (and the one that gives best performance per §C.3).

Long-video synthesis splits the video into fixed-length chunks with frame overlap. The first chunk decodes normally; subsequent chunks condition the flow decoder on the last decoded frames of the previous chunk, providing the “fill-in” detail needed when only 32 tokens/frame are generated by the AR model.

Results

Class-to-video, Kinetics-600, 2.2B AR model / 164B tokens, 160 tokens used at inference: rFVD 48.7, gFVD 80.0, Cls. Score 0.833 vs. LARP at 1024 tokens (42.1 / 87.5 / 0.739) and Omnitokenizer at 1280 tokens (63.6 / 102.6 / 0.858) [Table 1].
C2V efficiency: at matched gFVD/Cls. Score, VideoFlexTok needs 5–10× less compute than the 3D-grid tokenizer baseline across both axes (model size and training tokens) [Fig. 6].
T2V scaling, Panda70M subset: under compute-optimal scaling from 1e19→5e21 FLOPs, VideoFlexTok matches the 3D-grid baseline an order of magnitude earlier and overtakes it at the largest budget [§4.3].
Long video: 10-second 81-frame T2V with 672 tokens/video on a 3.2B model trained for 55B tokens (≈mid-range of the scaling sweep). 3D-grid would need 5376 tokens for the same window [§4.4].
Ablations on decoder attention: time-causal beats full attention on gFVD (175.1 vs. 211.5) despite worse reconstruction (rFVD 80.9 vs. 58.3) [Table 3].
Ablations on register structure: 2D time-causal beats flat 1D on gFVD (287.6 vs. 352.1), reverse on rFVD (69.9 vs. 48.9) [Table 2].
REPA loss is required for the few-token regime: removing it severely degrades both fidelity and alignment scores when only the first few tokens are used [Fig. 9, §C.1].

Why it’s interesting

VideoFlexTok plants a different flag in the long-AR-video Pareto than any of the Context Length / Quality Trade-off in Video Generation papers filed so far: it compresses at the tokenizer, not at the KV cache or memory encoder. Pretraining Frame Preservation in Autoregressive Video Memory Compression (PFP) bypasses the VAE channel bottleneck with a learned residual memory encoder; Context Forcing: Consistent Autoregressive Video Generation with Long Context (Context Forcing) partitions the cache functionally; PackForcing: Short Video Training Suffices for Long Video Sampling and Long Context Inference (PackForcing) compresses a mid-buffer with a learned dual-branch module. VideoFlexTok says: instead of doing more clever things downstream of a fixed-budget 3D grid, redesign the grid into a flexible-length 2D structure and let the downstream model choose its budget. The 8× token reduction for 10s video is a stronger compression than any of the cache schemes get, but at the cost of needing to train a new tokenizer (and a new AR model that respects the time-first generation order).

It also extends the FlexTok line (Apple/EPFL, image-only) to video and validates a specific claim that’s been implicit since Don't Look Twice: Faster Video Transformers with Run-Length Tokenization (RLT) and PyramidalWan: On Making Pretrained Video Model Pyramidal for Efficient Inference (PyramidalWan): raw video tokens are heavily redundant, and content-adaptive compression is leaving compute on the table for the standard fixed-grid recipe. Unlike RLT (a pre-network discriminative-side trick) or PyramidalWan (a noise-level vs resolution coupling on a pretrained generative model), VideoFlexTok pushes the adaptivity into the tokenizer itself.

The REPA dependence is consistent with the broader “internal-vs-external representation guidance” story on Diffusion training efficiency: VideoFlexTok uses external DINOv2 supervision via REPA, while No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselves (SRA) (SRA) and Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis (Self-Flow) argue you can get the same effect from internal self-distillation. An obvious follow-up experiment: does VideoFlexTok still get its few-token semantic emergence if REPA is replaced with an SRA/Self-Flow-style internal teacher?