Skip to content
Bud's Wiki

SemanTok: Predictable Semantic Tokens for Efficient Autoregressive Video Generation

TL;DR

Section titled “TL;DR”

SemanTok is a flexible-length, coarse-to-fine video tokenizer that directly targets the weakness VideoFlexTok’s REPA auxiliary loss leaves on the table: the REPA loss sits on an early decoder hidden state, which the decoder can partially satisfy by passing information from its noised input rather than from the retained token prefix. SemanTok instead feeds frozen DINO features into the encoder and adds lightweight heads that reconstruct those DINO features from each retained token prefix alone — forcing the first few tokens to carry high semantic content independent of the noisy decoder input. A 201M SemanTok AR generator reportedly matches or beats a VideoFlexTok AR generator 3.4× its size at every tested scale; semantic alignment is preserved on OOD classes and holds at every noise level including pure noise; short prefixes are both cheaper to predict and give better generation fidelity.

Key claims

Section titled “Key claims”
  • VideoFlexTok’s REPA-on-early-decoder-layer objective is partially satisfiable from the noised decoder input rather than from the retained prefix — the token prefix therefore carries less semantic content than the loss would suggest [Abstract].
  • Feeding frozen DINO features into the encoder (rather than only using them as a REPA target on the decoder) and adding lightweight heads that reconstruct them from each retained token prefix alone removes this leakage path [Abstract].
  • The resulting tokenizer sustains high semantic alignment at every retained-prefix length, not just at full-length reconstruction [Abstract].
  • Downstream AR generation benefits scale with model size: a 201M SemanTok-AR model matches or beats a 3.4×-larger VideoFlexTok-AR model, and larger SemanTok-AR models further improve fidelity [Abstract].
  • Semantic alignment is preserved on out-of-distribution classes [Abstract].
  • The decoder reaches higher semantic alignment at every noise level — including pure noise — than the VideoFlexTok-style baseline, consistent with the “no leakage from decoder input” prediction [Abstract].
  • Short token prefixes are both cheaper to AR-predict and give better generation fidelity than long prefixes under this training recipe — the coarse-to-fine Pareto that VideoFlexTok established is sharpened: pixel detail is deferred to later tokens without hurting generation quality in the few-token regime [Abstract].

Method

Section titled “Method”

SemanTok sits in the same architectural family as VideoFlexTok (VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization): a flexible-length coarse-to-fine video tokenizer feeding a downstream AR diffusion video model. The two differences relative to VideoFlexTok are on the semantic-supervision side.

First, instead of only using DINOv2 as a REPA target on an early decoder layer, SemanTok feeds frozen DINO features into the encoder alongside the standard latent inputs. This makes DINO’s semantic representation an input signal to the token-production path, not only a downstream supervision target.

Second, SemanTok adds lightweight prediction heads that reconstruct the DINO features from each retained token prefix alone — meaning for every valid prefix length k, there is a head that must recover the DINO features from the first k tokens with no access to the (noised) decoder input or to later tokens. This is the key design change: VideoFlexTok’s REPA sat on the decoder, where the noised input could leak information and satisfy the loss without forcing the early tokens to carry semantics. SemanTok’s prefix-only reconstruction heads close that leakage path.

The downstream AR diffusion decoder remains flow-matching-based in the VideoFlexTok tradition and is reconstructed from token prefixes of arbitrary length; the training recipe produces a predictable coarse-to-fine ordering — the first few tokens are the most reconstruction-expensive but also the most semantically concentrated and most compressible under the AR model.

(The paper is single-version at filing; §-level anchors beyond the Abstract are not yet available in the fetched text.)

Results

Section titled “Results”
  • Class-conditional / AR video generation: 201M SemanTok-AR matches or beats a VideoFlexTok-AR model 3.4× its size (Abstract, headline claim).
  • Scaling: larger SemanTok-AR models further improve fidelity (Abstract).
  • OOD generalization: semantic alignment of the token prefix holds on classes not seen in training (Abstract).
  • Noise robustness: at every noise level — including pure noise — the SemanTok decoder reaches higher semantic alignment than the baseline (Abstract).
  • Short-prefix regime: short retained prefixes are cheaper to AR-predict and give better generation fidelity (Abstract).

(No section-anchored numeric tables are available in the v1 abstract snapshot; headline comparisons are to VideoFlexTok on matched settings.)

Why it’s interesting

Section titled “Why it’s interesting”

This is a precise mechanistic sharpening of VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization: it accepts VideoFlexTok’s framing (flexible-length, coarse-to-fine video tokens; REPA supervision from DINO to induce semantic content in the early tokens) and identifies the specific optimization failure in VideoFlexTok’s recipe — REPA on an early decoder layer is partially satisfiable from the decoder’s noised input. The fix (DINO features into the encoder + reconstruct DINO from each retained prefix alone) is the kind of leakage-closing change that would be hard to discover without careful ablation. If the headline 3.4× efficiency claim replicates, this becomes the preferred flexible video tokenizer going forward.

For Context Length / Quality Trade-off in Video Generation, SemanTok reinforces VideoFlexTok’s claim that compressing at the tokenizer rather than at the KV cache or memory encoder can dominate the long-AR-video Pareto — and sharpens it: the benefits only materialize when the semantic supervision isn’t leaky. This has a direct implication for the open question on that concept page about which workaround dominates (“compress at the tokenizer” vs. “partition the cache” vs. “learned memory encoder”): the tokenizer-side approach is still improving along axes orthogonal to the cache-side approaches.

For Diffusion training efficiency and the broader DINOv2-aligned-tokenizer line (What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusion, Towards Scalable Pre-training of Visual Tokenizers for Generation, Diffusion Transformers with Representation Autoencoders), SemanTok adds a concrete datapoint on the “where should the semantic signal enter?” question. REPA on decoder features is one answer; SemanTok’s encoder-input + prefix-conditioned-reconstruction is another, and empirically the prefix-only reconstruction appears to make the semantic signal land where it’s load-bearing — in the token sequence itself, not just at the loss term. The internal-representation-guidance line (No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselves (SRA) SRA, Self-Flow: Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis Self-Flow) is an orthogonal direction worth watching here: does the SemanTok design still need an external DINO signal, or would an internal self-distillation suffice once the leakage path is closed?

Kyrylo’s accompanying question — whether these semantic spaces could reduce compute spent on video pixels while retaining what’s needed for robotic tasks — points at an obvious composition: SemanTok-style tokens feeding a WAM-side consumer such as Long-WAM: Scaling the Context of World-Action Models (Long-WAM already demonstrates that AR pretraining is the gating condition for robot context scaling) or the Rhoda/LingBot-VA causal-video-for-control line (Direct Video-Action Models — Causal Video Models Are Data-Efficient Robot Policy Learners, Causal World Modeling for Robot Control (LingBot-VA)). If the “cheap predictable prefix” property holds up, it offers an alternative to Photon-1’s extreme FSQ-based compression for policy pretraining (Scaling Video Pretraining with Imagination Models (Photon-1)).

See also

Section titled “See also”