TV2TV: A Unified Framework for Interleaved Language and Video Generation

TL;DR

TV2TV is a Meta FAIR omni model that decomposes video generation into a globally autoregressive sequence of interleaved text segments and video frame chunks, jointly trained with next-token prediction on text and rectified-flow next-frame prediction on continuous video latents. The architecture is a Mixture-of-Transformers (MoT) with separate text and video towers sharing one global attention sequence; the text tower is initialized from Llama-3.2-3B. The interleaving lets the model “think in words” about what should happen next before “acting in pixels”, which on CS:GO video-game data gives a 91% pairwise preference over a matched T2V baseline and a 19-point fine-grained instruction-following improvement over a “think-then-act” (Think2V) baseline. The same recipe scales to natural sports videos by synthesizing interleaved captions with a VLM.

Key claims

Globally autoregressive interleaving of discrete text tokens and continuous video frame chunks, with bidirectional attention within a frame chunk and causal attention across the sequence, can be trained end-to-end with a joint cross-entropy + rectified-flow loss [§2.1, §2.2, Eq. 13-15].
A beginning-of-frame (BOF) special token is sufficient to make a single decoder dynamically switch modes at inference: emitting BOF triggers initialization of N noisy video tokens from N(0,I) and N ODE-solver steps under the existing KV cache (optionally with CFG); afterward the model returns to text generation [§2.3].
Maintaining two copies of each video frame’s representation in the input sequence — a noisy chunk followed by a clean chunk — resolves the teacher-forcing conflict between needing clean conditioning context and learning to denoise; this contrasts with MAGI-1’s monotone-noise scheme, which is less compatible with interleaved text conditioning [§2.1].
MoT-style modality-specific QKV, FFN, and pre-attention layer norms with a single global self-attention (and a hybrid causal/block-causal mask) are sufficient for joint text + video training; a global 1D RoPE is shared across modalities with duplicated position IDs for the noisy/clean video-token pair of the same frame chunk [§2.2, Eq. 5-10].
On CS:GO at 16 FPS / 256² with a 3B-MoT (3B text tower + 3B video tower), TV2TV is preferred over the matched T2V baseline 91% of the time vs. 1% (8% ties) in human pairwise evaluation; the long-form (64 s) gap is wider than the short-form (~6 s) gap [§3.4, Fig. 4].
Against the Think2V baseline (full text plan generated up-front before any frames), TV2TV wins 48% vs. 14% on visual quality and shows a 19-point improvement on fine-grained intervention-following accuracy, isolating the value of interleaving (vs. just adding a planning stage) [§3.4, Fig. 4, Fig. 5].
The recipe scales to real-world videos that lack temporally aligned text by using a VLM to synthesize interleaved natural-language action descriptions; an 8K-hour sports corpus trained this way wins 54.0% vs. 34.7% over a T2V baseline and 53.3% vs. 41.3% over Think2V in holistic preference evaluations [§4, §1].
Because the model is globally autoregressive, it can natively extend videos beyond its trained context window via a sliding-window where the second half of the previous interleaved sequence becomes the condition for the next window [§2.3].
Users can intervene mid-sequence by inserting a text action (e.g., “left-click”, “reload”, “jump”); TV2TV’s frame-chunk conditioning on the immediately preceding text segment is what gives it stronger intervention-following than Think2V’s up-front-plan scheme [§3.3, §3.4, Fig. 5].

Method

TV2TV models a sequence of the form [meta-prompt text; (text₁, noisy frame₁, clean frame₁, text₂, noisy frame₂, clean frame₂, ...)] chronologically by timestamp. Text uses a Llama BPE tokenizer; video uses Cosmos-Tokenize1-CV4x8x8-360p with a temporal compression factor of 4 so each “frame chunk” is 0.25 s at 16 FPS in latent space. Within a chunk, the model runs rectified-flow matching against logit-normal-sampled timesteps in the configuration of Esser et al. (2024); across chunks the model is autoregressive. Two distinct linear/U-Net projections handle text-token embedding and video-latent down/up-sampling; the timestep is added via a time embedding in the U-Net downsampler.

The MoT backbone uses modality-specific QKV matrices, FFNs, and layer norms but one shared global self-attention with a hybrid mask: causal on text, block-causal on noisy+clean video tokens, with the additional rule that noisy tokens cannot be attended by any future token. The text tower is initialized from Llama-3.2-3B; the video tower is trained from scratch. Training optimizes the sum of a text cross-entropy loss and a video flow-matching loss on all parameters. Inference is the dynamic-mode procedure above, with CFG implemented by maintaining a text-conditional and a text-unconditional sequence in parallel and taking ODE steps contrastively.

In the CS:GO controlled experiment, controller actions (keyboard + mouse) serve as the text interleaved with the gameplay video — a natural, temporally-aligned text+video corpus. For sports videos, a VLM is run over clips to produce per-segment natural-language action descriptions, manufacturing the interleaved supervision. All three model variants (TV2TV, T2V, Think2V) share the same MoT backbone, differing only in the text content of the training sequence.

Results

CS:GO visual quality: TV2TV vs T2V 91% / 1% / 8% ties; TV2TV vs Think2V 48% / 14% / 38% ties; long-form gains exceed short-form gains [§3.4, Fig. 4].
CS:GO fine-grained controllability: at 1 s and 3 s intervention timestamps over four action types (back, left-click/fire, reload, jump) plus a no-op control, TV2TV gains 19 points on intervention correctness over Think2V at matched visual-quality scores [§3.4, Fig. 5].
Sports holistic preference: TV2TV wins 54.0% vs. 34.7% (11.3% ties) against T2V and 53.3% vs. 41.3% (5.4% ties) against Think2V [§1, §4].
All models in the controlled experiments use 3B-MoT (3B text + 3B video towers) trained for 50K steps at batch size 128 on 95 hours of CS:GO video, with cosine LR schedule to a max of 1e-4 [§3.1, Table 6].

Why it’s interesting

TV2TV is the cleanest filed instance of Transfusion-style joint AR+diffusion training extended to video output, sitting at an intersection of three live wiki threads. Against Unified Multimodal Models — where the filed AR+Diffusion examples (BAGEL, DuoGen, MetaQuery, BLIP-3-o) all target image generation — this adds a video-output instance with a uniquely tight architectural commitment (separate towers, single global attention, dual noisy+clean representation) that is closer to LMFusion than to a stitched MLLM-then-DiT pipeline. Against Autoregressive Video Generation, it does something none of the filed long-horizon AR video papers (Context Forcing: Consistent Autoregressive Video Generation with Long Context, Pretraining Frame Preservation in Autoregressive Video Memory Compression, MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation) do: it interleaves generated text between frame chunks as the primary mechanism for offloading semantic decision-making out of the video tower, sidestepping the forgetting-vs-drifting dilemma by making the high-level plan a first-class autoregressive output. Most strikingly, this is the first filed instance of substrate (a) from Thinking with Modalities — tagged latent skills inside an autoregressive CoT — applied to generation rather than understanding: the text segments are the model’s own emitted reasoning trace conditioning subsequent pixel generation, and the 19-point gap over the “think-then-act” Think2V baseline is direct evidence that interleaving (CoT-during-video) strictly beats pre-planning (CoT-before-video). The CS:GO setup also overlaps World Foundation Models — TV2TV is functionally an action-conditioned game world model where the action stream is co-generated rather than user-supplied (cf. Genie 3’s external action input).