VideoMAR: Autoregressive Video Generation with Continuous Tokens

TL;DR

VideoMAR is a decoder-only autoregressive image-to-video model that lifts MAR-style continuous-token generation from images to video by composing temporal frame-by-frame causal AR with spatial bi-directional masked generation, supervised by a next-frame diffusion loss. To make long-sequence AR tractable, the recipe combines temporal short-to-long curriculum learning, spatial progressive-resolution training, and a progressive-temperature inference schedule to dampen accumulation error. The resulting 9.3%-parameter, 0.5%-data, 0.2%-GPU model surpasses Cosmos I2V on VBench-I2V and exposes language-model-style capacities — temporal-wise KV caching plus spatial-wise parallel decoding, and joint spatial+temporal extrapolation via 3D-RoPE.

Key claims

Temporal causality and spatial bi-directionality are the right first principle for video AR: frames have a natural sequential order (causal) while the within-frame token order does not (bi-directional masked) [§1].
A next-frame diffusion loss integrates masking and continuous-token AR — masking ratio 0.7–1.0 during training, cosine schedule from 1.0 → 0 across 64 AR steps at inference, with a 3-block 1280-dim MLP denoiser and a 1000-step noise schedule trained / 100-step inference [§4.1, MAR-style].
Temporal short-to-long curriculum learning and spatial progressive-resolution training are claimed to be the operative levers for making long-sequence AR training tractable at the reported budget [Abstract].
A progressive-temperature inference schedule explicitly targets accumulation error in the AR rollout [Abstract].
VideoMAR inherits two LM-style efficiencies absent from bidirectional video DiTs: temporal-wise KV cache reuse across already-generated frames, and spatial-wise parallel masked decoding within the current frame [§1].
3D rotary positional encoding gives simultaneous spatial and temporal extrapolation — generating videos at resolutions and lengths beyond training [Abstract, §4].
On VBench-I2V, VideoMAR surpasses Cosmos I2V at 9.3% of the parameters, 0.5% of the training data, and 0.2% of the GPU resources [Abstract].
The backbone is a 36-layer 1536-dim decoder-only transformer; text conditioning uses Qwen2-1.5B as the text encoder injected via cross-attention, following FAR [§4.1].

Method

The architecture is a single decoder-only transformer that processes a video as a sequence of frame chunks. Within each frame, tokens are continuous (MAR-style, not VQ-discrete) and trained with a per-token diffusion loss applied by a small denoiser MLP; the mask ratio over spatial tokens follows the MaskGIT-style cosine schedule and the diffusion timestep is sampled per training step. Across frames, attention is causal and frames are emitted one at a time, allowing a KV cache to be carried forward.

Training stages chain two curricula: temporally, the model is first trained on short clips and progressively extended; spatially, training resolution is increased progressively. At inference, mask ratio decays from 1 to 0 over 64 AR steps per frame, and an additional progressive temperature schedule (varying the sampling temperature across AR steps) is applied to keep the rollout on-manifold. 3D-RoPE (separately on H, W, and T axes) underpins extrapolation to longer sequences and larger resolutions than seen at training.

Results

VBench-I2V: surpasses Cosmos I2V (the previous SOTA AR I2V baseline) while using only 9.3% of the parameters, 0.5% of the training data, and 0.2% of the GPU compute [Abstract].
Demonstrates spatial+temporal extrapolation: generates videos at resolutions and lengths beyond the training distribution, attributed to 3D-RoPE and the progressive-resolution training [§4].
The companion implementation (Ming-VideoMAR, Inclusion AI) reports better quality and finer detail than Cosmos at lower resolution (480×768 vs Cosmos’s 640×1024) [README of inclusionAI/Ming-VideoMAR].

Why it’s interesting

VideoMAR is a clean instantiation of the MAR-on-video direction that the autoregressive-video lineage has been circling. It complements MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation and Context Forcing: Consistent Autoregressive Video Generation with Long Context by attacking the long-sequence problem at the architecture and training-curriculum layer (temporal causality + spatial masking + temporal short-to-long + 3D-RoPE) rather than at the KV-cache structure or distillation layer — and the headline efficiency numbers (0.2% GPU vs Cosmos I2V) make a strong case that MAR’s continuous-token recipe transfers to video with minimal scale. It contrasts with NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scale which scaled MAR’s continuous-token AR to large image models without the temporal frame-by-frame structure VideoMAR adds. The progressive-temperature inference idea is also a different lever from the noise-augmented-memory drift correctors in MALT / Context Forcing / FlowAct-R1.