Next Forcing: Causal World Modeling with Multi-Chunk Prediction

TL;DR

Next Forcing ports the multi-token prediction (MTP) training objective from the DeepSeek-lineage LLM recipe into causal autoregressive video generation for World Action Models. Lightweight auxiliary MCP heads are stapled onto the main video denoiser and trained to simultaneously denoise future chunks at horizons next¹, next², next³ — providing dense multi-scale temporal supervision that the standard “current-chunk-only” objective lacks. At 50 fps the recipe is reported to converge 2.3× faster than Causal World Modeling for Robot Control (LingBot-VA) (LingBot-VA) and reach a 93.1% relative improvement at 5k steps, with new SOTA 94.1/93.5 on RoboTwin Clean/Random and 2× inference acceleration when the MCP heads are kept at deployment to predict the next chunk in parallel with the current one. Also reports >50% FVD reduction on general video pretraining and significant gains on PhyWorld.

Key claims

Existing causal autoregressive video denoisers suffer slow training convergence and limited converged accuracy — particularly at high frame rates — because training supervision is confined to the current chunk with no explicit signal about future dynamics [§1, Abstract].
Multi-chunk prediction (MCP) augments the main model with lightweight auxiliary MCP modules that simultaneously denoise video chunks at multiple future temporal horizons (next¹, next², next³) during training [Abstract, §1].
The MCP modules form a causal chain across prediction depths: intermediate features fused from multiple layers of the main model are used to predict future dynamics, letting near-future predictions inform farther-future ones and providing dense multi-scale temporal supervision back to the main model [Abstract].
At 50 fps, Next Forcing achieves a 93.1% relative improvement over LingBot-VA at 5k training steps and 2.3× faster convergence [Abstract].
Establishes new state-of-the-art on the RoboTwin benchmark: 94.1 / 93.5 % on Clean / Random [Abstract].
At inference, the auxiliary MCP modules can be retained to predict the next video chunk in parallel with the current one, yielding 2× inference acceleration [Abstract].
Significant improvements on PhyWorld, a benchmark evaluating adherence to physical laws in video generation [Abstract].
Over 50% FVD reduction on general video pretraining [Abstract].

Method

Next Forcing keeps the standard causal-AR video denoiser as its main model (treating prior chunks as conditioning and denoising the current chunk via flow matching, as in Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation / LingBot-VA), then attaches K small auxiliary MCP heads during training. Head k is responsible for denoising the chunk at temporal horizon next^k given fused intermediate features of the main model’s hidden states at the current step. The heads are chained: head k+1 consumes head k’s features so near-future denoising informs farther-future denoising — the same “predict multiple steps ahead with a chain of lightweight predictors” pattern that DeepSeek V3 / GLM-4.5 / MiMo-V2-Flash use for LLM multi-token prediction (see LLM Architecture Gallery) but applied across chunks of denoised video latents rather than over text tokens.

The auxiliary heads return gradients only to the main model during training, which both accelerates convergence (multi-scale temporal supervision) and improves converged accuracy. At inference time, the system has two operating modes: (a) drop the heads, recovering a vanilla causal-AR generator with no extra cost but keeping the training-time accuracy wins; or (b) keep the heads and run the next-chunk prediction in parallel with the current chunk’s denoising — the same speculative-decoding trick MTP enables for LLMs (predict-then-verify) — buying 2× inference acceleration.

Results

Headline numbers from the abstract — full tables live in the paper body beyond the 30KB extraction window:

RoboTwin Clean / Random: 94.1 / 93.5 % — claimed SOTA [Abstract].
Convergence vs. Causal World Modeling for Robot Control (LingBot-VA) at 50 fps: +93.1 % relative improvement at 5k training steps; 2.3× faster convergence [Abstract].
Inference: 2× speedup when MCP heads predict next chunk in parallel with current [Abstract].
PhyWorld: “significant improvements” (no quantitative number in abstract) [Abstract].
General video pretraining: >50% FVD reduction [Abstract].

Why it’s interesting

This is the cleanest filed instantiation of MTP-for-video — a transfer of one of the load-bearing LLM training recipes (the DeepSeek V3 multi-token prediction head; see LLM Architecture Gallery and GLM-5: from Vibe Coding to Agentic Engineering) into the streaming video AR setting. The Slack note framed it exactly this way: “apply the concept of multi-token prediction to streaming video gen”. Where Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation and Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generation attack the initialization of few-step AR students (causal ODE / CD distillation), Next Forcing attacks a different axis — the training-time supervision shape of the main model itself — and is therefore upstream-orthogonal: in principle the MCP heads could be added to a Causal-Forcing-initialized student. The choice of Causal World Modeling for Robot Control (LingBot-VA) (LingBot-VA) as the head-to-head baseline, plus the RoboTwin / PhyWorld evaluation, also frames Next Forcing as a control-oriented contribution — a faster, more accurate visual-dynamics core for VLA pipelines — which sets it apart from the long-video-generation framing of the rest of the Autoregressive Video Generation cluster. The 2× inference speedup via parallel next-chunk prediction is also the first filed video-AR instance of MTP-as-speculative-decoding (Speculative Decoding: Performance or Illusion? discusses the LLM-side caveats).