Understanding and Mitigating the Video-Action Generalization Gap via Temporal Ratio

TL;DR

Video-Action Models (VAMs) built on pretrained video foundation models often lose the backbone’s compositional priors after finetuning on robot action data — a discrepancy the authors call the video-action generalization (VAG) gap. This paper systematically sweeps a VAM design space (adaptation strategy, video-feature noise level, temporal rollout horizon) on top of Cosmos-Predict 2.5-2B and finds no clean pattern in raw success rates. Introducing Temporal Ratio (TR), an attention-based diagnostic that measures how much the flow-matching action head attends to future latent frames versus the current frame, reveals that the design-space noise is actually an interpretable future-reliance signal. TR-Adaptive Guidance uses TR at inference to amplify compositional video conditioning when the policy is planning and back off during precise manipulation, improving compositional OOD success on LIBERO and real-world bimanual YAM tasks while preserving in-distribution precision.

Key claims

Video backbones (Cosmos-Predict 2.5-2B) autoregressively rolled out for 1.7-second chunks over 10 steps hallucinate plausible novel object–goal combinations never seen in training, so the video prior itself has the compositional generalization needed to solve the target OOD tasks — the gap is downstream in the VAM [§Overview, §Video Models Generalize].
The VAM design space (LoRA vs full finetuning adaptation × video noise level σ × rollout horizon) shows no clean monotonic pattern in either absolute success rate or the ID-vs-OOD gap; a raw sweep leaves the practitioner without a design rule [§Analyzing the Space, §Noisy Design Space Manifold].
The mechanism behind the noise is that the action head ignores whichever video prediction it is fed — misleading predictions on ID tasks (still succeeds) and correct compositional predictions on OOD tasks (still fails) — so the failure is one of head–backbone coupling, not backbone capability [§Noisy Design Space Manifold].
The denoising process inpaints the current frame at every step while future frames denoise gradually, creating a persistent SNR asymmetry between current and future video latents at every extraction noise level; intermediate σ exposes future intent while preserving SNR [§Analyzing the Space, §Temporal Ratio].
Temporal Ratio (TR) — action-token attention over future video tokens divided by action-token attention over current-frame video tokens — turns the apparent design-space noise into an interpretable future-reliance scalar and varies with task phase, rising during planning and falling during precise manipulation [§Temporal Ratio].
TR-Adaptive Guidance uses TR at each replan step to amplify compositional video conditioning when the policy is future-reliant (planning) and reduce it when the policy is anchored to the current frame (precise manipulation), improving compositional OOD success on LIBERO spatial/goal/object suites while preserving ID precision [§Performance — Simulation (LIBERO)].
The same TR signal used in simulation transfers to a real-world YAM bimanual manipulation setup trained on 22 tasks drawn from ABC-130k, improving object–goal compositional OOD success under multi-candidate target and receptacle distractors [§Real Robot Analysis, §Performance — Real-world (YAM)].

Method

The setup is a Video-Action Model in which partially denoised latent video features from a pretrained video backbone (Cosmos-Predict 2.5-2B) are fed as conditions to a flow-matching action head. The paper first evaluates a broad VAM design space along three axes — adaptation strategy (LoRA → full finetuning), video-feature noise level σ (the denoising step at which latents are extracted), and temporal horizon (how far the video model rolls out before conditioning the action). Rollouts are analyzed to show that the content of the video prediction is largely uncorrelated with policy success in both directions (right predictions ignored on OOD, wrong predictions ignored on ID).

The diagnostic is the Temporal Ratio: at each replan step the action tokens attend over the current-frame video tokens and the predicted future video tokens; TR = (attention mass on future) / (attention mass on current). TR is used both as a post-hoc explanatory variable that collapses the design-space noise onto a single future-reliance axis, and as a runtime signal for TR-Adaptive Guidance, which amplifies the contribution of compositional video conditioning when TR indicates future-reliance and attenuates it during precise manipulation. Evaluation is on LIBERO’s object-goal compositional OOD variants (spatial, goal, object) in simulation, and on 22 bimanual YAM tasks (trained on ABC-130k trajectories) with multi-candidate target/receptacle distractors in the real world.

Results

Absolute success rates across (adaptation × video noise level × horizon) show no clear ID-vs-OOD pattern; the design-space manifold is described as “noisy” until TR is applied as the axis [§Noisy Design Space Manifold].
TR-Adaptive Guidance improves OOD success on LIBERO spatial/goal/object compositional suites while preserving ID precision; the bar chart shows the largest gains on the compositional splits and no ID regression [§Performance — Simulation (LIBERO)].
On real-world YAM bimanual tasks (22 tasks trained on ABC-130k), TR-Adaptive Guidance improves ID and compositional OOD success under multi-candidate distractors versus the baseline flow-matching VAM [§Performance — Real-world (YAM)].
Qualitative rollouts on LIBERO show the executed trajectory and the policy’s predicted video side by side, with a small number of tasks the guided policy still fails marked explicitly [§LIBERO OOD Rollouts].

Absolute numbers are not disclosed in the text on the project page; the LIBERO and YAM performance panels report success rates only as grouped bar charts.

Why it’s interesting

This paper adds a new WFM deployment pattern to the World Foundation Models ledger — WFM-as-source-of-inference-time-guidance-signal — distinct from the six already tracked there (backbone, predictive-surprise reward, interruptible tool, policy-internal substrate, edit-prior via KV cache, frozen trace-WM + small action expert): the video backbone is kept in the VAM stack, but its internal attention statistics are read out and fed back into the same policy’s guidance schedule. Complements the diagnostic frame of World Value Models for Robotic Manipulation (world-model-as-value-model) with an even lighter-weight signal that doesn’t require training a separate value head.

For the VLA Models cluster this is a sharper articulation of the failure mode Malik and Yunzhu Li’s counter-position (Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level) implicitly targets — the action head can silently ignore the perception stream — and offers an interpretability-first mitigation rather than a modality change: complements Causal World Modeling for Robot Control (LingBot-VA)‘s “unify the world model and action decoder end-to-end” recipe with a “keep them factored, but instrument the interface” recipe. TR itself sits in the Mechanistic Interpretability “attention-pattern-as-diagnostic” thread alongside Do Audio-Visual Large Language Models Really See and Hear? (attention shows AV-LLMs ignore audio on counterfactuals), pushed here from diagnosis into a runtime control lever — the pattern Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls and HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization have argued for in other domains.