Modality-Autoregressive World-Action Models

TL;DR

ModAR (CMU) is the first World-Action Model that autoregressively denoises multiple future modalities — depth, DINO features, point tracks, and optionally RGB — before predicting actions, so each modality is conditioned on the previously generated ones rather than denoised in parallel or from a single fused representation. The authors train from scratch to systematically vary training-data mixtures, predicted modalities, and WAM formulations. Empirically: predicting point tracks + DINO + depth helps, adding RGB does not help consistently, and ModAR’s sequential generation beats existing WAM formulations at every data scale evaluated. ModAR trained from scratch achieves slightly higher observed success (75% vs 72%) than the video-model-initialized Flex-π on the same data, using ~20× fewer training FLOPs and no pretraining. On three real-world bimanual tasks ModAR outperforms baselines and improves further with human video co-training [Abstract].

Key claims

Existing WAMs typically predict future RGB frames; other modalities (depth, DINO, point tracks) can capture geometric / semantic / motion structure more efficiently, but the question of how to combine them within a WAM is open [Abstract].
Autoregressive denoising across modalities — generate modality k conditioned on modalities 1..k−1 — outperforms parallel or single-representation WAM formulations at every evaluated data scale [Abstract].
Predicting point tracks, DINO features, and depth maps yields consistent WAM benefits; additionally predicting future RGB does not provide a consistent benefit [Abstract].
Trained from scratch, ModAR reaches a slightly higher observed average success rate than the video-model-initialized WAM Flex-π: A Multi-Stream World-Action Model with Compute Flexibility fine-tuned on the same data (75% vs 72%) at ~20× fewer training FLOPs and no pretraining [Abstract].
On three real-world bimanual tasks, ModAR outperforms baselines and improves further with the addition of human videos [Abstract].

Method

ModAR treats the WAM’s world-prediction target as an ordered set of modalities rather than a single output. Given the current observation, a sequence of denoisers produces future depth, DINO features, and point tracks (with RGB an optional additional stream), where each stream’s denoising is conditioned on the already-generated streams via autoregressive factorization over modalities. Actions are predicted from the shared backbone that carries this modality chain. The paper trains from scratch (no video-model initialization) and reports systematic sweeps over three axes: (i) which modalities to include, (ii) which training-data mixtures (robot only vs +human video), and (iii) which WAM formulation (parallel vs single-fused-representation vs modality-autoregressive). Real-world evaluation uses three bimanual tasks; the abstract does not name them.

Results

Modality-autoregressive generation outperforms existing WAM formulations at every evaluated data scale [Abstract].
Best modality set = point tracks + DINO features + depth; adding RGB is not a consistent lift [Abstract].
75% vs 72% average observed success against fine-tuned Flex-π on matched data at ~20× fewer training FLOPs and no pretraining [Abstract].
On three real-world bimanual tasks: ModAR beats baselines; human-video co-training gives further improvements [Abstract].

(Per-task success rates, benchmark names, and the identity of the “existing WAM formulations” beyond Flex-π are not in the abstract and would need the paper body to be extracted.)

Why it’s interesting

ModAR is the sharpest filed answer to a specific question left open by EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data and Flex-π: A Multi-Stream World-Action Model with Compute Flexibility: EgoWAM asked which single world-model target transfers under WAM co-training (pixel-VAE / DINO / 3D-flow, ran them separately, and found DINO wins on OOD while 3D-flow wins in-domain); Flex-π kept the set fixed (RGB + 3D pointmap + DINOv3) but denoised them jointly under one shared latent with per-stream dropout so any subset can be dropped at inference. ModAR adds the missing third axis — ordering the modalities as an autoregressive chain — and reports that (a) which modalities matter aligns with EgoWAM’s ranking (DINO + point tracks + depth are the load-bearing three; RGB is not), and (b) the autoregressive coupling itself is a load-bearing recipe piece beyond just the modality set, empirically beating parallel-denoising WAMs at every data scale.

The efficiency claim — matching or beating a video-generator-initialized WAM at ~20× fewer FLOPs with no pretraining — is the kind of “video-generator pretraining is a crutch you can drop” result that sharpens the wiki’s tracked debate between Pete Florence’s Going Beyond World Models & VLAs “video pretraining is a crutch” position and the from-scratch native pretraining bet of LingBot-VA 2.0: Native Video-Action Pretraining for Generalizable Robot Control. ModAR is a small-scale point on that curve; whether the crossover holds at Flex-π / LingBot-VA scale is not tested. The human-video co-training improvement — same finding as EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data on the human-to-robot channel — reinforces that the WAM head is the interface where human data enters the policy.