Unified Motion-Action Modeling for Heterogeneous Robot Learning

TL;DR

UMA (Unified Motion-Action) treats 3D object motion trajectories and robot actions as co-evolving variables under a single masked generative objective, where the mask pattern selects whether the model is doing motion-conditioned visuomotor control, dynamics modeling, or task adaptation at deployment. Pretraining uses hindsight-relabeled motion contexts plus a contrastive objective that disentangles task intent from scene geometry, removing the need for manually annotated task instructions across heterogeneous data sources (robot demos, human videos, simulated data). The pitch is that one set of weights, plus a mask choice at inference, replaces three specialized models — and that the headline claim is matching or beating mode-specialized baselines on each.

Key claims

Object motion trajectories and robot actions are modeled jointly as co-evolving variables, and the mask pattern at inference time selects between motion-conditioned visuomotor control, motion-based dynamics modeling, and few-shot task adaptation [Abstract].
Hindsight-relabeled motion contexts let the same masked-generative objective serve as supervision across robot demos, human videos, and simulated data without manual task-instruction annotation [Abstract].
A contrastive auxiliary objective disentangles task intent from scene geometry during pretraining, which the paper argues is what enables multi-task pretraining on heterogeneous sources [Abstract].
At deployment, the same pretrained parameters support all three inference modes — control, dynamics modeling, and few-shot demonstration-based adaptation — without per-mode finetuning [Abstract].
UMA outperforms state-of-the-art baselines that are each specialized for one of those inference modes, across the same backbone weights [Abstract].

Method

UMA is a masked generative model over two interleaved variable streams — 3D object motion trajectories and robot actions — both conditioned on visual observations and (where present) language. Training data is the union of robot teleop demos, human videos, and simulation rollouts; supervision is action labels where they exist and hindsight-relabeled motion contexts otherwise (i.e. the future motion of objects in the scene serves as the conditioning signal that the model learns to predict and respond to). The mask pattern over the joint (motion, action) sequence controls what the model sees vs. predicts: mask the action stream → motion-conditioned visuomotor control; mask the motion stream → dynamics modeling; mask both partially given a small in-context demo → few-shot adaptation.

The contrastive objective is the lever that lets a single masked head learn from such heterogeneous data without instruction labels: it pulls together representations that share task intent while pushing apart ones that share only scene geometry. Without it, the mask-only objective would let the model collapse into “copy the scene”, per the abstract framing. The exact architecture, mask schedule, contrastive negatives, and trajectory parametrization are body-level details not visible in the abstract alone and would need a PDF pass to fill in.

Results

The abstract makes one quantitative summary statement: across the three inference modes (motion-conditioned control, motion-based dynamics, few-shot adaptation) UMA consistently outperforms the SOTA baseline that is specialized for that mode, using the same pretrained parameters. No suite names, success rates, or per-baseline numbers are disclosed in the abstract; the v2 PDF body (uploaded 2026-06-16, 4.3 MB) contains the experimental tables.

Why it’s interesting

UMA is the wiki’s second 2606 entry — alongside μ₀: A Scalable 3D Interaction-Trace World Model — to bet on object motion trajectories as the shared interface between perception and control, and the most direct one to also fold action prediction into the same objective. µ₀ keeps the world model frozen and trains an embodiment-specific action expert on top of its 3D traces; UMA instead unifies motion and action under a single masked objective, picking the inference mode by changing the mask rather than swapping a head. That gives a cleaner “one model, three modes” story but loses µ₀’s clean cross-embodiment factorization. If both papers’ headline claims hold, the cluster has two distinct trace-as-interface designs to compare side-by-side.

It is also a structural counter-recipe to the action-pretraining VLAs the VLA Models page is built around — π*0.6: a VLA That Learns From Experience (RECAP) (π*0.6 RECAP), Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models (Embodied-R1.5), π0.7: A Steerable Robotic Foundation Model with Emergent Compositional Generalization (π0.7) — by claiming you can get competitive control out of a masked-generative model whose primary signal is object motion rather than action labels, with human video and sim rollouts admitted on equal terms. It complements μ₀: A Scalable 3D Interaction-Trace World Model (frozen 3D-trace world model + action expert) and Direct Video-Action Models — Causal Video Models Are Data-Efficient Robot Policy Learners (causal video model as the policy + inverse-dynamics decoder) as the third member of the “small action interface on top of a heavy motion/video pretrain” family.