MolmoMotion: Language-guided 3D motion forecasting

TL;DR

MolmoMotion is Ai2’s open motion-forecasting model built on the Molmo 2 VLM backbone: given a single RGB frame, a small set of 3D query points on a target object, and a natural-language action description, it predicts each point’s future 3D world-coordinate trajectory over the next few seconds. Two variants are released — an autoregressive variant (MolmoMotion-AR) that emits coordinates as structured text token-by-token, and a flow-matching variant (MolmoMotion-FM) that denoises continuous 3D trajectories. The release bundles MolmoMotion-1M (1.16M videos, 736 motion types, 5.6K objects with auto-extracted object-grounded 3D trajectories) and PointMotionBench (2.7K human-validated held-out clips spanning indoor manipulation, egocentric HOI, and outdoor scenes). The model is reported to beat both pixel video generators (Wan2.2-5B, Cosmos Predict) and parametric/track baselines (Track2Act, Robot4DGen, EgoScaler, ObjectForesight) on 3D average displacement error, and to transfer non-trivially to robotics planning (76.3% closed-loop pick-and-place vs. 56.0% Molmo 2 baseline) and trajectory-conditioned video generation.

Key claims

Motion is represented as object-attached 3D points in a shared world frame, chosen as the only candidate satisfying three required properties simultaneously: class-agnostic (no per-category templates), view-stable (consistent across cameras), and directly consumable by downstream policies / video generators [§MolmoMotion: Under the hood].
The architecture re-uses Molmo 2 as a VLM backbone: it first grounds the referenced object and identifies the query points from language, then a per-variant head emits trajectories — the AR variant writes coordinates as structured text in temporal order (VLM coordinate-token style), the FM variant denoises continuous 3D positions [§MolmoMotion: Under the hood].
The FM variant is positioned as the uncertainty tool: for instructions admitting multiple plausible futures, flow-matching draws from a distribution rather than committing to a single rollout; AR is the strongest-accuracy tool when the future is well-defined [§MolmoMotion: Under the hood].
MolmoMotion-1M is auto-extracted from 1.16M unconstrained videos via a five-stage pipeline: (1) ground the moving object from the action description, (2) sample query points on it, (3) dense 2D point tracking, (4) lift tracks into a metric 3D world frame, (5) filter via object-level spatial+temporal consistency priors and clip to intervals of meaningful motion [§Introducing MolmoMotion-1M and PointMotionBench].
PointMotionBench reports MolmoMotion-AR (3-frame input) at 0.109m 3D average displacement error on HOT3D, beating ObjectForesight 0.129, Wan2.2-5B 0.200, Cosmos Predict 0.225, and Track2Act 0.294 [§3D motion forecasting, PointMotionBench bars].
On WorldTrack (outdoor dynamic scenes), MolmoMotion-AR (3f) reaches 0.143m vs. Cosmos Predict 0.831, Wan2.2-5B 0.852, Track2Act 1.230 — a much larger gap to pixel-space video generators than on HOT3D [§3D motion forecasting, WorldTrack bars].
After fine-tuning on DROID, a downstream MolmoBot manipulation policy initialized from MolmoMotion reaches 76.3% closed-loop pick-and-place success vs. 56.0% for the same policy initialized from Molmo 2, and 51% success at 10K steps vs. 19% — a sample-efficiency gain that holds across seen/unseen scene × seen/unseen object splits [§Downstream evaluation: robotics planning, Fig. 5a].
On real-robot fine-tuning, the MolmoMotion-initialized policy reaches the Molmo 2 baseline’s 12K-step test L2 error in ~2K steps [§Downstream evaluation: robotics planning].
Used as a trajectory-conditioning signal for DaS, MolmoMotion improves CogVideoX-5B-based generation on all five VBench motion metrics and beats Wan2.2-I2V-A14B on four of five (temporal consistency 0.968, subject consistency 0.950, motion smoothness 0.990, background consistency 0.948; loses only on dynamic degree) [§Downstream evaluation: video generation, Table 2].
Stated limitation: only 8 query points per object during training — enough to forecast useful trajectories but insufficient to densely represent surface geometry, limiting handling of complex deformable motion [§Limitations and what’s next].
Released openly: model weights, MolmoMotion-1M dataset, PointMotionBench, and code under Apache 2.0 [§Limitations and what’s next; license noted in source pointer].

Method

The model factors motion forecasting through Molmo 2 as a VLM stage that resolves the referent (“which object”, “which query points”, “what motion”) from the language instruction and image, followed by a trajectory-emitting head. MolmoMotion-AR treats 3D coordinates as structured text — concretely, the same coordinate-as-text format Molmo uses for pointing — and emits the future trajectory token-by-token in temporal order; this autoregressive conditioning is what the post credits for smooth rollouts. MolmoMotion-FM keeps the same VLM front-end but swaps the head for a flow-matching denoiser operating in continuous 3D coordinate space, so the same image+instruction+query-points conditioning produces a distribution over future trajectories rather than a single mode. Training data comes from MolmoMotion-1M, where the auto-annotation pipeline is the non-trivial engineering: dense 2D point tracking → metric-3D lifting → object-level coherence filtering (points whose motion is incoherent with the rest of the object are dropped as tracking/depth noise) → temporal segmentation to the window where the object actually moves. The pipeline is what lets the model train on unconstrained internet video despite the absence of native 3D annotations there. For robotics planning, MolmoMotion is fine-tuned on the DROID open robot-manipulation video dataset; for trajectory-conditioned video generation, its predicted point trajectories are fed as the motion condition to DaS on top of a CogVideoX-5B base.

Results

The headline 3D motion forecasting numbers on PointMotionBench are reported by domain: HOT3D (indoor manipulation, 0.109m best vs. 0.180m static baseline, 0.200m Wan2.2-5B); WorldTrack (outdoor dynamic, 0.143m best vs. 0.167m static, 0.852m Wan2.2-5B); DAVIS (generic in-the-wild segmentation videos, 1.146m best vs. 2.281m static, 3.074m Wan2.2-5B). The 3-frame-input AR variant is the top model on HOT3D and WorldTrack; the 1-frame AR variant leads on DAVIS. Robotics transfer: 76.3% average closed-loop pick-and-place success vs. 56.0% Molmo 2 baseline on the same MolmoBot policy, with the gap larger on unseen objects (74.5 vs. 51.2 seen-scene-unseen-object; 74.2 vs. 48.7 unseen-scene-unseen-object). Real-robot fine-tuning reaches the Molmo 2 baseline’s 12K-step error in ~2K steps. Video generation: DaS + MolmoMotion improves CogVideoX-5B on all five reported VBench motion metrics and beats the much larger Wan2.2-I2V-A14B on temporal consistency, subject consistency, motion smoothness, and background consistency; loses to Wan2.2 only on dynamic degree (0.876 vs. 0.908).

Why it’s interesting

This is the most direct on-wiki sibling to Forecasting Motion in the Wild (Thakkar et al., “Forecasting Motion in the Wild”) — both papers commit to point trajectories as the motion-representation primitive over either pixel video or parametric 3D models, both build a custom in-the-wild data pipeline (RANSAC homography + animal masks there, object grounding + metric-3D lifting + coherence filtering here), and both report beating pixel video generators on motion-forecasting metrics; MolmoMotion is the language-conditioned 3D variant and adds explicit downstream robotics + video-generation transfer experiments that the Berkeley/DeepMind paper does not. As a counter-position to World Foundation Models pixel-rollout WFMs, this strengthens the “third representation” framing — Wan2.2-5B and Cosmos Predict are reported to lose to a much smaller motion-track model on 3D ADE by 2–6× on outdoor scenes. The robotics transfer story slots into VLA Models as a structural counter-recipe to both end-to-end VLA training and Embodied-R1.5’s pointing-as-glue position: MolmoMotion is pretrained as a motion forecaster (not an action policy, not a pointer), and the resulting backbone yields ~20-percentage-point gains over the same Molmo 2 backbone when reused as a policy initializer — evidence that explicit object-motion prediction may be a more useful pretraining target than image grounding alone. The Molmo 2 → MolmoPoint → MolmoMotion progression at Ai2 (MolmoPoint: Better pointing architecture for vision-language models introduced the grounding-token mechanism MolmoMotion’s referent resolution presumably reuses) is a coherent open-stack alternative to the closed Gemini Robotics / Cosmos / π* lineages.