Skip to content
Bud's Wiki

Grounded Action Model: 3D Grounding as a Foundation for Robotics

TL;DR

Section titled “TL;DR”

Grounded Action Model (GAM) proposes 3D grounding as the primary pretrained substrate for robot foundation models, in contrast to VLA models (which start from a VLM backbone) and World-Action Models (which start from a video generator). A pretrained 3D grounding model (WildDet3D in the reference implementation, with SAM3D as a plug-in alternative) localizes task objects and estimates their geometry from language / point / box prompts, producing an object-centric representation — target-focused image tokens plus detection tokens carrying object point clouds and metric geometry — that a multi-stream MM-DiT combines with robot state history and language to predict flow-matching action chunks. Because the visual conditioning is filtered through selected-object geometry, GAM can be re-targeted mid-episode by pointing at a different object, and it responds to task-relevant spatial changes while attenuating irrelevant scene variation. On RoboTwin 2.0 (50 tasks) GAM reports 55.3% avg success vs 52.0% for Spatial Forcing, and 47.6% under scene randomization vs 30.4% for Abot-M0 — with action policies trained only on clean-scene demonstrations. On LIBERO-PRO (16 perturbation settings), GAM achieves 61% avg vs 53% for π₀.₅ with the largest gains when targets are relocated or newly designated; on LIBERO-Spatial with a changed target, GAM reaches 88% vs π₀.₅’s 1%.

Key claims

Section titled “Key claims”
  • 3D grounding — localize task objects and estimate their geometry with a pretrained model, then act on that representation — is a foundation-model substrate for manipulation policies distinct from VLM-backbone VLAs and video-generation-backbone WAMs [Abstract, §1].
  • GAM is prompt-agnostic: language, image-space points, or bounding boxes all map to the same object-centric selected-object representation, giving a high-level VLM a direct interface for target designation without retraining the low-level controller [Abstract, §Method].
  • The visual conditioning path retains image features only around selected objects and the robot arm, while detection tokens encode object point clouds and metric geometry — an object-centric multi-stream input rather than a whole-image feature map [Abstract, §Method].
  • On RoboTwin 2.0 across 50 tasks, GAM reports 55.3% avg success vs 52.0% for Spatial Forcing; under scene randomization, 47.6% vs 30.4% for Abot-M0 — with the action policy trained only on clean-scene demonstrations and the grounding backbone adapted to simulator images separately then frozen [Abstract, §Results].
  • On LIBERO-PRO (16 perturbation settings), GAM reaches 61% avg success vs 53% for π₀.₅, with the widest margins when targets are relocated or newly designated [Abstract, §Results].
  • On LIBERO-Spatial when the target changes, GAM reaches 88% vs π₀.₅’s 1% and outperforms Flexi-π (a recent 3D WAM); the objects and actions are familiar, so the isolated variable is acting on a newly designated target [§Results, launch thread 7/].
  • Real-hardware bimanual YAM: 17/20 successes under visual shift vs 4/20 for π₀.₅ [Abstract].
  • Composed with a Molmo2 planner on a Franka, GAM + Molmo2 achieves 64.7% ID and 49.8% OOD step completion on long-horizon and memory-dependent tasks — the planner supplies target-point sequences from history, GAM handles per-step actions [Abstract, §Results].
  • Author-flagged limitation: grounding errors propagate into the action expert, and unselected obstacles (task-irrelevant to the grounding decision but present in the scene) are omitted context that the action expert has no channel to consider [launch thread 9/].

Method

Section titled “Method”

GAM’s architecture has three components in series. First, a pretrained 3D grounding model (WildDet3D in the reference implementation; SAM3D is called out as a plug-in alternative) receives a prompt — language, point, or bounding box — and returns two things: a set of image tokens focused on the selected objects and the robot arm, and detection tokens carrying per-selected-object point clouds and metric geometry. Second, a multi-stream MM-DiT combines these two token streams with robot state history and language, denoising an action chunk under flow matching. Third, actions are executed on the target embodiment.

The training recipe holds the grounding backbone fixed during action-policy learning: the paper reports adapting the grounding model to simulator images separately (a domain-adaptation stage), then freezing it while training the action policy on clean-scene demonstrations only. Because the object-centric representation is invariant under scene randomization and camera perturbations by construction (the grounding model absorbs them), the action expert never has to learn perturbation robustness — it inherits it from where the grounding model has been trained.

Composition with a high-level planner (Molmo2 in the paper) is a clean pattern: the planner reads current observation + episode history and emits a target designation (a point on the object of interest) which becomes GAM’s next prompt. This is the same interface GAM accepts from a human via point-click, so no additional adapters are needed to connect the two.

Results

Section titled “Results”
  • RoboTwin 2.0 (50 tasks): 55.3% avg success vs 52.0% for Spatial Forcing (+3.3 pp).
  • RoboTwin 2.0 under scene randomization: 47.6% vs 30.4% for Abot-M0 (+17.2 pp) — action policy trained only on clean-scene demos.
  • LIBERO-PRO (16 perturbation settings): 61% avg vs 53% for π₀.₅ (+8 pp), largest gains when targets are relocated or newly designated.
  • LIBERO-Spatial with target change: 88% vs π₀.₅ 1% — the sharpest isolated datapoint for the object-designation channel.
  • Real bimanual YAM under visual shift: 17/20 successes vs π₀.₅ 4/20.
  • Long-horizon Franka with Molmo2 planner: 64.7% ID / 49.8% OOD step completion.

Why it’s interesting

Section titled “Why it’s interesting”

GAM adds a 3D-grounding-first row to the VLA Models recipe-lever board that is structurally distinct from every existing entry. Where π*0.6 (π*0.6: a VLA That Learns From Experience (RECAP)) puts the action head at the center and scales RL on top, Embodied-R1.5 (Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models) puts a pointing-capable unified VLM at the center, and Isaac 0.5 (Isaac 0.5: An Open-Weight Embodied Foundation Model) puts general video pretraining at the center, GAM puts a pretrained 3D detector at the center and treats the action expert as a small, embodiment-specific module downstream. The perturbation robustness claims (47.6% vs 30.4% under scene randomization, 88% vs 1% on target change) fall out of the architecture rather than from a training-side robustness recipe — the grounding backbone absorbs the perturbations before the action expert sees them.

The closest sibling on file is LIT — two-stage VLA/WAM training constrains visual conditioning via latent interface to prevent visual shortcuts LIT (same lead author) — LIT also argues that vision should enter the action expert only through a goal-relevant bottleneck, but LIT’s bottleneck is a learned latent interface trained to recover a spatial goal, while GAM’s bottleneck is a pretrained 3D grounding module whose output is metric geometry of selected objects. LIT and GAM read as two attempts at the same underlying thesis from the same lab — that current VLA action experts pick up spurious visual correlations, and that the fix lives in restructuring what information enters the action expert — with LIT betting on training-time restructuring and GAM betting on a modular 3D-perception frontend. Together they make the sharpest filed argument for an object-centric intermediate representation as the missing lever, and structurally counter the Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level sensorimotor-counter-position on a different axis: not by adding tactile / force, but by pushing perception upstream so the action expert operates on already-geometrized inputs.

See also

Section titled “See also”