Geometric Action Model for Robot Policy Learning

TL;DR

Geometric Action Model (GAM) is a 1.4B-parameter language-conditioned manipulation policy that repurposes a pretrained Geometric Foundation Model (GFM) — specifically DA3-Giant — as a single shared backbone for perception, future prediction, and action decoding. The GFM is split at an intermediate layer: shallow blocks act as a 3D-aware observation encoder, a causal future predictor inserted at the split forecasts future latent tokens conditioned on language, proprioception, and action history, and the predicted tokens are routed through the remaining GFM blocks where the original depth head decodes future geometry and a lightweight action head emits the action chunk in one forward pass. On LIBERO-Plus GAM (1.4B) leads the field at 85.5% with the smallest accuracy drop from clean LIBERO and a +9.7pp margin on the camera-perturbation split, while being up to 55× faster than diffusion-based VLAs of equivalent capability and roughly half the parameter count of π0.5 / Cosmos-Policy / OpenVLA-OFT.

Key claims

A pretrained 3D-geometry foundation model can be split at an intermediate layer and re-stitched into a language-conditioned world-action policy with minimal architectural change — shallow GFM layers as observation encoder, a causal transformer at the split for future-latent prediction, deep GFM layers reused as the decoder for both geometry and actions [§Method, project page §Architecture].
Predicting the next latent geometric state (rather than next RGB frame or next 2D latent) preserves explicit 3D priors and gives stronger robustness to camera, lighting, and layout perturbations than 2D-image-based VLAs/WAMs of the same family [project page §Robustness, LIBERO-Plus camera split].
A single feed-forward pass with KV-cached history is enough to emit an action chunk, yielding up to 55× wall-clock speedup over diffusion-based policies at comparable accuracy [project page §Speed].
The same backbone produces both decoded future depth maps (visualizable rollouts) and executable actions, so future-prediction supervision and action supervision share a representation rather than being two separate networks [project page §Method, Fig. on dual decoding heads].
On a custom real-robot suite of four contact-rich tasks (~200 teleop demos each), GAM is robust to held-out external-camera shifts of 85 cm and 45° where 2D-based policies collapse [project page §Real-robot].
On LIBERO-Plus overall, GAM (1.4B) reaches 85.5% vs π0.5 (3.3B) 84.6, Cosmos-Policy (2B) 82.4, OpenVLA-OFT (7B) 69.6, π0 (3.3B) 69.3, and Fast-WAM (6B) 50.0; on the camera-perturbation split it reaches 83.1 vs the next best Cosmos-Policy 73.4 [project page LIBERO-Plus table].
Pretraining mixture is Open-X Embodiment (72%) + MimicGen (18%) + RoboCasa365 (10%) for single-arm robots; benchmark fine-tuning uses future-depth supervision from simulator ground truth or teacher pseudo-depth [project page §Training].

Method

GAM builds directly on DA3-Giant (Depth Anything 3: Recovering the Visual Space from Any Views) — ByteDance Seed’s any-view geometry foundation model that jointly predicts depth and ray maps from one or many views through a 24-block DINOv2 ViT plus Dual-DPT head. GAM treats DA3 as a substrate to be cut open. The backbone is split into two segments at an intermediate layer: the shallow segment maps multi-view RGB frames into spatially meaningful latent geometric tokens (no task-specific encoder trained from scratch); the deep segment is preserved for feature propagation and decoding. Between them, a causal future predictor — a transformer conditioned on the task instruction, proprioceptive state, and action history — performs next-token prediction over the GFM’s latent geometric tokens, i.e. forecasts what the latent 3D world should look like one step ahead.

The forecasted future tokens are then run through the remaining (deep) GFM blocks. Two heads attach to this shared computation: the original DA3 depth/ray head decodes the future-frame depth (used both as auxiliary supervision and as visualization), and a small action head regresses an executable action chunk. Crucially, both outputs come out of the same forward pass, with the future-prediction loss and the action-regression loss sharing representations. Inference uses KV-caching over the action-history axis so each new step is a single feed-forward pass, which is what underwrites the 55× speedup vs diffusion-based action experts.

Pretraining mixes Open-X Embodiment (72%), MimicGen (18%), and RoboCasa365 (10%) for single-arm manipulation. Fine-tuning on each benchmark uses future-depth supervision — sourced either from the simulator’s ground-truth depth (when available) or from teacher pseudo-depth produced by a fixed depth model.

Results

LIBERO (saturated): GAM 97.6% — matches Cosmos-Policy 98.5%, OpenVLA-OFT 97.1, π0.5 96.9, Fast-WAM 97.6 [project page LIBERO table].
LIBERO-Plus overall: GAM 85.5% (drop −12.1pp from clean LIBERO) — best across the table; next is π0.5 84.6 (−12.3), Cosmos-Policy 82.4 (−16.1), then a wide gap to OpenVLA-OFT 69.6, π0 69.3, π0.5 + ROCKET 47.5, Fast-WAM 50.0, and π0.5 + Spatial Forcing 25.7 [project page LIBERO-Plus table].
LIBERO-Plus camera-perturbation split: GAM 83.1 vs Cosmos-Policy 73.4 vs π0.5 72.0 vs π0 61.0 vs OpenVLA-OFT 56.4 vs π0.5+ROCKET 30.9 vs Fast-WAM 16.4 vs π0.5+Spatial Forcing 0.1 — +9.7pp over second best [project page LIBERO-Plus table].
Parameter count: GAM 1.4B vs π0.5 3.3B, OpenVLA-OFT 7B, π0 3.3B, Cosmos-Policy 2B, Fast-WAM 6B — GAM is the smallest model in the comparison [project page LIBERO-Plus table].
Speed: up to 55× faster than diffusion-based policies (no diffusion sampling, KV-cached one-pass action emission) [project page §Speed].
Real-robot OOD camera shift: four contact-rich manipulation tasks, ~200 demos each; half of evaluation trials shift the external camera by 85 cm and 45°. GAM continues to work in this OOD setting where 2D-based policies collapse [project page §Real-robot].
Future-depth visualizations: because future prediction lives in the GFM latent space, GAM’s forecasts decode into dense future depth maps that closely track simulator ground truth across rollouts [project page §Future rollouts].

Why it’s interesting

GAM operationalizes a different recipe than every other VLA on VLA Models: instead of starting from a VLM backbone and bolting on an action head (the Qwen-VL + action-head pattern of Spirit-v1.5: Clean Data Is the Enemy of Great Robot Foundation Models, LingBot-VLA: A Pragmatic VLA Foundation Model, Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models, and Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Models), it starts from a geometry foundation model (Depth Anything 3: Recovering the Visual Space from Any Views) and stitches a future-token predictor into its middle. This makes it a strong test of whether 3D-aware perception priors are the load-bearing piece — exactly the bet Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level makes against pure VLM-scaling and that DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation makes by baking dynamics into the vision encoder rather than the policy head. The camera-perturbation result is the cleanest empirical signal: at 1.4B parameters GAM beats every much larger 2D-VLA baseline by 9.7+pp under camera shift, the regime where 2D priors should fail. GAM also sits squarely in the World Foundation Models cluster as a world-action model — directly comparable to Cosmos-Policy and Fast-WAM in the Pretrained to Imagine, Fine-Tuned to Act: The Rise of World-Action Models WAM taxonomy — but with a “predict next geometry, not next pixel” formulation that the taxonomy doesn’t yet name explicitly.