See like a Robot: Robot-Centric Pointmaps for Vision-Language-Action Models

TL;DR

VLAs observe scenes in the camera frame but must predict actions in the robot’s own frame, and this observation-to-action frame mismatch gets harder as training data aggregates demonstrations across diverse camera setups. See like a Robot injects robot-centric pointmaps — images whose pixels store per-pixel 3D coordinates in the robot frame — as a second input stream through one extra encoder plus one element-wise addition into a pretrained 2D VLA. On RoboCasa this lifts both π0.5 (+7.6 avg SR) and SmolVLA (+4.2) and beats all filed camera-aware, 3D-augmented, and point-cloud baselines; on a real FR3 with three training camera viewpoints the advantage over an RGB-only policy grows from +5.0 at seen viewpoints to +11.7 at unseen ones.

Key claims

Framing the VLA generalization problem as an observation-to-action frame mismatch correctly predicts that RGB-only policies degrade with training-viewpoint variation while pointmap-augmented policies stay flat: on 24 RoboCasa tasks, sweeping viewpoint randomization drops RGB-only by 9.6 points but RGB+Pointmap only by 1.8 [§Motivation, Fig. 1].
On RoboCasa with randomized third-person viewpoints, adding pointmaps to π0.5 and SmolVLA lifts them by +7.6 and +4.2 average success respectively, with π0.5+Pointmap reaching 62.9% average — outperforming FP3 (42.8), OC-VLA (56.3), KYC (59.1), GeoVLA (57.1), and PointVLA (57.3) at matched pretraining [Table 1 / RoboCasa].
On a real Franka FR3 across four tasks with three training camera positions and 180 demonstrations, pointmap advantage over π0.5 grows from +5.0 at seen viewpoints (78.3 vs 73.3) to +11.7 at an unseen held-out viewpoint (66.7 vs 55.0) [Real-robot table].
Providing robot-frame geometry as a pointmap is more informative than providing the ingredients (Plücker rays + depth) and asking the policy to infer robot-frame geometry: RGB+Pointmap 34.7% SR vs RGB+Plücker+Depth 31.6% vs RGB 27.9% at controlled scale [Input-ablation table].
Pointmap-shaped inputs beat point-cloud inputs at matched information content because they preserve the H×W grid the pretrained VLA expects: encoding via image encoder with element-wise addition reaches 34.7% SR vs PTv3-encoder+concat at 32.8% and point-cloud MLP+concat at 24.2% [Encoder/fusion ablation].
Centering the pointmap at the current end-effector position (rather than the robot base) reduces variation from absolute workspace location and improves success under both fixed and randomized evaluation viewpoints (36.9 vs 34.7 fixed; 36.6 vs 32.7 randomized, so a −0.3 drop under randomization vs −2.0 for base-frame) [Origin-ablation table].

Method

The paper’s central object is a robot-centric pointmap: an H×W image whose pixel at (u,v) stores the 3D coordinates in the robot’s coordinate frame of whatever scene point projected to that pixel. Depth from an RGB-D sensor is unprojected into camera-frame 3D and then transformed into the robot frame via the known extrinsics; optionally, coordinates are re-centered at the current end-effector position (the paper’s preferred setting) instead of the robot base to further reduce workspace-position variation. Because the pointmap has the same dense grid layout as the RGB input, it can be tokenized by a second image encoder identical in shape to the RGB encoder — and the two token streams are fused by element-wise addition before entering the VLA backbone. The whole architectural delta is “one extra encoder + one add”; the pretrained VLA backbone (π0.5 or SmolVLA, both PaliGemma-based) is unchanged. The paper argues this is why pointmaps compose with pretrained VLAs where point-cloud inputs do not: PTv3 on point clouds does work (32.8% SR vs 34.7 for pointmap+add), but concat rather than add is required because the token layout differs, and the concat-fusion approach forfeits ~2 points on RoboCasa relative to the additive fusion that only pointmap-shaped inputs enable.

Results

Controlled RoboCasa study without pretraining (24 tasks): as camera-viewpoint randomization scales up, RGB baseline drops from 37.5% to 27.9% (−9.6), RGB+Pointmap only from 36.5% to 34.7% (−1.8). Full-scale RoboCasa with pretrained backbones: π0.5 55.3 → π0.5+Pointmap 62.9 (+7.6); SmolVLA baseline lifted by +4.2; π0.5+Pointmap beats all filed camera-aware and 3D-augmented baselines at parity. Real Franka FR3 over four tasks × 3 training viewpoints × 180 demos: at seen viewpoints, π0.5+Pointmap 78.3 vs π0.5 73.3 (+5.0, largest gain on pick-and-place +6.7); at the unseen held-out viewpoint, π0.5+Pointmap 66.7 vs π0.5 55.0 (+11.7, largest gain on stack-blocks +20.0 and pick-and-place +13.3). DP3 point-cloud baseline drops from 63.3 seen to 48.3 unseen (Δ −15) whereas π0.5+Pointmap drops from 78.3 to 66.7 (Δ −11.6), so pointmaps are also more viewpoint-robust than an explicit 3D point-cloud policy. Input-modality ablation isolates the pointmap effect from Plücker + depth alone (+3.1 SR); encoder-and-fusion ablation isolates the additive-fusion effect from concat (+4.0 SR); origin ablation isolates end-effector recentering from base-frame (+3.9 SR under viewpoint randomization).

Why it’s interesting

This is the cleanest filed instance to date of the “just give the VLA robot-frame 3D” hypothesis that sits between the four dominant recipes on VLA Models — action-pretraining-at-scale (π*0.6: a VLA That Learns From Experience (RECAP)), unified-VLM-pointing (Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models), clean-teleop-data (Spirit-v1.5: Clean Data Is the Enemy of Great Robot Foundation Models), and frozen-WFM+action-expert (μ₀: A Scalable 3D Interaction-Trace World Model / World Pilot: Steering Vision-Language-Action Models with World-Action Priors) — by making the input representation itself carry the frame transform rather than pushing it into the policy, into a value model, or into a peer world model. The controlled study is a sharper isolation of the viewpoint-variation-as-cause-of-hardness diagnosis than HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining‘s dataset-side ID/OOD split provided, and it composes with rather than replaces the camera-motion-as-action recipes on the viewpoint-as-action candidate theme (LIME: Learning Intent-aware Camera Motion from Egocentric Video and ActiveMimic: Egocentric Video Pretraining with Active Perception) — LIME chooses a good viewpoint, See-like-a-Robot makes the policy robust to whichever viewpoint it ends up at.