GPS: Geometric Primary Structure for Articulated Parts Perception in Robot Manipulation

TL;DR

GPS (Shanghai Jiao Tong; CVPR 2026 Findings) introduces a lightweight articulated-part representation — two axis anchor points plus one part point on the rotation/translation plane — as a middle ground between pose-based and point-flow-based affordance for robot manipulation. The representation is co-designed with a Meta Quest 3 VR annotation rig that registers the axis as a fixed virtual point in 3D space and tracks the human hand as a stable motion proxy, yielding 41K RGB-D frames over 234 objects across six part classes (Lid, Lid-thin, Lid-book, Handle, Door, Drawer) at ~1 minute per video. A single PointNet++-based GPS predictor trained on this data, plugged into a heuristic policy that re-scores AnyGrasp proposals and rolls out axis-constrained waypoints, hits 73% real-robot success across 270 trials on 9 unseen objects with no in-domain fine-tuning, beating CAPNet (pose-based, 33%) and GFlow (flow-based, 35%) under matched conditions.

Key claims

Articulated-part affordance is decomposed into three keypoints — two axis anchors q1, q2 and one part point p on the rotation plane — which captures the motion axis exactly while relaxing the contact-point constraint to a plane condition p · ((q1 − m) × (q2 − m)) = 0, making p easier to learn and easier to benchmark than the exact contact point m [§3.1, Eq. 2].
The same {q1, q2, p} triplet handles prismatic parts (drawers) by changing the geometric constraint to (p − m) · (q1 − q2) = 0 and fixing the axis through the part’s geometric center [§3.2, Eq. 4].
A portable Meta Quest 3 + RealSense D435 rig with built-in SLAM annotates the axis as a virtual point that stays fixed in 3D space across head motion, and tracks the midpoint between thumb and index finger as the part point — eliminating both per-frame manual annotation and the post-processing tracking pipeline that affordance methods like GFlow rely on [§4.1, Fig. 2 and Fig. 4].
The collected VR-GPS dataset is 41K frames over 234 objects across six part classes (Lid, Lid-thin, Lid-book, Handle, Door, Drawer), at ~1 minute per object-video, with three camera views per object, on ~$800 of hardware [§4.2, Fig. 5].
On HOI4D (out-of-domain), GPS beats the pose-based baseline CAPNet on all five evaluated categories — e.g. on Safe, APDE drops from 85.10° (CAPNet) to 33.75° (GPS), and on Laptop AADE drops from 37.33° to 17.17° [Table 1].
On RGBD-Art (also OOD), GPS beats the flow-based baseline GFlow by similarly wide margins — e.g. on Trashcan AADE 52.40° → 27.65° and APDE 120.00° → 20.79°; fine-tuning GFlow on the VR-GPS data (“Ours-Flow”) improves it substantially but still trails GPS, supporting the claim that the representation itself — not just the data — accounts for the gap [Table 2].
Real-robot manipulation across 9 objects × 30 trials each (270 states total, 6 camera views × 5 initial part poses): GPS reaches 73% overall success, vs CAPNet 33%, GFlow 35%, GPS-m (using exact contact point instead of relaxed part point) 58%, and human-annotated ground-truth GPS-GT 91% [Table 3].
Replacing the relaxed plane constraint with an exact-contact-point variant (GPS-m) drops success from 73% to 58%, confirming the ablation argument that loosening the constraint to a plane is what makes the predicted affordance robust enough to drive a grasp-scoring policy [Table 3, §6.3].
The policy itself is heuristic, not learned: predicted {q1, q2, p} re-score AnyGrasp proposals (penalizing grasps far from the plane spanned by GPS), then axis-constrained waypoints are rolled out using Eq. 1 / Eq. 3, with RRT* used only for kinematic-feasibility checks [§6.1, Fig. 6].

Method

The pipeline has three components: (i) a VR-based annotation rig, (ii) a single-image RGB-D GPS predictor, and (iii) a heuristic manipulation policy that consumes GPS predictions.

Annotation rig. A Meta Quest 3 headset with a wrist-mounted Intel RealSense D435 records RGB-D video while the annotator first places two virtual points along the part’s motion axis (these stay anchored in 3D world space via the headset’s SLAM) and then physically grasps and moves the part. The headset’s built-in hand-tracking renders the midpoint of thumb + index as a third virtual point, attached to the fingers. Coordinate transforms between the Quest’s world frame and the RealSense camera frame are applied post-recording, and SAM-2 segments the object. The result is, per object, three RGB-D videos from different headset views, each with frame-wise {q1, q2, p} labels and no manual post-processing.

Predictor. Following CAPNet, RGB features come from SAM-2 + FeatUp (480-dim per pixel) and are concatenated point-wise with the 3D point cloud. A CLIP text-encoder embedding of the part-class description is reduced to a 6-dim feature and concatenated. The fused features go through PointNet++, and three separate MLPs predict the axis direction û, axis anchor q̂1, and part point p̂. Losses are direction cosine losses on the axis and on the part-plane condition, with explicit axis offset and (for prismatic parts) part offset terms (Eqs. 5–8).

Policy. AnyGrasp produces K grasp proposals. Each proposal’s confidence is re-weighted by its distance to the GPS-predicted plane, so grasps geometrically incompatible with the predicted axis are demoted (Fig. 7’s box example: AnyGrasp’s most-confident grasp drops from rank-1 to rank-4 after re-scoring). The chosen initial grasp pose T1 is then rolled forward by repeated 5° rotations or 0.5 cm translations along the predicted axis, generating a waypoint sequence {T_t}. RRT* checks kinematic feasibility; infeasible trials are counted as failures (a strict evaluation choice).

Results

HOI4D OOD evaluation (Table 1): GPS dominates CAPNet on Laptop (AADE 17.17° vs 37.33°, APDE 9.26° vs 62.78°), Trashcan, Safe, Bucket, and Drawer (AADE 14.32° vs 37.82°, APOE 0.14 vs 0.12 — comparable on the offset, much better on direction).
RGBD-Art OOD evaluation (Table 2): GPS dominates GFlow by 2–3× on direction errors across all five categories, with the largest gaps on Trashcan (AADE 27.65° vs 52.40°, APDE 20.79° vs 120.00°) and Bucket (AADE 36.25° vs 59.76°, APDE 38.85° vs 128.74°).
“Ours-Flow” — GFlow’s architecture trained on the VR-GPS dataset — improves substantially over public-checkpoint GFlow but still trails GPS on every category, isolating the contribution of the representation itself.
Real-robot success rate (Table 3, 30 trials per object): Box 93%, Document-Box 90%, Bucket 67%, Door 67%, Drawer 60%, Notebook 73%, Folder 70%, Lamp 67%, Clapperboard 73% — overall 73%, vs GPS-GT (oracle perception) 91%, GPS-m 58%, GFlow 35%, CAPNet 33%.
The first 5 categories overlap with what was evaluated in Tables 1/2 (“Avg-overlap”); GPS scores 75% there, demonstrating that the OOD perception-benchmark gains transfer to closed-loop manipulation.
Sim-to-real and category transfer: GPS-GT itself caps the policy at 91%, meaning the remaining ~18 pp gap from GPS to GPS-GT is GPS-prediction error, while the floor (CAPNet at 33%) is roughly what synthetic-trained pose models achieve when shipped onto a real robot.

Why it’s interesting

GPS is the most evidence-backed instance to date of an intermediate-representation counter-recipe to the action-pretraining and unified-VLM camps in VLA Models: instead of training a large policy to predict actions end-to-end, it trains a small perception model to predict a tightly-constrained geometric abstraction, then drives a heuristic policy off it — and still gets 73% real-robot success on 9 unseen objects with no in-domain fine-tuning, in a regime where most VLAs are quoting LIBERO/SimplerEnv numbers. This complements Geometric Action Model for Robot Policy Learning (which similarly bets on a geometric foundation model as the policy backbone) and μ₀: A Scalable 3D Interaction-Trace World Model (which factors the policy as a frozen action-free trace world model + a small action expert) — all three argue, from different angles, that the right factorization is “rich geometry up front, simple action head behind.”

The VR annotation rig is also a sharp counterpoint to the human-video-to-robot pipelines tracked under Pose Estimation and Motion Capture (e.g. EgoInfinity: A Web-Scale 4D Hand-Object Interaction Data Engine for Any-View Robot Retargeting and Video-to-Action Robot Learning, Do as I Do: Dexterous Manipulation Data from Everyday Human Videos): rather than scaling up automated tracking on in-the-wild RGB and absorbing the tracking noise, GPS pays the cost of explicit human-in-the-loop annotation, but the annotation cost per object is only ~1 minute thanks to the VR axis-anchoring. As a Synthetic Training Data story this is closer to the “small, clean, geometric” school than the “scale via internet video” school, and the GPS-m ablation gives unusually clean evidence that the form of the supervision (relaxed plane vs exact contact point) matters more than its volume.