LAMP: Localization Aware Multi-camera People Tracking in Metric 3D World

TL;DR

LAMP (Meta Reality Labs, CVPR 2026) is a 3D human motion tracker designed for the egocentric multi-camera-headset setting (Project Aria Gen1/Gen2, smart glasses) where prior monocular trackers break down. The core idea is an “early world-space ray lifting” pipeline: per-camera 2D keypoints are unprojected into Plücker rays in a gravity-aligned world frame using the headset’s known 6-DoF VIO/SLAM pose, then a spatio-temporal transformer (LAMP-Net) fits SMPL motion directly to that ray cloud. Because the network never sees pixels, training data can be synthesized for any camera rig from any motion-capture dataset — Meta trains on Nymeria (Aria Gen1) and shows zero-shot transfer to Aria Gen2. On Nymeria’s egocentric setting LAMP roughly halves MPJPE versus PromptHMR (54.8 vs 109.2 mm) while running real-time on a single RTX 4090.

Key claims

Disentangling observer egomotion via known 6-DoF SLAM/VIO at input time is a substantially better factorization than learning to jointly estimate camera and human motion as in PromptHMR/TRAM/WHAM; on Nymeria with monocular input LAMP-mono beats PromptHMR by 16.9 mm MPJPE / 21.2 mm W-MPJPE [Table 1].
“Lift then fit” — per-camera 2D-keypoint detection (ViTPose on RF-DETR boxes) followed by SMPL fitting on the unified 3D ray cloud — naturally handles partial visibility, camera hand-offs and asynchronous multi-view input without an explicit late-fusion stage [§3.1, §3.4].
Because the network input is rays, not pixels, training data for an arbitrary multi-camera rig can be synthesized by projecting any 3D motion dataset through the rig’s calibration; this collapses the data-scarcity problem that has kept most prior work monocular [§4.1, §4.2].
Multi-view input is worth more than smoothing or world-space lifting alone: in the Nymeria ablation, going from monocular real keypoints (92.3 MPJPE / 80.4 W-MPJPE) to 3-camera real keypoints (54.8 / 58.7) gives the single largest jump in the table [Table 2].
The sim-to-real gap is small under the multi-view setup: simulated-vs-real keypoint inputs differ by 2.8 mm MPJPE multi-view (52.0 vs 54.8) but by 31.9 mm monocular (60.4 vs 92.3), suggesting multi-view geometric constraints absorb most of the keypoint-detector noise that would otherwise hurt monocular fitting [Table 2].
LAMP introduces W-MPJPE — MPJPE in the global world frame without any per-window Procrustes/translational alignment — as a metric better suited to scenarios where scale and global position are recoverable from calibrated multi-view rigs rather than ambiguous from monocular input [§4.1].
On EMDB (slow-moving monocular) LAMP-mono leads on world-space metrics (W-MPJPE 77.8 vs PromptHMR 63.9 is a regression, but RTE 165.1 vs 278.1 and jitter 0.2 vs 0.4 are SOTA) and trails on PA-MPJPE / per-frame local metrics — the trade-off attributed to collapsing pixels into sparse keypoint rays [§4.2, Table 1].

Method

LAMP separates 2D detection from 3D motion fitting via two factorizations. (1) Detection stage: per camera per timestamp, RF-DETR produces person boxes and ViTPose produces MS-COCO 2D keypoints; a bipartite Hungarian matcher associates detections with active tracklets by projecting each tracklet’s last 3D state into all cameras and computing match cost. (2) World-aligned ray lifting: each 2D keypoint is unprojected with the calibrated camera model, transformed by the camera’s known 6-DoF VIO pose into a gravity-aligned local frame anchored on the first frame of the temporal window, and stacked as a 6-D Plücker ray with detection confidence. The resulting tensor of shape (cameras × keypoints × time × 7) is fed to LAMP-Net: three spatio-temporal transformer encoder-decoder blocks with 256-D inner state and self-attention along both joint and frame axes. A learnable readout embedding plus time encoding queries a cross-attention decoder that interacts with every encoder layer (not just the last) and regresses SMPL parameters — 6-D rotation, shape, root translation, root rotation — per frame. The window is 4 s at 30 Hz (120 frames). At inference, sliding the window forward one frame at a time yields up to T predictions for each frame, which are averaged to reduce jitter (a free byproduct of the causal sliding scheme).

Results

EMDB-2 zero-shot (monocular): LAMP-mono RTE 165.1 mm vs PromptHMR 278.1 mm, jitter 0.2 vs 0.4 (Hz), W-MPJPE 77.8 vs 63.9 mm, MPJPE 82.3 vs 68.1, PA-MPJPE 46.3 vs 40.1 — world-trajectory metrics SOTA, local pose metrics a regression vs pixel-based PromptHMR [Table 1].
Nymeria (Aria Gen1 egocentric): LAMP-mono 92.3 MPJPE / 80.4 W-MPJPE / 203.4 RTE; PromptHMR 109.2 / 101.6 / 246.0. With 3 cameras (1 RGB + 2 SLAM): LAMP-mv 54.8 / 58.7 / 113.3 — large step from multi-view aggregation [Table 1].
Ablation on Nymeria (monocular, simulated 2D unless noted): no-pose lifting 98.5 MPJPE / 95.8 W-MPJPE / 296.3 RTE / 0.50 jitter / 93.1 FS; +world-pose-lifting 98.3 / 89.3 / 209.6 / 0.09 / 91.7; +sliding-window smoothing 92.3 / 80.4 / 203.4 / 23.8 FS; real keypoints (monocular) 60.4 / 71.7; real keypoints (multi-view) 52.0 / 58.2 [Table 2].
Generalization to Aria Gen2 (4-camera rig with ~280° FoV, never seen in training because Nymeria was captured on Gen1): qualitatively persists tracking across long trajectories and tracks multiple people across the wide FoV in real time [§4.2, Fig. 1]. The multi-camera coverage analysis over four 5-min Aria Gen2 Pilot sequences shows monotonic coverage gain 1→2→4 cameras during social interactions [Fig. 6].
Training cost: 200 epochs on 4 × H100 nodes (~19 h); inference and viser-based 3D visualization on a single RTX 4090 [§4.1].

Why it’s interesting

LAMP is an explicit bet that for posed, calibrated egocentric rigs the right factorization is to give up on pixel-level features and let an externally solved 6-DoF tracker do all the geometric heavy lifting. That bet pays off in two places the wiki has been actively tracking. First, on the Pose Estimation and Motion Capture axis it complements MAMMA: Markerless & Automatic Multi-Person Motion Action Capture — also a multi-view SMPL fitter, but for static studio rigs — and SAM 3D Body: Robust Full-Body Human Mesh Recovery which is the strongest single-image HMR; LAMP fills the egocentric moving-rig gap with much weaker per-frame signal but stronger temporal/multi-view fusion. Second, on the Synthetic Training Data axis, the keypoint-rays-only input means the entire training set is a function of (motion capture dataset × virtual rig calibration), which sidesteps the data-collection bottleneck that has kept egocentric trackers monocular and dovetails with EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World and ActiveMimic: Egocentric Video Pretraining with Active Perception as candidate real-data sources for fine-tuning. The W-MPJPE metric and the explicit “Procrustes alignment hides the multi-view scale advantage” argument is also a useful methodological correction for anyone benchmarking on EMDB-style data.