MAMMA: Markerless & Automatic Multi-Person Motion Action Capture

TL;DR

MAMMA is a multi-view markerless motion-capture pipeline from Max Planck (Black group) + CMU that fits SMPL-X to two-person interaction sequences with accuracy within 1.6 mm of a commercial marker-based Vicon system. It replaces sparse-keypoint triangulation with a transformer-based dense-landmark detector (MammaNet) that uses one learnable query per surface landmark, conditioned on SAM 2 segmentation masks to disambiguate overlapping subjects, and predicts per-landmark uncertainty plus visibility for downstream optimization. Training is purely synthetic — MAMMASyn extends BEDLAM with two-person dancing/interaction motions, hand-focused subsets (Interhand2.6M, SignAvatars), and a 32-camera digital-twin renderer — and a three-stage L-BFGS fit recovers SMPL-X pose, shape, and translation per frame. Code, weights, datasets, and benchmarks will be released for research; this is the talk Black called his “biggest day in 30 years.”

Key claims

Dense per-landmark queries beat single-token landmark heads on close-interaction scenes: MammaNet (one learnable query per landmark) reaches 7.81 px 2D error with masks on RICH vs 8.85 px for CameraHMR’s single-token design, and 13.96 px vs 21.53 px on the Harmony4D two-person benchmark [§5.1, Table 1].
Segmentation-mask conditioning is the dominant lever on two-person disambiguation, not on single-person: with-masks vs without-masks shaves only 0.74 px on RICH but 7.12 px on Harmony4D [§5.1, Table 1].
Markerless capture is now competitive with marker-based “ground truth” — held-out marker error vs Vicon-MoSh++ differs by only 1.6 mm on MAMMAEval-Extra, and the resulting animations are described as visually indistinguishable [§1, §5.3].
Synthetic-only training generalizes to real video: MammaNet+masks trained on MAMMASyn-ALL+BEDLAM hits 23.60 mm MPJPE / 20.14 mm PVE on RICH vs 27.52 / 22.30 for CameraHMR, with no real training data [Table 4].
Synthetic-data sources compose, not just augment: MAMMASyn-ALL+B (all subsets + BEDLAM) beats every single subset across all six eval benchmarks, including MOYO where BEDLAM* alone is 11.04 px and MAMMASyn-ALL+B reaches 6.12 px [Table 2].
Per-landmark visibility prediction matters because uncertainty alone goes confidently wrong on occluded parts during interaction — the paper introduces an explicit visibility head trained with BCE alongside the standard Gaussian-NLL landmark loss [§4.1, Eq. 1-2].
Mesh-silhouette IoU on Harmony4D exceeds the dataset’s own published pipeline: 73.35% vs 69.80% (Harmony4D pseudo-GT via SAM2) — MAMMA’s projected mesh fits people’s silhouettes better than the dataset that pseudo-labels its own test set [§5.3].
Cross-dataset generalization on Hi4D (SMPL, not SMPL-X) holds after removing Hi4D-motion synthetic clips and retraining — outperforms prior multi-person fits by a large margin under a 19-joint SMPL evaluation protocol [§5.3, Table 3].
Method is a two-stage approach: predict dense 2D landmarks per view, then fit SMPL-X by minimizing reprojection error with L-BFGS in three stages (translation+rotation only with reprojection loss, then pose+shape+translation under Geman-McClure, then a Huber refinement) [§4.2, Eq. 3-4].
Backbone is ViT-Base initialized from ViTPose-B with positional-embedding interpolation to 768² input, plus a CNN that encodes the segmentation mask into the same feature space and is fused by element-wise summation à la SAM [§4.1, §5.1].
MAMMASyn is a digital-twin of the authors’ physical 32-camera rig — calibration extrinsics/intrinsics are reused so synthetic and real share the same projection geometry, with 8 views per sample selected via FPS across the 32 cameras [§3.1].
Two new real eval datasets are introduced: MAMMAEval-Singles (3 subjects × 22 sequences), MAMMAEval-Dance (West Coast Swing couple, 17 sequences), and MAMMAEval-Extra (4 subjects × 4 sequences with 37 extra markers held out from MoSh++ fitting, to give independent ground truth) [§3.2].
Per-landmark loss weights w_i and visibility losses are weighted per-landmark; landmark sampling weights extra vertices toward hands, feet, and head — explicit acknowledgement that SMPL-X surface uniformity hurts articulated regions [§3.1, Fig. 5; §4.1].

Method

MAMMA is a two-stage pipeline: dense-landmark prediction per camera view, then multi-view SMPL-X optimization. The landmark detector, MammaNet, takes a cropped RGB image (768²) and an optional segmentation mask. The RGB goes through a ViTPose-B-initialized ViT-Base; the mask through a small CNN; the two are projected to the same embedding space and fused by element-wise summation (the same SAM-style mask-conditioning trick). The decoder is a transformer with one learnable query embedding per surface landmark — unlike CameraHMR’s single-token decode-and-MLP scheme — so each landmark cross-attends to the image patches relevant to its body part while self-attention layers learn pairwise landmark correlations. For each landmark i the head predicts pixel location (x_i, y_i), log-variance uncertainty σ_i, and visibility probability v_i. The landmark loss is a per-landmark-weighted Gaussian NLL on (x, y, σ); visibility is supervised with per-landmark-weighted BCE against ground-truth visibility.

The 512 landmark locations are sampled on the SMPL-X mesh by Farthest Point Sampling with up-weighting on hands, feet, and head (the small, articulated regions where uniform mesh-area sampling under-allocates).

Training data is MAMMASyn, an extension of BEDLAM. The authors build a digital twin of their 32-camera rig (matching extrinsics, intrinsics, focal lengths) inside the BEDLAM render pipeline, capturing a 3³ m³ virtual volume. Sequences are populated with 2–6 SMPL-X subjects (100 skin textures × 1,700 garment textures × 95 HDR backgrounds) and three subsets are mixed:

MAMMASyn-S (singles): BEDLAM motions augmented with MOYO yoga poses.
MAMMASyn-I (interactions): Hi4D, Harmony4D, Inter-X train splits plus two new Vicon-captured datasets — Latin-Dance (10 sequences) and Interacting Couples (48 sequences).
MAMMASyn-H (hands): Interhand2.6M and SignAvatars hand motions, SMPL-X-fit and merged with BEDLAM body poses.

Total: 2,800 sequences (5.5 h) × 8 views (one random + 7 FPS-selected), rendered at 6 fps and 2056×1504 — roughly 2× BEDLAM’s resolution to capture hand detail — yielding 955k images, ~2.5M bounding boxes. Per-subject segmentation masks, depth, and per-vertex visibility are saved alongside.

Inference fits SMPL-X (16 shape coefficients) to multi-view landmark predictions in three L-BFGS stages: (1) translation + rotation only, against raw reprojection; (2) translation + rotation + pose + shape under Geman-McClure robust loss; (3) pose refinement under Huber loss (less aggressive on the now-small residuals). The energy is E = λ_repro · E_reproj + λ_smooth · E_smooth + λ_β · ||β||² + λ_θ · ||θ||², with E_reproj weighting each landmark by predicted uncertainty and visibility — a landmark that’s marked invisible doesn’t pull the body toward an occluded view. Bodies are initialized with arms in an “L” pose at the 3D point that best minimizes camera-ray distances; no regression-based pose initializer is used.

Results

Landmark accuracy (2D error, pixels) — MammaNet+masks: 7.81 (RICH), 13.96 (Harmony4D, two-person), 6.80 (CHI3D), 6.94 (MAMMAEval-Singles), 6.51 (MAMMAEval-Dance), 10.18 (MOYO). The Harmony4D and CHI3D gaps against CameraHMR (21.53 / 8.88) and Look-Ma* (22.64 / 13.20) confirm that masks plus per-landmark queries are doing most of the work on close interaction [Table 1].
3D fit accuracy (MPJPE / PVE, mm) — MammaNet+masks on MAMMASyn-ALL+B: 23.60 / 20.14 (RICH), 42.23 / 31.48 (Harmony4D), 31.0 / 31.14 (CHI3D), 16.27 / 19.61 (MAMMAEval-S), 17.02 / 18.39 (MAMMAEval-D), 22.66 / 25.34 (MOYO). Beats CameraHMR (27.52 / 22.30 on RICH) and SMPLify-X-multi-view (96.18 / 71.42) across the board [Table 4].
Vs gold-standard Vicon-MoSh++ — On MAMMAEval-Extra (where 37 extra markers per subject are held out from the marker-based fit), MAMMA’s held-out marker error is within 1.6 mm of MoSh++‘s [§1, §5.3] — the headline “indistinguishable from marker-based capture” claim.
Silhouette IoU on Harmony4D — 73.35% vs 69.80% (Harmony4D’s own pipeline using SAM2 silhouettes as pseudo-GT) [§5.3, Fig. 9].
Hi4D cross-dataset (SMPL eval) — MAMMA outperforms prior multi-person methods by a large margin under the Lu et al. 2025 19-joint protocol after dropping Hi4D-motion synthetic clips from training [§5.3, Table 3].
Data ablations — Training longer on BEDLAM alone (BEDLAM*) degrades slightly vs the converged BEDLAM model; adding any MAMMASyn subset improves MOYO accuracy (BEDLAM 11.04 → MAMMASyn-S 6.08 px); the full mix (MAMMASyn-ALL+B) is best on every benchmark [Table 2].

Why it’s interesting

MAMMA is the strongest recent argument that the markerless-vs-marker gap for SMPL-X capture has essentially closed for the cases that matter to a generative-video team — close two-person interaction with hand detail. The 1.6 mm gap to a Vicon-MoSh++ pipeline on held-out markers, plus visually-indistinguishable animations, means a body of ground-truth training data for Audio-Driven Character Animation and LPM 1.0: Video-based Character Performance Model-style character-performance models can now be captured at the cost of a multi-camera rig + automated pipeline, with no marker cleanup. For the Pose Estimation and Motion Capture cluster this is a complement, not a competitor, to SAM 3D Body: Robust Full-Body Human Mesh Recovery: SAM 3D Body is a single-image promptable HMR with a new body rig (MHR) and weak hand fidelity; MAMMA is multi-view, SMPL-X-locked, and explicitly excellent on hands and contact. The two would compose — SAM 3 mask → MAMMA dense-landmark + mask conditioning — and the SAM-3-as-prompt path is what MAMMA already uses for SAM 2 silhouettes. The MammaNet architecture choice (one learnable query per landmark) is also a concrete instance of a recurring theme: per-output learnable embeddings beat single-token decoders when the output set is structured and large. MAMMASyn’s BEDLAM-extension recipe is the latest entry in the synthetic-training-data line (L2P: Unlocking Latent Potential for Pixel Generation, Seedance 2.0: Advancing Video Generation for World Complexity) where a digital twin of the actual capture rig is the unlock; pure-synthetic training generalizes here because rendered geometry matches the inference rig’s projection.