Skip to content
Bud's Wiki

Pose Estimation and Motion Capture

TL;DR

Section titled “TL;DR”

Recovering 2D/3D body pose and motion from images or video — humans, animals, and arbitrary rigged characters. The current wiki cluster spans three layers: (a) probabilistic 2D→3D pose lifters that generate multiple 3D hypotheses from a single frame (FMPose3D, via flow matching), (b) productized markerless monocular mocap that streams into DCC engines (CYANPUPPETS), and (c) category-agnostic full-mocap systems that take a video plus an arbitrary rigged asset as input and produce a BVH-style joint-rotation animation (MoCapAnything, via 3D-keypoint trajectories + IK). The cross-cutting question is how skeleton-agnostic conditioning composes with parametric body models like SMPL-X / MHR (SAM 3D Body: Robust Full-Body Human Mesh Recovery).

Key claims

Section titled “Key claims”
  • Flow matching is competitive as a probabilistic 3D-pose lifter. FMPose3D casts monocular 3D pose estimation as 2D-conditioned flow matching, generating multiple 3D hypotheses per input and aggregating them — competitive with diffusion-based lifters at much lower step count (FMPose3D: monocular 3D pose estimation via flow matching §3, Table 1).
  • Direct rotation regression is brittle in monocular settings. MoCapAnything explicitly argues that regressing joint rotations directly from monocular video is brittle because rotations are defined in asset-local frames, depth/camera motion entangles local angles, and per-frame regression yields poor temporal continuity — motivating a 3D-keypoints-then-IK factorization (MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos §3.2).
  • Mesh tokens carry more signal than dense RGB for joint localization on unusual morphologies. MoCapAnything ablates each input branch: removing the coarse 4D mesh stream raises Unseen MPJPE from 25.63 to 46.01, vs 41.57 for removing the appearance image set and 26.57 for removing graph multi-head attention on the skeleton (MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos Table 2).
  • Category-agnostic mocap is achievable from a single checkpoint. Same MoCapAnything model handles humans, quadrupeds, birds, reptiles, fish, and non-biological rigs, and supports zero-shot cross-species retargeting (bird flight → quadruped flapping, dog running → bipedal bird) (MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos §4.5–§4.8).
  • Real-time markerless mocap is shipping as a commercial product. CYANPUPPETS 1B claims a 1B-parameter real-time motion model that streams camera input directly to UE/Unity/Blender, indicating the latency × VRAM envelope for productized pose estimation in 2026 (CYANPUPPETS 1B real-time motion model (camera → UE/Unity/Blender)).
  • GenZoo (current animal-mocap SOTA) is template-locked to quadrupeds. MoCapAnything reports CD-Skeleton 0.4466 (GenZoo) vs 0.2354 (MoCapAnything) on quadrupeds and 0.4740 vs 0.2821 on non-quadrupeds — the gap widens precisely where GenZoo’s SMAL-style quadruped assumption breaks (MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos Table 1).
  • Parametric body models are a parallel axis. SAM 3D Body: Robust Full-Body Human Mesh Recovery introduces the Momentum Human Rig (MHR), which decouples skeleton from soft-tissue shape (unlike SMPL/SMPL-X), but stays locked to a human topology — complementary to the category-agnostic skeleton handling in MoCapAnything.

Recent contributions

Section titled “Recent contributions”
  • [2026-07-13] Video Generation Models are General-Purpose Vision Learners: GenCeption (DeepMind, ECCV 2026) reports SOTA on 3D keypoint prediction and camera pose using a single instruction-tuned text-to-video diffusion backbone — beating Sapiens on human keypoints and VGGT-Ω on camera pose within the same model that also handles depth, normals, and expression-referring segmentation. Also reports that a model fine-tuned only on synthetic human videos transfers zero-shot to animals and robots, a stronger category-transfer claim than MoCapAnything’s cross-species retargeting because it comes from a single video-generative pretraining prior rather than category-agnostic skeleton prompting.
  • [2026-07-12] 3D-Object Perception Transformer (3PT): 3PT (Intrinsic, CVPR 2026 Highlight) extends the cluster from human/animal skeletons to CAD-defined rigid industrial objects: a single 3D-conditioned two-tower ViT (early-fusion of CAD-render prior embeddings into image tokens) unifies zero-shot detection, segmentation, and 6-DoF pose in one system, matches multi-view RGB refinement against MV-PnP with reciprocal edge correspondences, and beats every zero-shot RGB-D method on BOP-Industrial (+29.1 AP-mm) using only RGB — even surpassing per-object-fine-tuned FRTPose-WAPRv2 (MV-RGB-D) by 12.3 AP-mm. Ranked first in 7 of 11 BOP Challenge 2025 tracks; trained entirely on ~1B synthetic Blender pairs from ~100K CAD meshes.
  • [2026-07-09] LIME: Learning Intent-aware Camera Motion from Egocentric Video: LIME extends flow-matching pose modeling from perception (2D→3D lifting, FMPose3D) to action: a continuous flow-matching head predicts a relative SE(3) target camera pose from an RGB view + language intent, with the multi-hypothesis property load-bearing because a single intent (e.g. “look under the shelf”) admits many valid poses. Supervision is mined at ~3M pairs from RoomTour3D + Nymeria egocentric video.
  • [2026-07-09] ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation: ProxyPose broadens the cluster from skeleton-topology methods (MoCapAnything, SAM 3D Body) to arbitrary-surface 6-DoF tracking with no rig or CAD-model assumption: a fine-tuned video diffusion model translates the input clip into a “proxy video” of a colored polyhedron matching the local rigid-body motion at a single query pixel, so 6-DoF trajectory recovery reduces to classical PnP; achieves SOTA on HO3D / YCBInEOAT / ProxyPose Synthetic and generalizes zero-shot to non-rigid faces, camera pose (where COLMAP fails), event-camera video, and SPAD arrays.
  • [2026-06-23] GPS: Geometric Primary Structure for Articulated Parts Perception in Robot Manipulation: GPS reframes articulated-object pose as a 3-keypoint task (two axis anchors + one part point on the rotation/translation plane) rather than 6-DoF segmentation+pose, and shows on HOI4D and RGBD-Art that this relaxed representation beats both pose-based (CAPNet) and flow-based (GFlow) baselines by 2–3× on direction errors — a task-specific simplification that complements category-agnostic full mocap (MoCapAnything) on the opposite end of the representation spectrum.
  • [2026-06-23] Jun Gao thread (4/5) on Cosmos2.5-2B finetuned with cleaned data + kinematic skeleton conditioning: Kinematic skeleton conditioning on a Cosmos2.5-2B backbone is reported to drive action-following gains in character video generation, a NVIDIA-side counterpart to the Wan-Animate / PoseAnything pose-conditioned line.
  • [2026-06-21] HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos: HumanEgo’s empirical claim that “the accuracy of the upstream SLAM and hand-tracking signal directly bounds downstream policy performance” makes Aria MPS (6-DoF SLAM + 21-keypoint hand pose) a load-bearing dependency of a minutes-of-data manipulation policy — a sharper precision-matters argument than prior egocentric-pretraining work which mostly leaned on data volume.
  • [2026-06-21] LAMP: Localization Aware Multi-camera People Tracking in Metric 3D World: LAMP (Meta Reality Labs, CVPR 2026) fills the egocentric multi-camera-headset gap: lifts per-camera ViTPose 2D keypoints into world-frame Plücker rays using the headset’s known 6-DoF VIO pose, then a spatio-temporal transformer fits SMPL directly — halves MPJPE vs PromptHMR on Nymeria (54.8 vs 109.2 mm) and transfers zero-shot from Aria Gen1 training to Aria Gen2.
  • [2026-06-19] EgoInfinity: A Web-Scale 4D Hand-Object Interaction Data Engine for Any-View Robot Retargeting and Video-to-Action Robot Learning: EgoInfinity is a modular 4D hand-object engine (perception + segmentation + reconstruction + interaction-aware refinement + retargeting) that lifts arbitrary in-the-wild RGB into metric hand trajectories, 6-DoF object poses, and contact states without human annotation — sibling to MoCapAnything’s category-agnostic mocap but specialized to hand-object interaction and any-view footage.
  • [2026-06-19] What Matters When Cotraining Robot Manipulation Policies on Everyday Human Videos?: TriHands ships 3D hand pose labels for 28 hours of natural cooking video via a multi-view exocentric triangulation pipeline, then provides downstream evidence that hand-label quality (not just scale) measurably affects transfer to robot manipulation policies — a concrete utility-for-VLAs argument for high-quality 3D hand annotation infra.
  • [2026-06-18] Do as I Do: Dexterous Manipulation Data from Everyday Human Videos: DO AS I DO extends the cluster from body/skeleton pose to 4D hand-object reconstruction (time-varying 3D hand pose + object pose + contact) from monocular RGB, claiming SOTA on hand-object interaction estimation against ground-truth datasets and on dexterous trajectory extraction from online video — handling both egocentric and exocentric in-the-wild footage with one algorithm.
  • [2026-06-18] ActiveMimic: Egocentric Video Pretraining with Active Perception: ActiveMimic recovers synchronized 6-DoF camera and wrist trajectories from a single body-worn RGB stream — no IMU, depth, or paired sensor rig — and shows the recovered trajectories are accurate enough to supervise downstream VLA pretraining that matches robot-data-pretrained baselines.
  • [2026-06-06] MAMMA: Markerless & Automatic Multi-Person Motion Action Capture: MAMMA (Black group + CMU, CVPR oral) — multi-view markerless SMPL-X capture with per-landmark transformer queries + SAM-2 mask conditioning; within 1.6 mm of Vicon-MoSh++ on held-out markers and visually indistinguishable from marker-based capture on two-person interaction. First filed paper to convincingly close the markerless-vs-marker gap for hands + close contact.
  • [2026-06-02] PoseAnything: Universal Pose-guided Video Generation with Part-aware Temporal Coherence: PoseAnything inverts the cluster’s perception focus to generation conditioned on arbitrary skeletons: XPose (50K clips) is built via Qwen-2.5-VL-7B filter + Grounded-SAM-2 masks + BlumNet skeleton extraction, providing the first public non-human pose-video dataset and an automated annotation pipeline that complements the perception-side category-agnostic mocap line (MoCapAnything).
  • [2026-06-03] HumanNet: Scaling Human-centric Video Learning to One Million Hours: HumanNet bundles hand- and body-related signals alongside ~1M h of egocentric and third-person video, positioning it as a candidate pretraining corpus for hand/body pose models at internet scale.
  • [2026-06-01] EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World: EgoVerse-A’s Aria Gen 1 captures supply accurate camera poses, 3D head trajectories, and 21-keypoint hand skeletons across 1,362 h / 2,087 demonstrators with shared protocols mirrored across four academic labs — a candidate pretraining corpus for egocentric pose models with controlled cross-lab variation.
  • [2026-05-25] WildPose: A Unified Framework for Robust Pose Estimation in the Wild: Widens the cluster from body/object pose to monocular camera pose: frozen MASt3R backbone + differentiable dense BA + learned motion masks; claims unified SOTA across dynamic (Wild-SLAM, Bonn), static (TUM, 7-Scenes), and low-ego-motion (Sintel) benchmarks.
  • [2026-05-25] MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos: Category-Agnostic Motion Capture (CAMoCap) from monocular video + arbitrary rigged asset prompt; factorizes into 3D-keypoint trajectories then IK; sets a new bar on Truebones Zoo (CD-Skeleton 0.2549 vs GenZoo 0.4580) and demonstrates zero-shot cross-species retargeting.
  • [2026-05-23] CYANPUPPETS 1B real-time motion model (camera → UE/Unity/Blender): CYANPUPPETS 1B real-time markerless motion model — camera → UE/Unity/Blender streaming pipeline (product signal for latency × VRAM tradeoff).
  • [2026-05-23] FMPose3D: monocular 3D pose estimation via flow matching: Flow-matching-based monocular 3D pose lifter; probabilistic, multi-hypothesis, competitive with diffusion lifters at lower step counts.

Open questions

Section titled “Open questions”
  • Does the category-agnostic skeleton prompting in MoCapAnything compose cleanly with a high-fidelity parametric body like MHR for the human-specific case, or does the IK constraint set need to be specialized?
  • For productized real-time mocap (CYANPUPPETS-class), where is the inflection between direct image→3D regressors and probabilistic 2D→3D lifters in the latency × accuracy frontier?
  • Should “pose estimation” live under World Foundation Models perception backbones (DINOv2 / VGGT / V-JEPA 2) as a downstream task, or is the rig/IK side enough of its own discipline that it warrants a separate stack?
  • How much of MoCapAnything’s transfer benefit comes from the coarse 4D-mesh bridge specifically, versus from any geometry-aware bridge between RGB and joint space (point clouds, depth, surface normals)?
  • Datasets are still small (Truebones Zoo at 1,038 clips). How does cross-species generalization scale with dataset diversity vs depth?

Papers

Section titled “Papers”

2026-02

Section titled “2026-02”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”