Skip to content
Bud's Wiki

Pose Estimation and Motion Capture

TL;DR

Section titled “TL;DR”

Recovering 2D/3D body pose and motion from images or video — humans, animals, and arbitrary rigged characters. The current wiki cluster spans three layers: (a) probabilistic 2D→3D pose lifters that generate multiple 3D hypotheses from a single frame (FMPose3D, via flow matching), (b) productized markerless monocular mocap that streams into DCC engines (CYANPUPPETS), and (c) category-agnostic full-mocap systems that take a video plus an arbitrary rigged asset as input and produce a BVH-style joint-rotation animation (MoCapAnything, via 3D-keypoint trajectories + IK). The cross-cutting question is how skeleton-agnostic conditioning composes with parametric body models like SMPL-X / MHR (SAM 3D Body: Robust Full-Body Human Mesh Recovery).

Key claims

Section titled “Key claims”
  • Flow matching is competitive as a probabilistic 3D-pose lifter. FMPose3D casts monocular 3D pose estimation as 2D-conditioned flow matching, generating multiple 3D hypotheses per input and aggregating them — competitive with diffusion-based lifters at much lower step count (FMPose3D: monocular 3D pose estimation via flow matching §3, Table 1).
  • Direct rotation regression is brittle in monocular settings. MoCapAnything explicitly argues that regressing joint rotations directly from monocular video is brittle because rotations are defined in asset-local frames, depth/camera motion entangles local angles, and per-frame regression yields poor temporal continuity — motivating a 3D-keypoints-then-IK factorization (MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos §3.2).
  • Mesh tokens carry more signal than dense RGB for joint localization on unusual morphologies. MoCapAnything ablates each input branch: removing the coarse 4D mesh stream raises Unseen MPJPE from 25.63 to 46.01, vs 41.57 for removing the appearance image set and 26.57 for removing graph multi-head attention on the skeleton (MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos Table 2).
  • Category-agnostic mocap is achievable from a single checkpoint. Same MoCapAnything model handles humans, quadrupeds, birds, reptiles, fish, and non-biological rigs, and supports zero-shot cross-species retargeting (bird flight → quadruped flapping, dog running → bipedal bird) (MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos §4.5–§4.8).
  • Real-time markerless mocap is shipping as a commercial product. CYANPUPPETS 1B claims a 1B-parameter real-time motion model that streams camera input directly to UE/Unity/Blender, indicating the latency × VRAM envelope for productized pose estimation in 2026 (CYANPUPPETS 1B real-time motion model (camera → UE/Unity/Blender)).
  • GenZoo (current animal-mocap SOTA) is template-locked to quadrupeds. MoCapAnything reports CD-Skeleton 0.4466 (GenZoo) vs 0.2354 (MoCapAnything) on quadrupeds and 0.4740 vs 0.2821 on non-quadrupeds — the gap widens precisely where GenZoo’s SMAL-style quadruped assumption breaks (MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos Table 1).
  • Parametric body models are a parallel axis. SAM 3D Body: Robust Full-Body Human Mesh Recovery introduces the Momentum Human Rig (MHR), which decouples skeleton from soft-tissue shape (unlike SMPL/SMPL-X), but stays locked to a human topology — complementary to the category-agnostic skeleton handling in MoCapAnything.

Recent contributions

Section titled “Recent contributions”
  • [2026-09-01] Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling: Lucida raises CA-1M ADD-SB@0.05 (strict surface-alignment 6-DoF pose) from 57.8% to 83.4% by treating placement as a VLM-driven closed-loop refinement rather than a feed-forward regression — the sharpest single-number gain on strict-alignment pose from a learned iterative-refinement recipe on this page. Complements ProxyPose’s video-to-video translation approach (single-object, generative-prior-based) with a scene-scale multi-object counterpart driven by an explicit tool-use policy over a 3D editor’s gizmo.
  • [2026-08-24] Compute-Optimal Scaling Laws for Human Motion Generation: Cartwheel’s motion-generation scaling-law paper depends on running a 3D human-reconstruction stack (“Comic” with physical QA) at ~423k-hour catalog scale, producing the 11,969-hour motion corpus that all downstream scaling fits run on. Puts pose estimation squarely as the bottleneck upstream stage for motion generation the same way GVHMR is for HY-Motion 1.0: Scaling Flow Matching Models for Text-To-Motion Generation and MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos would be if its category-agnostic mocap were plugged in place of Comic. First filed datapoint that quantifies what a curated pose-estimation pipeline yields in effective corpus hours — 5.56M clips survive from a 423k-hour raw catalog after all quality gates.
  • [2026-08-20] OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interaction: OakInk provides 230K RGB-D frames with 3D MANO hand pose + 6-DoF object pose + per-part contact regions across 12 subjects × 100 objects × up to 5 intents; hand pose is obtained by crowd-sourced multi-view 2D keypoints then MANO-layer optimization against multi-view reprojection + geometric consistency + temporal smoothing + silhouette constraints. Cross-dataset training with HO3D improves DexYCB test MPJPE — a load-bearing supervision substrate for the RGB-based hand-pose family on this page.
  • [2026-08-17] Yu Xiang — 12 hand-object interaction datasets worth knowing for robot learning: Yu Xiang (IRVL) posts a curated 12-item directory of hand-object interaction datasets (FreiHAND, RHD, HO-Cap, HO3D/H2O-3D, ObMan, DexYCB, HOI4D, OakInk v1/v2, ARCTIC, PALM, TACO) with download + code links per entry, positioning cross-dataset annotation comparability (pose formats, camera models, coordinate frames) as the current bottleneck rather than data collection.
  • [2026-08-13] HOT3D — Egocentric Multi-View Dataset for 3D Hand-Object Interaction Understanding: HOT3D (Meta Reality Labs / Project Aria) is a dataset landing page for the >1M-multi-view-frame egocentric hand-object benchmark that several filed pose papers already report headline numbers on (MolmoMotion: Language-guided 3D motion forecasting, ReViV: Reconstructing the Viewer and the View in 4D from Monocular Egocentric Video); dual Project Aria + Quest 3 capture with mocap 3D ground truth for hands + 33 PBR-scanned objects and Aria gaze — a substrate slot no prior filed dataset fills at this fidelity.
  • [2026-08-08] MotionBricks: Scalable Real-Time Motions with Modular Latent Generative Model and Smart Primitives: MotionBricks (NVIDIA, TOG 2026) adds the second filed generation-side character-motion system after HY-Motion 1.0 and takes the opposite architectural bet: a VQ-token-based MaskGIT-style backbone (structured multi-head tokenizer + root/pose disentanglement) with all control diversity absorbed by a “smart primitives” authoring layer above the network, rather than a monolithic text-conditioned MMDiT. Runs at 15,000 FPS with 2 ms latency on 350k-clip corpus, and deploys zero-shot from character animation onto a Unitree G1 humanoid — a distinct bridge to physical robot control from ScaleBFM’s PPO-on-retargeted-mocap or ReActor’s physics-in-loop kinematic retargeting.
  • [2026-07-25] ReViV: Reconstructing the Viewer and the View in 4D from Monocular Egocentric Video: ReViV (Lyu, Li, Qian, Zhang, Pollefeys, Tang — ETH + TU Delft + Microsoft) unifies six egocentric modalities (RGB, camera, gaze, body, hand, depth) into a single Masked Generative Egocentric Transformer that emits all six jointly from a monocular first-person video in one feed-forward pass. Removes the pre-computed-camera dependency that HumanEgo-style pipelines require, adds gaze to the cluster (which no prior filed pose paper handles), and reports SOTA across HoloAssist, HOT3D, ARCTIC, Aria Digital Twin, and TACO — the first joint-generative treatment of first-person perception on file.
  • [2026-07-22] Introducing Perceptron Mk1: Perceptron Mk1 productizes hand pose as a first-class VLM output — per-frame bounding boxes + 21-keypoint skeletons (wrist + 4 joints/finger) with left/right identity, tracked through video — as a sidecar on top of a video/embodied-reasoning VLM. First filed closed-source vendor offering in the hand-tracking-as-API slot that HumanEgo and MoCapAnything occupy research-side.
  • [2026-07-21] Lifting Embodied World Models for Planning and Control: Lifted World Model (Wang, Darrell, Izmailov, Bai, Bar — ECCV 2026) inverts the usual pose-estimation direction: forward-kinematics + camera-matrix projection of future XSens leaf-joint 3D positions into the current egocentric frame is used to generate ground-truth 2D “waypoints” for the pelvis/head/hands, which then serve as a low-dim goal-conditioning signal for a diffusion policy. A data point for the pipeline layer of the cluster (given a rigged embodiment + camera, project rather than regress) and a concrete argument that ~8-dim projected 2D goals beat 12-dim 3D goals when the downstream consumer is CEM search.
  • [2026-07-13] Video Generation Models are General-Purpose Vision Learners: GenCeption (DeepMind, ECCV 2026) reports SOTA on 3D keypoint prediction and camera pose using a single instruction-tuned text-to-video diffusion backbone — beating Sapiens on human keypoints and VGGT-Ω on camera pose within the same model that also handles depth, normals, and expression-referring segmentation. Also reports that a model fine-tuned only on synthetic human videos transfers zero-shot to animals and robots, a stronger category-transfer claim than MoCapAnything’s cross-species retargeting because it comes from a single video-generative pretraining prior rather than category-agnostic skeleton prompting.
  • [2026-07-12] 3D-Object Perception Transformer (3PT): 3PT (Intrinsic, CVPR 2026 Highlight) extends the cluster from human/animal skeletons to CAD-defined rigid industrial objects: a single 3D-conditioned two-tower ViT (early-fusion of CAD-render prior embeddings into image tokens) unifies zero-shot detection, segmentation, and 6-DoF pose in one system, matches multi-view RGB refinement against MV-PnP with reciprocal edge correspondences, and beats every zero-shot RGB-D method on BOP-Industrial (+29.1 AP-mm) using only RGB — even surpassing per-object-fine-tuned FRTPose-WAPRv2 (MV-RGB-D) by 12.3 AP-mm. Ranked first in 7 of 11 BOP Challenge 2025 tracks; trained entirely on ~1B synthetic Blender pairs from ~100K CAD meshes.
  • [2026-07-09] LIME: Learning Intent-aware Camera Motion from Egocentric Video: LIME extends flow-matching pose modeling from perception (2D→3D lifting, FMPose3D) to action: a continuous flow-matching head predicts a relative SE(3) target camera pose from an RGB view + language intent, with the multi-hypothesis property load-bearing because a single intent (e.g. “look under the shelf”) admits many valid poses. Supervision is mined at ~3M pairs from RoomTour3D + Nymeria egocentric video.
  • [2026-07-09] ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation: ProxyPose broadens the cluster from skeleton-topology methods (MoCapAnything, SAM 3D Body) to arbitrary-surface 6-DoF tracking with no rig or CAD-model assumption: a fine-tuned video diffusion model translates the input clip into a “proxy video” of a colored polyhedron matching the local rigid-body motion at a single query pixel, so 6-DoF trajectory recovery reduces to classical PnP; achieves SOTA on HO3D / YCBInEOAT / ProxyPose Synthetic and generalizes zero-shot to non-rigid faces, camera pose (where COLMAP fails), event-camera video, and SPAD arrays.
  • [2026-06-23] GPS: Geometric Primary Structure for Articulated Parts Perception in Robot Manipulation: GPS reframes articulated-object pose as a 3-keypoint task (two axis anchors + one part point on the rotation/translation plane) rather than 6-DoF segmentation+pose, and shows on HOI4D and RGBD-Art that this relaxed representation beats both pose-based (CAPNet) and flow-based (GFlow) baselines by 2–3× on direction errors — a task-specific simplification that complements category-agnostic full mocap (MoCapAnything) on the opposite end of the representation spectrum.
  • [2026-06-23] Jun Gao thread (4/5) on Cosmos2.5-2B finetuned with cleaned data + kinematic skeleton conditioning: Kinematic skeleton conditioning on a Cosmos2.5-2B backbone is reported to drive action-following gains in character video generation, a NVIDIA-side counterpart to the Wan-Animate / PoseAnything pose-conditioned line.
  • [2026-06-21] HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos: HumanEgo’s empirical claim that “the accuracy of the upstream SLAM and hand-tracking signal directly bounds downstream policy performance” makes Aria MPS (6-DoF SLAM + 21-keypoint hand pose) a load-bearing dependency of a minutes-of-data manipulation policy — a sharper precision-matters argument than prior egocentric-pretraining work which mostly leaned on data volume.
  • [2026-06-21] LAMP: Localization Aware Multi-camera People Tracking in Metric 3D World: LAMP (Meta Reality Labs, CVPR 2026) fills the egocentric multi-camera-headset gap: lifts per-camera ViTPose 2D keypoints into world-frame Plücker rays using the headset’s known 6-DoF VIO pose, then a spatio-temporal transformer fits SMPL directly — halves MPJPE vs PromptHMR on Nymeria (54.8 vs 109.2 mm) and transfers zero-shot from Aria Gen1 training to Aria Gen2.
  • [2026-06-19] EgoInfinity: A Web-Scale 4D Hand-Object Interaction Data Engine for Any-View Robot Retargeting and Video-to-Action Robot Learning: EgoInfinity is a modular 4D hand-object engine (perception + segmentation + reconstruction + interaction-aware refinement + retargeting) that lifts arbitrary in-the-wild RGB into metric hand trajectories, 6-DoF object poses, and contact states without human annotation — sibling to MoCapAnything’s category-agnostic mocap but specialized to hand-object interaction and any-view footage.
  • [2026-06-19] What Matters When Cotraining Robot Manipulation Policies on Everyday Human Videos?: TriHands ships 3D hand pose labels for 28 hours of natural cooking video via a multi-view exocentric triangulation pipeline, then provides downstream evidence that hand-label quality (not just scale) measurably affects transfer to robot manipulation policies — a concrete utility-for-VLAs argument for high-quality 3D hand annotation infra.
  • [2026-06-18] Do as I Do: Dexterous Manipulation Data from Everyday Human Videos: DO AS I DO extends the cluster from body/skeleton pose to 4D hand-object reconstruction (time-varying 3D hand pose + object pose + contact) from monocular RGB, claiming SOTA on hand-object interaction estimation against ground-truth datasets and on dexterous trajectory extraction from online video — handling both egocentric and exocentric in-the-wild footage with one algorithm.
  • [2026-06-18] ActiveMimic: Egocentric Video Pretraining with Active Perception: ActiveMimic recovers synchronized 6-DoF camera and wrist trajectories from a single body-worn RGB stream — no IMU, depth, or paired sensor rig — and shows the recovered trajectories are accurate enough to supervise downstream VLA pretraining that matches robot-data-pretrained baselines.
  • [2026-06-06] MAMMA: Markerless & Automatic Multi-Person Motion Action Capture: MAMMA (Black group + CMU, CVPR oral) — multi-view markerless SMPL-X capture with per-landmark transformer queries + SAM-2 mask conditioning; within 1.6 mm of Vicon-MoSh++ on held-out markers and visually indistinguishable from marker-based capture on two-person interaction. First filed paper to convincingly close the markerless-vs-marker gap for hands + close contact.
  • [2026-06-02] PoseAnything: Universal Pose-guided Video Generation with Part-aware Temporal Coherence: PoseAnything inverts the cluster’s perception focus to generation conditioned on arbitrary skeletons: XPose (50K clips) is built via Qwen-2.5-VL-7B filter + Grounded-SAM-2 masks + BlumNet skeleton extraction, providing the first public non-human pose-video dataset and an automated annotation pipeline that complements the perception-side category-agnostic mocap line (MoCapAnything).
  • [2026-06-03] HumanNet: Scaling Human-centric Video Learning to One Million Hours: HumanNet bundles hand- and body-related signals alongside ~1M h of egocentric and third-person video, positioning it as a candidate pretraining corpus for hand/body pose models at internet scale.
  • [2026-06-01] EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World: EgoVerse-A’s Aria Gen 1 captures supply accurate camera poses, 3D head trajectories, and 21-keypoint hand skeletons across 1,362 h / 2,087 demonstrators with shared protocols mirrored across four academic labs — a candidate pretraining corpus for egocentric pose models with controlled cross-lab variation.
  • [2026-05-25] WildPose: A Unified Framework for Robust Pose Estimation in the Wild: Widens the cluster from body/object pose to monocular camera pose: frozen MASt3R backbone + differentiable dense BA + learned motion masks; claims unified SOTA across dynamic (Wild-SLAM, Bonn), static (TUM, 7-Scenes), and low-ego-motion (Sintel) benchmarks.
  • [2026-05-25] MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos: Category-Agnostic Motion Capture (CAMoCap) from monocular video + arbitrary rigged asset prompt; factorizes into 3D-keypoint trajectories then IK; sets a new bar on Truebones Zoo (CD-Skeleton 0.2549 vs GenZoo 0.4580) and demonstrates zero-shot cross-species retargeting.
  • [2026-05-23] CYANPUPPETS 1B real-time motion model (camera → UE/Unity/Blender): CYANPUPPETS 1B real-time markerless motion model — camera → UE/Unity/Blender streaming pipeline (product signal for latency × VRAM tradeoff).
  • [2026-05-23] FMPose3D: monocular 3D pose estimation via flow matching: Flow-matching-based monocular 3D pose lifter; probabilistic, multi-hypothesis, competitive with diffusion lifters at lower step counts.

Open questions

Section titled “Open questions”
  • Does the category-agnostic skeleton prompting in MoCapAnything compose cleanly with a high-fidelity parametric body like MHR for the human-specific case, or does the IK constraint set need to be specialized?
  • For productized real-time mocap (CYANPUPPETS-class), where is the inflection between direct image→3D regressors and probabilistic 2D→3D lifters in the latency × accuracy frontier?
  • Should “pose estimation” live under World Foundation Models perception backbones (DINOv2 / VGGT / V-JEPA 2) as a downstream task, or is the rig/IK side enough of its own discipline that it warrants a separate stack?
  • How much of MoCapAnything’s transfer benefit comes from the coarse 4D-mesh bridge specifically, versus from any geometry-aware bridge between RGB and joint space (point clouds, depth, surface normals)?
  • Datasets are still small (Truebones Zoo at 1,038 clips). How does cross-species generalization scale with dataset diversity vs depth?

Papers

Section titled “Papers”

2026-02

Section titled “2026-02”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”