Skip to content
Bud's Wiki

Pose Estimation and Motion Capture

TL;DR

Section titled “TL;DR”

Recovering 2D/3D body pose and motion from images or video — humans, animals, and arbitrary rigged characters. The current wiki cluster spans three layers: (a) probabilistic 2D→3D pose lifters that generate multiple 3D hypotheses from a single frame (FMPose3D, via flow matching), (b) productized markerless monocular mocap that streams into DCC engines (CYANPUPPETS), and (c) category-agnostic full-mocap systems that take a video plus an arbitrary rigged asset as input and produce a BVH-style joint-rotation animation (MoCapAnything, via 3D-keypoint trajectories + IK). The cross-cutting question is how skeleton-agnostic conditioning composes with parametric body models like SMPL-X / MHR (SAM 3D Body: Robust Full-Body Human Mesh Recovery).

Key claims

Section titled “Key claims”
  • Flow matching is competitive as a probabilistic 3D-pose lifter. FMPose3D casts monocular 3D pose estimation as 2D-conditioned flow matching, generating multiple 3D hypotheses per input and aggregating them — competitive with diffusion-based lifters at much lower step count (FMPose3D: monocular 3D pose estimation via flow matching §3, Table 1).
  • Direct rotation regression is brittle in monocular settings. MoCapAnything explicitly argues that regressing joint rotations directly from monocular video is brittle because rotations are defined in asset-local frames, depth/camera motion entangles local angles, and per-frame regression yields poor temporal continuity — motivating a 3D-keypoints-then-IK factorization (MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos §3.2).
  • Mesh tokens carry more signal than dense RGB for joint localization on unusual morphologies. MoCapAnything ablates each input branch: removing the coarse 4D mesh stream raises Unseen MPJPE from 25.63 to 46.01, vs 41.57 for removing the appearance image set and 26.57 for removing graph multi-head attention on the skeleton (MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos Table 2).
  • Category-agnostic mocap is achievable from a single checkpoint. Same MoCapAnything model handles humans, quadrupeds, birds, reptiles, fish, and non-biological rigs, and supports zero-shot cross-species retargeting (bird flight → quadruped flapping, dog running → bipedal bird) (MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos §4.5–§4.8).
  • Real-time markerless mocap is shipping as a commercial product. CYANPUPPETS 1B claims a 1B-parameter real-time motion model that streams camera input directly to UE/Unity/Blender, indicating the latency × VRAM envelope for productized pose estimation in 2026 (CYANPUPPETS 1B real-time motion model (camera → UE/Unity/Blender)).
  • GenZoo (current animal-mocap SOTA) is template-locked to quadrupeds. MoCapAnything reports CD-Skeleton 0.4466 (GenZoo) vs 0.2354 (MoCapAnything) on quadrupeds and 0.4740 vs 0.2821 on non-quadrupeds — the gap widens precisely where GenZoo’s SMAL-style quadruped assumption breaks (MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos Table 1).
  • Parametric body models are a parallel axis. SAM 3D Body: Robust Full-Body Human Mesh Recovery introduces the Momentum Human Rig (MHR), which decouples skeleton from soft-tissue shape (unlike SMPL/SMPL-X), but stays locked to a human topology — complementary to the category-agnostic skeleton handling in MoCapAnything.

Recent contributions

Section titled “Recent contributions”
  • [2026-11-06] Fast and Accurate Hand Estimation in Pixels, Interaction and World Space: FAH (Cao, solo) is the open-source predict-points-then-fit-MANO/MHR counterpart to EgoHand v1.0 — NVIDIA monocular 3D hand pose + MHR/SOMA foundation model for robotics EgoHand v1.0’s parameter-regression recipe — fine-tunes SAM 3D Body’s hand branch into 2D/3D joints + surface points + optional parts/contact (no object geometry at inference) + camera-in-world + causal temporal refinement; ViT-L hits 65 fps with mesh fitting on an RTX 4070 Ti SUPER, ViT-S hits 125 fps without fitting (30 FPS on a MacBook webcam per author). Argues the parameter-regression default makes predicted 2D and 3D joints inconsistent and that fit-to-points inverts the error budget — same architectural bet as MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos MoCapAnything at the body-mocap level.
  • [2026-11-05] Perceptron Egocentric 1.0 — self-service API with SOTA hand tracking and EgoHome benchmark: Perceptron Egocentric 1.0 announces a new 3D hand reconstruction model reported 12–25% more accurate than ACE-Ego-Hand on HOT3D and ARCTIC (with and without camera calibration) — the closed-source productized peer to NVIDIA’s open-weights EgoHand v1.0 — NVIDIA monocular 3D hand pose + MHR/SOMA foundation model for robotics; also productizes per-hand-per-gripper action-label timelines as a first-class API output.
  • [2026-11-04] EgoHand v1.0 — NVIDIA monocular 3D hand pose + MHR/SOMA foundation model for robotics: EgoHand v1.0 (NVIDIA) productizes monocular 3D hand pose + MHR/SOMA mesh recovery as an 883.5M-parameter open-weights foundation model: DINOv3 backbone initialized from SAM 3D Body + TransformerDecoder with learnable pose/shape/camera queries, predicting full-perspective camera translation conditioned on input intrinsics (not weak-perspective crop camera). Trained on ~14M images (~5M synthetic from NVIDIA’s RigPlay/DigitalHuman + ~9M in-the-wild/mocap) + ~126 h video with 2D+3D-keypoint + MHR/SOMA parameter losses; evaluated on HOT3D and H2O, qualitative-only on EgoDex. Explicitly positioned as upstream training-supervision infrastructure for robot foundation models — the public-weights counterpart to the in-house pipelines that Natura-1 (41% MPJPE reduction vs open baselines), EgoScale (22-DoF retargeting), and Dyna-2 (wrist-only at 1M+ h) all depend on. OpenMDW 1.1 + Apache 2.0 release; 25,053 downloads at filing with Benchmark Score undisclosed.
  • [2026-10-27] AHa-3D: Agentic Tool Use for Real2Sim with GPT-6 Astra: AHa-3D treats human pose/motion estimation (GVHMR, Kimodo) as one tool among many that a frontier agent (GPT-6 Astra) orchestrates alongside geometry (Pi3X) and segmentation (SAM3), with foot-ground and object contact refinement applied after per-person motion recovery.
  • [2026-10-20] TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations: TrackEverything (CMU + Meta Reality Labs; Jain et al.) is the first 3D dense point tracker that scales to all visible points across 1000+ frame videos within 40 GB of GPU memory, via a persistent world-coordinate 3D scene representation with voxelization-based cross-window de-duplication; beats VDPM by >20% APD-P on TAPVid-3D 48-frame clips and matches sparse-tracker SOTA on full-length sequences despite tracking orders of magnitude more points. Code + weights released 2026-10-20 (the subject of Paul’s Slack post), paper on arXiv since 2026-09-24. Complements the recurrent-SSM 2D long-horizon approach of TAPNext++ and the per-pair feed-forward VGGT-based Track4World — same cluster’s long-horizon all-points frontier.
  • [2026-10-03] tapnet — Tracking Any Point (TAP) model family and benchmarks: google-deepmind/tapnet is the official open-source release (Apache-2.0, JAX + PyTorch, HF-mirrored checkpoints) bundling the entire TAP family — TAP-Net, TAPIR, BootsTAPIR, RoboTAP, TAPVid-3D, TAPNext, TRAJAN, TAPNext++ — with Colab + real-time GPU demos; lowest-friction entry point for integrating online 2D point tracking into perception/animation pipelines, and ships the TAPNext++ checkpoint at both 256×256 and 512×512 (DAVIS AJ 65.6% / 67.0%).
  • [2026-10-03] TAPNext++: What's Next for Tracking Any Point (TAP)?: TAPNext++ (DeepMind) fine-tunes a recurrent-SSM TAPNext checkpoint on 1024-frame synthetic sequences via sequence-parallel training + periodic-roll augmentation + occluded-point loss supervision, lifting stable online point tracking from ~150 frames to 1024+ while keeping constant-sized memory; introduces a new R-AJ (Re-Detection Average Jaccard) metric that explicitly scores recovery after increasingly long occlusions, filling an evaluation gap in the TAP-Vid family. Online, frame-by-frame counterpart to windowed trackers like CoTracker3 / Track-On and complements the dense/3D tracking line (MegaFlow, Track4World, PointZero).
  • [2026-09-28] Delta Intelligence launches Δ₀ — whole-body humanoid model with latent world-action brain, 69-DoF controller, and 10,000+ h egocentric panoramic pretraining: Delta D1 — head-mounted panoramic full-body data-capture device from Delta Intelligence — is released as the labeling substrate for Δ₀’s 10,000+ h pretraining corpus of paired egocentric observations + whole-body motion. Sits next to Ego-OSCAR: Egocentric Open source Stereo CAptuRe System Ego-OSCAR (open-source head-mounted stereo-inertial) and Lightwheel AI open-sources EgoSuite-Open100K — 100K-hour egocentric human dataset with hand and body pose EgoSuite (100K-h egocentric with hand + body pose) as first-party data-collection hardware, but panoramic + whole-body rather than stereo-narrow or dataset-only.
  • [2026-09-24] Harsha Gundala — GPC-1 general-purpose classifier with continuous outputs for bboxes, poses, coordinates, mechanical control: GPC-1 (harshagundal) — Qwen + adapters classifier with continuous-output head for bboxes, poses, coordinates and mechanical control; no benchmarks, but ships as an easy-to-fine-tune HF release for the same primitives Perceptron Mk1 productizes and MolmoPoint / LocateAnything formalize.
  • [2026-09-16] Grounded API launch — hand tracking, depth, and SLAM from one RoboCap recording: Grounded SI launches Grounded API — a hosted enrichment product that emits metric 3D multi-view hand tracking, metric depth, and camera trajectory + SLAM from one 8-camera wireless RoboCap recording (co-designed with BitRobotNetwork), with LeRobot / Rerun integration. Two unquantified benchmark claims in the tweet: hand tracker “tops all of Meta’s released datasets” on their proprietary pretraining corpus, and GPU SLAM beats ORB-SLAM3 and cuVSLAM on unnamed public benchmarks; linked research blog unreachable at filing time. Fills the productized-full-egocentric-enrichment-stack slot alongside Perceptron Mk1 (hand pose only) and Rerun’s cuVSLAM integration (SLAM on Robocap via cuVSLAM, packaged with rattler-build (Claude Code build report)).
  • [2026-09-16] AgentSTAR: Agentic Shape Tracking and Reconstruction from Monocular Videos: AgentSTAR (Mazur, Karaev, Chang, Malik, Shafiullah — Berkeley/Meta) is a project-page-only agentic pipeline for joint hand-object shape tracking and articulated-pose reconstruction from monocular video, with an iterative optimisation loop that refines both mesh and pose. Adds a second Malik-group 2026 datapoint to the 4D hand-object cluster alongside DO AS I DO, and evaluates on ARCTIC + HOT3D — the canonical bimanual and egocentric hand-object benchmarks flagged by Yu Xiang — 12 hand-object interaction datasets worth knowing for robot learning. No paper, no numbers, no method details at filing time.
  • [2026-09-10] Gianni Hodel — microSLAM: monocular SLAM for scalable egocentric robot data: microagi (Zürich) + ETH CVG announce microSLAM — a second 2026 monocular-camera-SLAM datapoint alongside WildPose, reported first on the LaMaria benchmark for monocular SLAM and holding up against multi-camera + IMU systems on a single RGB stream. No method or numbers in the tweet; product framing is that monocular geometry is what makes head-worn glasses (rather than sensor rigs) viable as a worker-scale egocentric data-collection form factor.
  • [2026-09-04] Aria Digital Twin to MCAP — Load ADT Episodes in FiftyOne: Voxel51 tooling recipe for turning one Aria Digital Twin episode (.vrs + 8 GT zips under Meta’s non-commercial license) into a single 2.5 GB MCAP with all 12 modality streams (RGB, depth, seg, gaze, 3D+2D object boxes, human skeletons, IMU, 6-DoF pose, semi-dense cloud) synchronized in FiftyOne’s multimodal viewer. Complements HOT3D and ReViV’s use of Aria Digital Twin as a benchmark with the middleware layer that lets pose-estimation researchers actually inspect and curate the ground-truth streams end-to-end without writing a bespoke loader.
  • [2026-09-01] Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling: Lucida raises CA-1M ADD-SB@0.05 (strict surface-alignment 6-DoF pose) from 57.8% to 83.4% by treating placement as a VLM-driven closed-loop refinement rather than a feed-forward regression — the sharpest single-number gain on strict-alignment pose from a learned iterative-refinement recipe on this page. Complements ProxyPose’s video-to-video translation approach (single-object, generative-prior-based) with a scene-scale multi-object counterpart driven by an explicit tool-use policy over a 3D editor’s gizmo.
  • [2026-08-24] Compute-Optimal Scaling Laws for Human Motion Generation: Cartwheel’s motion-generation scaling-law paper depends on running a 3D human-reconstruction stack (“Comic” with physical QA) at ~423k-hour catalog scale, producing the 11,969-hour motion corpus that all downstream scaling fits run on. Puts pose estimation squarely as the bottleneck upstream stage for motion generation the same way GVHMR is for HY-Motion 1.0: Scaling Flow Matching Models for Text-To-Motion Generation and MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos would be if its category-agnostic mocap were plugged in place of Comic. First filed datapoint that quantifies what a curated pose-estimation pipeline yields in effective corpus hours — 5.56M clips survive from a 423k-hour raw catalog after all quality gates.
  • [2026-08-20] OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interaction: OakInk provides 230K RGB-D frames with 3D MANO hand pose + 6-DoF object pose + per-part contact regions across 12 subjects × 100 objects × up to 5 intents; hand pose is obtained by crowd-sourced multi-view 2D keypoints then MANO-layer optimization against multi-view reprojection + geometric consistency + temporal smoothing + silhouette constraints. Cross-dataset training with HO3D improves DexYCB test MPJPE — a load-bearing supervision substrate for the RGB-based hand-pose family on this page.
  • [2026-08-17] Yu Xiang — 12 hand-object interaction datasets worth knowing for robot learning: Yu Xiang (IRVL) posts a curated 12-item directory of hand-object interaction datasets (FreiHAND, RHD, HO-Cap, HO3D/H2O-3D, ObMan, DexYCB, HOI4D, OakInk v1/v2, ARCTIC, PALM, TACO) with download + code links per entry, positioning cross-dataset annotation comparability (pose formats, camera models, coordinate frames) as the current bottleneck rather than data collection.
  • [2026-08-13] HOT3D — Egocentric Multi-View Dataset for 3D Hand-Object Interaction Understanding: HOT3D (Meta Reality Labs / Project Aria) is a dataset landing page for the >1M-multi-view-frame egocentric hand-object benchmark that several filed pose papers already report headline numbers on (MolmoMotion: Language-guided 3D motion forecasting, ReViV: Reconstructing the Viewer and the View in 4D from Monocular Egocentric Video); dual Project Aria + Quest 3 capture with mocap 3D ground truth for hands + 33 PBR-scanned objects and Aria gaze — a substrate slot no prior filed dataset fills at this fidelity.
  • [2026-08-08] MotionBricks: Scalable Real-Time Motions with Modular Latent Generative Model and Smart Primitives: MotionBricks (NVIDIA, TOG 2026) adds the second filed generation-side character-motion system after HY-Motion 1.0 and takes the opposite architectural bet: a VQ-token-based MaskGIT-style backbone (structured multi-head tokenizer + root/pose disentanglement) with all control diversity absorbed by a “smart primitives” authoring layer above the network, rather than a monolithic text-conditioned MMDiT. Runs at 15,000 FPS with 2 ms latency on 350k-clip corpus, and deploys zero-shot from character animation onto a Unitree G1 humanoid — a distinct bridge to physical robot control from ScaleBFM’s PPO-on-retargeted-mocap or ReActor’s physics-in-loop kinematic retargeting.
  • [2026-07-25] ReViV: Reconstructing the Viewer and the View in 4D from Monocular Egocentric Video: ReViV (Lyu, Li, Qian, Zhang, Pollefeys, Tang — ETH + TU Delft + Microsoft) unifies six egocentric modalities (RGB, camera, gaze, body, hand, depth) into a single Masked Generative Egocentric Transformer that emits all six jointly from a monocular first-person video in one feed-forward pass. Removes the pre-computed-camera dependency that HumanEgo-style pipelines require, adds gaze to the cluster (which no prior filed pose paper handles), and reports SOTA across HoloAssist, HOT3D, ARCTIC, Aria Digital Twin, and TACO — the first joint-generative treatment of first-person perception on file.
  • [2026-07-22] Introducing Perceptron Mk1: Perceptron Mk1 productizes hand pose as a first-class VLM output — per-frame bounding boxes + 21-keypoint skeletons (wrist + 4 joints/finger) with left/right identity, tracked through video — as a sidecar on top of a video/embodied-reasoning VLM. First filed closed-source vendor offering in the hand-tracking-as-API slot that HumanEgo and MoCapAnything occupy research-side.
  • [2026-07-21] Lifting Embodied World Models for Planning and Control: Lifted World Model (Wang, Darrell, Izmailov, Bai, Bar — ECCV 2026) inverts the usual pose-estimation direction: forward-kinematics + camera-matrix projection of future XSens leaf-joint 3D positions into the current egocentric frame is used to generate ground-truth 2D “waypoints” for the pelvis/head/hands, which then serve as a low-dim goal-conditioning signal for a diffusion policy. A data point for the pipeline layer of the cluster (given a rigged embodiment + camera, project rather than regress) and a concrete argument that ~8-dim projected 2D goals beat 12-dim 3D goals when the downstream consumer is CEM search.
  • [2026-07-13] Video Generation Models are General-Purpose Vision Learners: GenCeption (DeepMind, ECCV 2026) reports SOTA on 3D keypoint prediction and camera pose using a single instruction-tuned text-to-video diffusion backbone — beating Sapiens on human keypoints and VGGT-Ω on camera pose within the same model that also handles depth, normals, and expression-referring segmentation. Also reports that a model fine-tuned only on synthetic human videos transfers zero-shot to animals and robots, a stronger category-transfer claim than MoCapAnything’s cross-species retargeting because it comes from a single video-generative pretraining prior rather than category-agnostic skeleton prompting.
  • [2026-07-12] 3D-Object Perception Transformer (3PT): 3PT (Intrinsic, CVPR 2026 Highlight) extends the cluster from human/animal skeletons to CAD-defined rigid industrial objects: a single 3D-conditioned two-tower ViT (early-fusion of CAD-render prior embeddings into image tokens) unifies zero-shot detection, segmentation, and 6-DoF pose in one system, matches multi-view RGB refinement against MV-PnP with reciprocal edge correspondences, and beats every zero-shot RGB-D method on BOP-Industrial (+29.1 AP-mm) using only RGB — even surpassing per-object-fine-tuned FRTPose-WAPRv2 (MV-RGB-D) by 12.3 AP-mm. Ranked first in 7 of 11 BOP Challenge 2025 tracks; trained entirely on ~1B synthetic Blender pairs from ~100K CAD meshes.
  • [2026-07-09] LIME: Learning Intent-aware Camera Motion from Egocentric Video: LIME extends flow-matching pose modeling from perception (2D→3D lifting, FMPose3D) to action: a continuous flow-matching head predicts a relative SE(3) target camera pose from an RGB view + language intent, with the multi-hypothesis property load-bearing because a single intent (e.g. “look under the shelf”) admits many valid poses. Supervision is mined at ~3M pairs from RoomTour3D + Nymeria egocentric video.
  • [2026-07-09] ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation: ProxyPose broadens the cluster from skeleton-topology methods (MoCapAnything, SAM 3D Body) to arbitrary-surface 6-DoF tracking with no rig or CAD-model assumption: a fine-tuned video diffusion model translates the input clip into a “proxy video” of a colored polyhedron matching the local rigid-body motion at a single query pixel, so 6-DoF trajectory recovery reduces to classical PnP; achieves SOTA on HO3D / YCBInEOAT / ProxyPose Synthetic and generalizes zero-shot to non-rigid faces, camera pose (where COLMAP fails), event-camera video, and SPAD arrays.
  • [2026-06-23] GPS: Geometric Primary Structure for Articulated Parts Perception in Robot Manipulation: GPS reframes articulated-object pose as a 3-keypoint task (two axis anchors + one part point on the rotation/translation plane) rather than 6-DoF segmentation+pose, and shows on HOI4D and RGBD-Art that this relaxed representation beats both pose-based (CAPNet) and flow-based (GFlow) baselines by 2–3× on direction errors — a task-specific simplification that complements category-agnostic full mocap (MoCapAnything) on the opposite end of the representation spectrum.
  • [2026-06-23] Jun Gao thread (4/5) on Cosmos2.5-2B finetuned with cleaned data + kinematic skeleton conditioning: Kinematic skeleton conditioning on a Cosmos2.5-2B backbone is reported to drive action-following gains in character video generation, a NVIDIA-side counterpart to the Wan-Animate / PoseAnything pose-conditioned line.
  • [2026-06-21] HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos: HumanEgo’s empirical claim that “the accuracy of the upstream SLAM and hand-tracking signal directly bounds downstream policy performance” makes Aria MPS (6-DoF SLAM + 21-keypoint hand pose) a load-bearing dependency of a minutes-of-data manipulation policy — a sharper precision-matters argument than prior egocentric-pretraining work which mostly leaned on data volume.
  • [2026-06-21] LAMP: Localization Aware Multi-camera People Tracking in Metric 3D World: LAMP (Meta Reality Labs, CVPR 2026) fills the egocentric multi-camera-headset gap: lifts per-camera ViTPose 2D keypoints into world-frame Plücker rays using the headset’s known 6-DoF VIO pose, then a spatio-temporal transformer fits SMPL directly — halves MPJPE vs PromptHMR on Nymeria (54.8 vs 109.2 mm) and transfers zero-shot from Aria Gen1 training to Aria Gen2.
  • [2026-06-19] EgoInfinity: A Web-Scale 4D Hand-Object Interaction Data Engine for Any-View Robot Retargeting and Video-to-Action Robot Learning: EgoInfinity is a modular 4D hand-object engine (perception + segmentation + reconstruction + interaction-aware refinement + retargeting) that lifts arbitrary in-the-wild RGB into metric hand trajectories, 6-DoF object poses, and contact states without human annotation — sibling to MoCapAnything’s category-agnostic mocap but specialized to hand-object interaction and any-view footage.
  • [2026-06-19] What Matters When Cotraining Robot Manipulation Policies on Everyday Human Videos?: TriHands ships 3D hand pose labels for 28 hours of natural cooking video via a multi-view exocentric triangulation pipeline, then provides downstream evidence that hand-label quality (not just scale) measurably affects transfer to robot manipulation policies — a concrete utility-for-VLAs argument for high-quality 3D hand annotation infra.
  • [2026-06-18] Do as I Do: Dexterous Manipulation Data from Everyday Human Videos: DO AS I DO extends the cluster from body/skeleton pose to 4D hand-object reconstruction (time-varying 3D hand pose + object pose + contact) from monocular RGB, claiming SOTA on hand-object interaction estimation against ground-truth datasets and on dexterous trajectory extraction from online video — handling both egocentric and exocentric in-the-wild footage with one algorithm.
  • [2026-06-18] ActiveMimic: Egocentric Video Pretraining with Active Perception: ActiveMimic recovers synchronized 6-DoF camera and wrist trajectories from a single body-worn RGB stream — no IMU, depth, or paired sensor rig — and shows the recovered trajectories are accurate enough to supervise downstream VLA pretraining that matches robot-data-pretrained baselines.
  • [2026-06-06] MAMMA: Markerless & Automatic Multi-Person Motion Action Capture: MAMMA (Black group + CMU, CVPR oral) — multi-view markerless SMPL-X capture with per-landmark transformer queries + SAM-2 mask conditioning; within 1.6 mm of Vicon-MoSh++ on held-out markers and visually indistinguishable from marker-based capture on two-person interaction. First filed paper to convincingly close the markerless-vs-marker gap for hands + close contact.
  • [2026-06-02] PoseAnything: Universal Pose-guided Video Generation with Part-aware Temporal Coherence: PoseAnything inverts the cluster’s perception focus to generation conditioned on arbitrary skeletons: XPose (50K clips) is built via Qwen-2.5-VL-7B filter + Grounded-SAM-2 masks + BlumNet skeleton extraction, providing the first public non-human pose-video dataset and an automated annotation pipeline that complements the perception-side category-agnostic mocap line (MoCapAnything).
  • [2026-06-03] HumanNet: Scaling Human-centric Video Learning to One Million Hours: HumanNet bundles hand- and body-related signals alongside ~1M h of egocentric and third-person video, positioning it as a candidate pretraining corpus for hand/body pose models at internet scale.
  • [2026-06-01] EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World: EgoVerse-A’s Aria Gen 1 captures supply accurate camera poses, 3D head trajectories, and 21-keypoint hand skeletons across 1,362 h / 2,087 demonstrators with shared protocols mirrored across four academic labs — a candidate pretraining corpus for egocentric pose models with controlled cross-lab variation.
  • [2026-05-25] WildPose: A Unified Framework for Robust Pose Estimation in the Wild: Widens the cluster from body/object pose to monocular camera pose: frozen MASt3R backbone + differentiable dense BA + learned motion masks; claims unified SOTA across dynamic (Wild-SLAM, Bonn), static (TUM, 7-Scenes), and low-ego-motion (Sintel) benchmarks.
  • [2026-05-25] MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos: Category-Agnostic Motion Capture (CAMoCap) from monocular video + arbitrary rigged asset prompt; factorizes into 3D-keypoint trajectories then IK; sets a new bar on Truebones Zoo (CD-Skeleton 0.2549 vs GenZoo 0.4580) and demonstrates zero-shot cross-species retargeting.
  • [2026-05-23] CYANPUPPETS 1B real-time motion model (camera → UE/Unity/Blender): CYANPUPPETS 1B real-time markerless motion model — camera → UE/Unity/Blender streaming pipeline (product signal for latency × VRAM tradeoff).
  • [2026-05-23] FMPose3D: monocular 3D pose estimation via flow matching: Flow-matching-based monocular 3D pose lifter; probabilistic, multi-hypothesis, competitive with diffusion lifters at lower step counts.

Open questions

Section titled “Open questions”
  • Does the category-agnostic skeleton prompting in MoCapAnything compose cleanly with a high-fidelity parametric body like MHR for the human-specific case, or does the IK constraint set need to be specialized?
  • For productized real-time mocap (CYANPUPPETS-class), where is the inflection between direct image→3D regressors and probabilistic 2D→3D lifters in the latency × accuracy frontier?
  • Should “pose estimation” live under World Foundation Models perception backbones (DINOv2 / VGGT / V-JEPA 2) as a downstream task, or is the rig/IK side enough of its own discipline that it warrants a separate stack?
  • How much of MoCapAnything’s transfer benefit comes from the coarse 4D-mesh bridge specifically, versus from any geometry-aware bridge between RGB and joint space (point clouds, depth, surface normals)?
  • Datasets are still small (Truebones Zoo at 1,038 clips). How does cross-species generalization scale with dataset diversity vs depth?

Papers

Section titled “Papers”

2026-02

Section titled “2026-02”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”