MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos

TL;DR

MoCapAnything (Huawei) formalizes Category-Agnostic Motion Capture (CAMoCap): given a monocular video and an arbitrary rigged 3D asset as a prompt, produce a rotation-based BVH-style animation that drives that specific asset. It factorizes the problem into (i) predicting 3D joint trajectories with a reference-guided transformer, then (ii) recovering asset-specific rotations via constraint-aware inverse kinematics — sidestepping the brittleness of direct rotation regression across heterogeneous rigs. On a 60-clip Truebones Zoo test split it cuts CD-Skeleton from 0.4580 (GenZoo) to 0.2549, generalizes to in-the-wild animal videos, and supports cross-species retargeting (bird → quadruped, dog → bipedal bird) from the same checkpoint.

Key claims

Category-Agnostic Motion Capture (CAMoCap) is defined as: given a monocular video V and a rigged asset R = (mesh, skeleton, image set), predict per-frame joint rotations Θ that drive R [§1, §3.1].
Direct joint-rotation regression is brittle in monocular settings because (i) rotations are defined in asset-local frames, (ii) depth/camera motion entangle local rotations, (iii) per-frame angle regression produces poor temporal continuity — the explicit motivation for factorizing into 3D-keypoints-then-IK [§3.2].
The Reference Prompt Encoder stacks three submodules per fusion block: graph multi-head self-attention with topology bias on the skeleton, cross-attention to mesh surface tokens, and cross-attention to DINOv2 appearance tokens [§3.3.1].
A coarse 4D deforming mesh is reconstructed from the input video and fed as point-cloud tokens alongside DINOv2 visual tokens — explicitly to bridge the modality gap between dense RGB and the point-cloud–like structure of joints [§3.3.2].
Training supervises only masked 3D joint positions (no rotation-space or temporal losses); rotations come from a two-stage IK fitting (closed-form geometric init + differentiable refinement, warm-started across frames) [§3.4, §3.5].
On Truebones Zoo (1,038 clips, 60-clip test split with Seen/Rare/Unseen stratification): CD-Skeleton drops from 0.4580 (GenZoo) to 0.2549 overall, with the gap widest on non-quadrupeds (0.4740 → 0.2821) [Table 1].
Ablations show mesh-token fusion is the most load-bearing component: removing mesh raises Unseen MPJPE from 25.63 to 46.01; removing image set raises it to 41.57; removing graph-MHA raises it to 26.57 [Table 2].
Best architecture is 4 encoder + 12 decoder layers; smaller variants (1 encoder + 12 decoder) hit lower MPJPE on Seen but much worse Unseen MPJVE (0.75 vs 0.38 for ours) [Table 3].
Same checkpoint supports motion capture (matched skeleton) and cross-species retargeting (different skeleton) without retargeting-specific training, including non-biological rigs [§4.5–§4.8].

Method

The architecture has three learnable branches plus a non-learned IK stage. The Reference Prompt Encoder turns a rigged asset into per-joint queries: each joint gets a sinusoidal-encoded position embedding, then passes through stacked blocks of (a) graph self-attention with a topology bias derived from skeleton adjacency / geodesic distance (the AnyTop design), (b) cross-attention to mesh surface tokens with positions + normals, and (c) cross-attention to DINOv2-encoded appearance tokens from a multi-image reference set. Padding masks accommodate variable joint counts.

The Video Feature Extractor runs two parallel streams: a DINOv2-frozen visual stream (per-frame dense tokens) and a geometry stream that runs a pretrained image-to-3D reconstructor over the video to produce a coarse 4D deforming mesh, downsampled and embedded as point tokens. The geometry stream is what bridges RGB to joint space.

The Unified Motion Decoder then refines the time-tiled joint queries through four operations per layer: intra-frame graph self-attention (skeleton topology bias), temporal cross-attention to a sliding window of DINOv2 visual tokens, temporal cross-attention to the corresponding 4D mesh point tokens, and windowed temporal self-attention per joint. An MLP head outputs per-frame 3D joint positions.

The IK fitting stage is two-step: closed-form geometric initialization that aligns rest-pose bone directions with predicted joints along each kinematic chain, then differentiable refinement of rotations that minimizes the FK-vs-prediction discrepancy with regularization toward the geometric init, warm-started across frames for temporal stability. Joint hierarchy, bone lengths, joint limits, and smoothness are all respected as constraints.

Results

Headline: CD-Skeleton 0.2549 vs GenZoo’s 0.4580 on the 60-clip Truebones Zoo test set, with the largest gains on non-quadrupeds where prior animal-mocap methods (template-coupled to SMAL-style quadrupeds) collapse [Table 1].
Generalization stratification: Seen MPJPE 8.06, Rare 12.04, Unseen 25.63; MPJVE on Unseen drops to 0.38 — the temporal-consistency win against smaller architectures (0.75) [Table 3].
Modality importance (Unseen MPJPE): full 25.63, no-mesh 46.01, no-image 41.57, no-GMHA 26.57 — mesh fusion is the single most important component for cross-species transfer [Table 2].
In-the-wild: plausible 3D motion for birds (chickens, eagles, seagulls), quadrupeds (tigers, leopards, elephants, cats, dogs), and exotic skeletons (crabs, fish, snakes), with no retraining [§4.7].
Cross-species retargeting (zero-shot): bird flight → quadruped flapping, fish swimming → crocodile/snake motion, dog running → bipedal bird, crocodile tail-whip → leopard/parrot — driven solely by the prompt skeleton + asset image set [§4.8].

Why it’s interesting

This is the natural Huawei-side response to the SAM 3D Body: Robust Full-Body Human Mesh Recovery direction from Meta: SAM 3D Body decoupled skeleton from soft-tissue shape but still locks the skeleton to a single human topology (the new MHR rig). MoCapAnything goes the other way — keep monocular video as input, but make the skeleton fully prompt-driven so the same network handles humans, animals, robots, and non-biological assets. The two are complementary if you’re building a Luma-style avatar/character pipeline: SAM 3D Body for high-fidelity humans, MoCapAnything for the long tail of creatures and custom rigs.

The methodological move that matters is the 3D-keypoints-then-IK factorization. ActionMesh: Animated 3D Mesh Generation with Temporal 3D Diffusion arrived at a similar “don’t regress rotations directly” insight from the mesh-animation side (predicting per-vertex displacements then registering to a reference mesh); MoCapAnything makes the same bet at the rig level, where the output is BVH-style joint rotations that drop into Maya/Blender. The IK stage handles the rig-specific bookkeeping that a learned head would have to memorize per asset.

The fusion recipe — graph self-attention on the skeleton + cross-attention to mesh surface + cross-attention to DINOv2 appearance, with a parallel coarse 4D mesh bridging RGB and joint space — is also the answer to a recurring VLM Perception Failures failure mode where a flat ViT encoder can’t reliably localize joints on unusual morphologies. The ablation that drops mesh fusion (Unseen MPJPE 25.63 → 46.01) is consistent with Cambrian-S: Towards Spatial Supersensing in Video‘s finding that explicit geometry tokens carry signal that pure dense visual tokens don’t recover.

For Luma the cross-species retargeting result is the standout — same checkpoint, no retargeting-specific training, drives a leopard rig from a crocodile tail-whip video. Pairs naturally with Shape-for-Motion: Precise and Consistent Video Editing with 3D Proxy (3D-proxy video editing) and LPM 1.0: Video-based Character Performance Model (video-based character performance model) as a stack that goes video → arbitrary character animation.