PoseAnything: Universal Pose-guided Video Generation with Part-aware Temporal Coherence

TL;DR

PoseAnything is a pose-guided video generation framework that, unlike prior human-only systems (Animate-Anyone, MagicPose, UniAnimate, Animate-X), accepts arbitrary skeletal inputs — non-human characters, custom rigs, animals — and is fine-tuned from a frozen Wan2.2-TI2V-5B backbone. Two architectural contributions: (a) a Part-aware Temporal Coherence Module that segments the skeleton into parts, matches parts across frames via attention weights, and runs cross-attention only between matched parts to preserve appearance during large pose deformation; (b) Subject and Camera Motion Decoupled CFG, which injects subject-pose conditioning into the positive CFG anchor and camera-motion conditioning into the negative anchor, enabling independent camera control in a pose-driven model. Ships XPose, a 50K-clip non-human pose-video dataset built via a Qwen-2.5-VL-7B filter + Grounded-SAM-2 mask + BlumNet skeleton pipeline.

Key claims

PoseAnything is the first pose-guided video generation model that accepts arbitrary skeleton types (human and non-human), rather than human poses adapted to non-humans (Animate-X) [§1, §2.2].
Channel-axis concatenation of the pose latent with the image latent outperforms MLP fusion and width-axis concatenation as the conditioning-injection strategy on a Wan2.2-TI2V-5B base [§4.1].
The Part-aware Temporal Coherence Module matches parts across frames using the observation that attention weights between corresponding parts in different frames exceed weights between non-corresponding parts; this drives a hard part-to-part cross-attention rather than full-object cross-attention [§4.2, Eq. 8].
Removing part segmentation and matching but keeping full-object cross-attention (the “EC” ablation) degrades quality vs the full PTCM, and removing PTCM entirely degrades it further [§5.5, Table 3].
Decoupling subject-pose control (positive CFG anchor) from camera-motion control (negative CFG anchor) prevents the mutual interference that arises from injecting both signals into a single conditioning channel [§4.3, Eq. 10].
Camera control via the negative anchor works by injecting an opposite-direction skeleton-shaped signal — to pan the camera left, inject a leftward-moving rectangle skeleton into the negative anchor, which steers the background flow rightward [§4.3].
On the TikTok human benchmark, PoseAnything beats Animate-X and UniAnimate across all five metrics (PSNR 31.50 vs 30.78/30.77, SSIM 0.836 vs 0.806/0.811, FVD 133.95 vs 139.01/148.06) [Table 1].
On non-human pose-guided generation against trajectory-control baselines (Tora, ATI, SG-I2V) over a 51-video held-out XPose split, PoseAnything wins on PSNR (30.29), SSIM (0.7114), LPIPS (0.3241), and FVD (99.97) [Table 2].
XPose is built by a three-stage automated pipeline: Qwen-2.5-VL-7B-Instruct filters single-non-human-subject videos from Koala-36M and UltraVideo; Grounded-SAM-2 generates per-frame masks with first-frame-IoU temporal anchoring; BlumNet extracts skeletons from masked frames, with frames discarded if too few skeletons are recovered [§3].
Training uses a three-stage curriculum: (1) baseline w/o PTCM on human-only data for 3k iters; (2) same model on human + non-human mixed data; (3) train only the PTCM module on mixed data for 8k iters with all other modules frozen [§5.1].

Method

PoseAnything is built on Wan2.2-TI2V-5B Wan-Image: Pushing the Boundaries of Generative Visual Intelligence. The reference image is encoded with the pretrained Wan2.2 VAE to a latent z; the pose sequence is encoded with the same VAE to a pose latent z_p. The image latent is concatenated with a noise latent along the temporal axis and patchified, then the pose latent is concatenated along the channel axis, with the patchify convolution’s input channel count expanded to accommodate the new dimensions. The DiT input dimensionality is held constant. Three injection strategies (channel-cat, MLP fusion, width-cat) are compared in the supplement; channel-cat wins.

The Part-aware Temporal Coherence Module (PTCM) sits after the final cross-attention layer of each DiTBlock. It runs in three steps: (1) Part-mask generation — each pose is segmented into N parts (one part per labeled skeleton segment), and each part skeleton is dilated by a frame-specific coefficient k chosen as the smallest dilation that covers the corresponding region of the reference image; (2) Part matching — for the first frame’s masks, compute the attention weight between first-frame and subsequent-frame tokens (averaged over a few preliminary inference steps), then match each first-frame part to its highest-attention counterpart in each subsequent frame; (3) Part-aware cross-attention — for each matched pair (P^1_n, P^t_n), compute Q from subsequent-frame tokens in P^t_n and K/V from first-frame tokens in P^1_n, then standard scaled-dot-product attention. This restricts appearance propagation to matched parts and refuses to propagate appearance across unrelated body regions.

Subject and Camera Motion Decoupled CFG modifies the standard classifier-free guidance update so subject-pose and camera-motion signals are anchored on opposite sides. Standard CFG mixes a conditional and unconditional prediction along a scaling axis; here, the positive anchor receives subject-pose conditioning and the negative anchor receives camera-motion conditioning (rendered as a moving rectangle skeleton). The negative anchor is constructed to push the model away from a chosen camera state, so injecting a leftward-translating rectangle in the negative anchor steers the model toward producing rightward background flow — i.e., a leftward camera pan. This means a model trained only on subject-pose conditioning can be driven, at inference time, to also obey camera-trajectory instructions without ever having seen explicit camera-motion training data.

Results

TikTok human pose-driven video generation [Table 1]: PoseAnything achieves PSNR 31.50, SSIM 0.836, L1 2.79e-05, LPIPS 0.224, FVD 133.95 — best across all five metrics against Disco, MagicAnimate, MagicPose, AnimateAnyone, Champ, UniAnimate (30.77 PSNR), and Animate-X (30.78 PSNR, 139.01 FVD).
Non-human pose-driven generation on a 51-video XPose split [Table 2]: PoseAnything is best on PSNR (30.29), SSIM (0.7114), L1 (8.19e-06), LPIPS (0.3241), and FVD (99.97) against trajectory-control baselines Tora, ATI, and SG-I2V.
PTCM ablation [Table 3]: Removing the PTCM module entirely (concat-only baseline) is worst on all metrics; replacing part-segmented cross-attention with full-object cross-attention (“EC” variant) is in between; full PTCM wins.
Camera-motion control [Fig. 8 qualitative]: With subject-pose driving a character and a separate camera command injected into the negative anchor, the character executes its pose sequence while the camera simultaneously executes the specified pan/tilt — no head-to-head numbers against dedicated camera-controlled video diffusion are reported.

Why it’s interesting

PoseAnything is the first paper filed that uses CFG anchors as an explicit control-channel separator — pushing subject conditioning to the positive anchor and camera conditioning to the negative anchor. The mainstream camera-controlled video diffusion recipes filed in Camera-Controlled Video Diffusion all add a spatial control channel (3D Gaussian trajectories in VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control, depth+RGB control maps in Video Analysis and Generation via a Semantic Progress Function, a reference-video branch in OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer) and train explicit camera-trajectory adapters. PoseAnything instead extracts camera control for free from a model trained only on subject-pose data, by manipulating the CFG anchor structure at inference. This is a clean addition to the open question in Camera-Controlled Video Diffusion about whether camera control needs its own trained channel.

The “arbitrary skeleton” framing complements MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos, which solves the inverse task — extracting universal-rig motion from monocular video — using a similar category-agnostic primitive. PoseAnything generates video from arbitrary skeletons; MoCapAnything recovers arbitrary skeletons from video. Together they sketch a rig-agnostic perception/generation loop (the Pose Estimation and Motion Capture page collects the perception half). Wan-Animate (Wan-Animate: Unified Character Animation and Replacement with Holistic Replication) uses spatially-aligned 2D skeleton maps added to noise latents on the same Wan family but is human-only and adapter-free; PoseAnything channel-concats pose latents and adds a learned PTCM module, which extends to non-human topologies.