ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation

TL;DR

ProxyPose recasts 6-DoF pose tracking from monocular video as video-to-video translation: given only an input video and a single marked pixel in the first frame, a fine-tuned video diffusion model generates a “proxy video” in which a colored polyhedron undergoes the same local rigid-body motion as the surface region at the marked pixel. Because the proxy’s geometry and appearance are known by construction, recovering its 6-DoF trajectory reduces to classical Perspective-n-Point (PnP) with off-the-shelf solvers. The formulation offloads the hard parts of pose tracking — reflective / transparent / textureless surfaces, occlusions, deformations — into the video model’s pixel-level translation step, requires no 3D models / depth / masks / task-specific features at inference, and is trained only on synthetic data.

Key claims

Video-to-video translation subsumes 6-DoF pose tracking without auxiliary inputs (no 3D model, depth, mask, or task-specific features), using only the video and one first-frame pixel [§Abstract].
The proxy is a colored polyhedron whose known geometry + appearance lets 6-DoF trajectory extraction reduce to classical PnP [§Abstract].
One query pixel achieves state-of-the-art pose tracking on HO3D, YCBInEOAT, and the ProxyPose Synthetic Benchmark, even without rigidity constraints or bundle adjustment; multi-query bundle adjustment further improves results on rigid objects [project page, “quantitative results”].
The video diffusion model is fine-tuned only on synthetic data, yet transfers to in-the-wild real footage including transparent, reflective, and heavily occluded objects [§Abstract, project page].
The proxy formulation extends beyond rigid objects to non-rigid surfaces (human faces), camera pose tracking (via a static world point, robust where COLMAP fails on textureless scenes), and non-standard sensing modalities without additional training — event cameras and single-photon avalanche diode (SPAD) arrays [project page, “extensions”].
Failure modes are inherited from the video backbone: pose drift on reflective / textureless surfaces under complex motion, ill-defined tracking on fluid surfaces, and blurred proxy frames when fast motion exceeds the video VAE’s capacity [project page, “Limitations”].

Method

Given a monocular video and one query pixel in frame 0, ProxyPose fine-tunes a video diffusion model to translate the input clip into a proxy video: a synthetic clip in which a colored polyhedron (fixed known 3D geometry, fixed known per-face colors) undergoes the same local rigid-body motion as the surface patch at the query pixel. The video model is fine-tuned only on synthetic data with paired (input clip, proxy clip) supervision. At inference, given the generated proxy video, contour detection recovers the polyhedron silhouette per frame, colored-face correspondences give 2D-3D correspondences on the known polyhedron mesh, and a standard PnP solver returns the full 6-DoF pose trajectory. For rigid objects, multiple query pixels can be fused via multi-query bundle adjustment as a straightforward post-processing improvement.

The design offloads the hard perception problem (materials, occlusions, deformations, unknown geometry) onto the large-scale video prior, while the geometry-recovery step remains classical and analytically well-posed.

Results

SOTA 6-DoF pose tracking accuracy on HO3D, YCBInEOAT, and the ProxyPose Synthetic Benchmark, competitive or better than methods that require CAD models, depth, or masks [§Abstract; project page].
Multi-query bundle adjustment gives further gains on rigid objects “in most cases,” while single-query ProxyPose (no rigidity assumption) already sets SOTA [project page].
Robust on challenging materials (transparent, reflective) and extreme occlusions where prior methods fail [project page, comparisons].
Zero additional training required to (a) track human faces, (b) recover camera pose on textureless scenes where COLMAP fails, (c) generalize to event-camera and SPAD-array inputs [project page, “extensions”].

Why it’s interesting

ProxyPose is the second filed paper (after Shape-for-Motion: Precise and Consistent Video Editing with 3D Proxy) to lean on a “proxy” abstraction that lifts a hard video task into a canonical, geometry-known intermediate — but the direction is inverted. Shape-for-Motion lifts the object to a 3D mesh proxy so the user can edit motion, then propagates edits back into pixels; ProxyPose synthesizes a proxy in pixels so a classical solver can extract motion. Both papers argue the same underlying bet: video diffusion models are strong enough to serve as the translation layer, and problem-specific hard parts should be pushed into that translation step rather than being solved analytically.

The result also broadens Pose Estimation and Motion Capture beyond skeleton-topology methods (MoCapAnything, SAM 3D Body) and toward arbitrary-surface tracking with no rig assumption — a category the wiki hadn’t cleanly covered. That the same recipe generalizes zero-shot to event cameras and SPAD arrays is a real-world test of the “video priors are universal perception backbones” claim that World Foundation Models papers have been making at a coarser granularity.