FMPose3D: monocular 3D pose estimation via flow matching

TL;DR

FMPose3D recasts monocular 3D pose lifting as a conditional flow-matching problem: a velocity field, conditioned on a 2D pose input, transports Gaussian samples to plausible 3D poses along an ODE in a few integration steps. Multiple noise seeds yield multiple 3D hypotheses; a Reprojection-based Posterior Expectation Aggregation (RPEA) module collapses them into a single prediction by re-projection-error-weighted averaging. The model is SOTA on human benchmarks (Human3.6M, MPI-INF-3DHP) and on the animal benchmarks Animal3D and CtrlAni3D, making it the first lifter to be competitive across both domains. It was accepted to CVPR 2026 and is slated for integration into DeepLabCut. Relevant to Luma as a clean instance of flow-matching being repurposed as a low-step alternative to diffusion-based probabilistic regressors.

Key claims

Monocular 3D pose estimation is treated as a conditional distribution transport problem from a Gaussian prior to the distribution of plausible 3D poses given a 2D input [Abstract].
A learned ODE-driven velocity field enables few-step inference, compared to the many timesteps needed by diffusion-based 3D pose hypothesis models [Abstract].
Multi-hypothesis generation is obtained “for free” by re-sampling the noise seed; ODE trajectories themselves remain deterministic [Abstract].
RPEA aggregates the generated hypotheses by approximating a Bayesian posterior expectation, using re-projection error against the 2D input as the likelihood proxy [Abstract].
FMPose3D surpasses prior methods on Human3.6M and MPI-INF-3DHP and is SOTA on the animal pose datasets Animal3D and CtrlAni3D [Abstract].
Code and a Python API are released; the method is planned to be integrated into DeepLabCut [Project README].

Method

The task is 2D→3D pose lifting: given 2D joint observations, produce 3D joint coordinates. FMPose3D parameterizes a neural velocity field $v_\theta(x, t, c)$ where $x$ is a partial-noise 3D pose, $t$ is flow time, and $c$ is the 2D-pose condition. Training uses the conditional flow-matching loss along a linear interpolation path between a Gaussian sample and a ground-truth 3D pose, regressing $v_\theta$ against the target velocity given by differentiating the interpolation. At inference, the conditional velocity field defines an ODE that is integrated in a few steps from a fresh Gaussian noise sample to a 3D pose; running it with $N$ different seeds yields $N$ hypotheses.

Because a single prediction is needed for benchmark evaluation, the hypotheses are aggregated by RPEA: each hypothesis is back-projected to 2D and assigned a likelihood that decays with its 2D re-projection error against the input; the final estimate is the posterior-expectation-weighted mean over hypotheses. The condition branch handles both human and animal inputs in the same backbone, which is what unlocks state-of-the-art performance across both domains.

Results

Human3.6M / MPI-INF-3DHP: FMPose3D surpasses existing methods on the standard MPJPE-style metrics on both benchmarks [Abstract; Table 1 in the paper].
Animal3D / CtrlAni3D: state-of-the-art on the 3D animal pose benchmarks, showing the method generalizes beyond human skeletons [Abstract; quantitative tables in §4].
Inference cost: few-step ODE integration vs many-step diffusion denoising — the headline efficiency motivation [Abstract; §3].
Acceptance: CVPR 2026 [project README].

Why it’s interesting

A second 2026 paper (after Holmquist’s FMPose at ICLR-time) confirming that flow matching is a strictly better fit than diffusion for low-dimensional structured-regression-with-uncertainty problems — both are probabilistic, but FM gets there in a handful of steps. For Luma the relevant transfer is to any place a diffusion head is in use as a probabilistic regressor (camera, depth, layout, pose) where step count matters at serving time. The RPEA trick — sampling many hypotheses then doing re-projection-weighted aggregation — is also worth filing as a generic pattern for “use the cheap forward model as your posterior likelihood proxy”; it generalizes to any setting where the conditioning input gives a tractable rendering or projection back to evidence. Finally, the human-and-animal generalization is a useful datapoint for how skeleton-agnostic 2D→3D lifters need to be parameterized.