What If: Understanding Motion Through Sparse Interactions

TL;DR

Flow Poke Transformer (FPT) is a single-step generative motion-understanding model from CompVis @ LMU Munich (ICCV 2025) that, given a still image and a sparse set of user “pokes” (a few (location, displacement) pairs), directly predicts a Gaussian-Mixture-Model distribution over the local motion at queried image points. Instead of synthesizing a full video to inspect “what would happen”, FPT outputs an interpretable, multimodal motion distribution at off-grid query locations in one forward pass — pokes condition, query points are predicted. The output GMMs preserve diverse modes (no mode-collapse from regression, no averaging), the predicted uncertainty correlates with ground-truth error, and downstream tasks (dense face motion, articulated-object motion estimation, moving-part segmentation from pokes) are solved by fine-tuning or by reading off the distribution.

Key claims

FPT directly models a continuous, multimodal distribution over local motion conditioned on sparse “pokes”, representing motion at individual points rather than on a dense grid — enabling sparse / off-grid motion processing [§1, §3].
Training objective is negative log-likelihood of per-query Gaussian Mixture Models, predicted in a single forward pass — no iterative diffusion sampling, no separate decoder [§3, project-page §“Method”].
A query-causal attention pattern lets queries attend to image + poke tokens but not to each other, cutting compute and enabling >100k motion queries per second per image on a single H200 [project-page §“Method”].
Training cost is 1 day on 8× H200 — small enough that this is positioned as a generic motion-understanding backbone rather than a flagship video model [project-page §“Method”].
On dense face motion generation the generic pre-trained FPT surpasses specialized task-specific baselines (Abstract; §1).
FPT fine-tunes well to strongly out-of-distribution synthetic-render datasets for articulated-object motion estimation, outperforming in-domain methods (Abstract; §1).
Predicting an explicit distribution makes moving-part segmentation from pokes a read-off operation on the GMM rather than a separate task head — competitive with task-specific baselines (Abstract; §1).
Predicted GMM uncertainty correlates strongly with prediction error against the ground-truth motion — i.e. the model knows when it’s wrong (project-page §“Method”).
Sparse GMM prediction in one forward pass occupies a different point in the latency / diversity trade-off than multi-step diffusion-based motion models: single-step like a regressor, but multimodal like a generative model (project-page §“Method”).

Method

FPT consumes (i) a DINOv2-style image encoding of the input frame and (ii) a set of input pokes — each poke being a (source-location, displacement) pair encoding “this point moves like this”. A second set of query points specifies where the model should predict motion. Image + poke tokens form a context; query tokens read from that context through a query-causal attention mask (queries see context tokens but not each other), so adding more query points is essentially free at inference time. Each query token’s output head parameterizes a Gaussian Mixture Model (mixture weights, means, covariances) over the 2D displacement at that point. Training minimizes NLL of the ground-truth flow under the predicted GMM on real video clips (a WebVid-trained variant and an open-set variant are reported; the open-set weights are released, the WebVid variant is held back on copyright grounds [GitHub README]).

This is a deliberately different posture from diffusion-based motion / video models: instead of an iterative score-based sampler over a dense pixel field, FPT predicts the full local motion distribution directly. The cost saved by being single-step and sparse is spent on richer per-point output (a GMM, not a point estimate), which preserves multimodality without iterative sampling.

Results

Sampling speed: >100k motion queries / sec / image on a single H200; whole-image dense motion generation is correspondingly cheap (project-page §“Method”).
Training: 1 day on 8× H200 for the released open-set model (project-page §“Method”).
Dense face motion: generic FPT > specialized baselines (Abstract; §1; Tab. 1 in arXiv version).
Articulated-object motion (out-of-distribution synthetic data): FPT after fine-tuning > in-domain methods (Abstract; §1).
Moving-part segmentation from pokes: competitive with task-specific baselines via direct read-off of the predicted GMM (Abstract; §1).
Calibration: predicted GMM uncertainty correlates with held-out prediction error (project-page §“Method”).
Two trained variants reported (WebVid-trained, open-set-trained) perform “approximately equally” on average across benchmarks, with per-case variation; only the open-set weights are released, under CC BY-NC 4.0 (HF model card; GitHub README).

Why it’s interesting

It sharpens an open question in the wiki’s World Foundation Models cluster: motion-understanding need not be tied to generative-rollout video models (Genie 3, Wan2.x derivatives) or to predictive-latent video encoders (V-JEPA 2). FPT is a third axis — a single-step generative model over local motion distributions — that’s cheaper than either and produces an artifact (a calibrated GMM per query) that’s directly consumable downstream (segmentation, control, evaluation).
The training-data shape echoes Magic Fixup: Streamlining Photo Editing by Watching Dynamic Videos: both papers mine paired (source-frame, target-frame, motion-supervision) tuples from raw video at scale and train a feed-forward model on them. Magic Fixup uses dense+sparse flow to render the target photo; FPT uses the same kind of supervision to predict the distribution over future motion. Different output, same underlying data resource.
The single-step, multimodal-distribution framing contrasts with the multi-step few-step-generation thread on the wiki (Mean Flows Mean Flows for One-step Generative Modeling, SoFlow, etc.): those compress a diffusion trajectory into 1–N steps for pixel/latent synthesis; FPT bypasses the trajectory by emitting a closed-form mixture density. The “single-step generative but multimodal” trick falls out of choosing a GMM head rather than a delta.
Calibrated uncertainty over motion is unusual in this corner of the literature — most camera/object-controlled video diffusion models (Camera-Controlled Video Diffusion) output deterministic motion conditioned on a control signal and have no usable uncertainty estimate. A poke-conditioned distribution-output model gives the downstream system something concrete to threshold on.