Direct Motion Models for Assessing Generated Videos

TL;DR

TRAJAN is a point-track autoencoder that produces a motion-only feature space for evaluating generated videos — both as a Fréchet-distance distributional metric (a motion-focused alternative to FVD) and as a per-video reconstruction-error score (Average Jaccard) usable when no reference exists. By auto-encoding BootsTAPIR point tracks rather than RGB patches or action-recognition logits, TRAJAN is markedly more sensitive to synthetic temporal distortions than I3D / VideoMAE / motion-histogram / RAFT-warp baselines and predicts human ratings of motion / appearance / realism / interaction consistency better than these alternatives across 11 EvalCrafter and 8 VideoPhy generators. The TRAJAN reconstruction score also spatiotemporally localizes which point at which moment is responsible for a video’s poor motion, giving interpretability that FVD lacks.

Key claims

FVD and other distribution-level metrics are biased toward frame-level content because their I3D backbone is trained for action recognition; replacing it with a motion-only feature space (TRAJAN over point tracks) raises sensitivity to synthetic temporal distortions while preserving the Fréchet-distance protocol [§1, §5.1, Fig. 4].
TRAJAN is a Perceiver-style autoencoder that encodes a variable-sized set of point trajectories into a fixed-size 128-token latent, then reconstructs held-out query tracks not in the input — forcing the latent to encode a dense motion field rather than memorize the input support set [§3.1].
A “readout” token plus self-attention with occlusion-masked attention gives each per-track embedding invariance to occluded points, which is essential because BootsTAPIR tracks frequently lose visibility under generative-video artifacts [§3.1].
Average Jaccard (AJ) of the autoencoder reconstruction is proposed as a single-video metric — the model trained on real motions reconstructs unrealistic motions worse, so AJ falls when the motion is implausible [§3.1, §5.3].
Distributional comparison: across five elastic-deformation corruption levels on UCF101 (low-frequency 1.1/1.2, high-frequency 2.1/2.2/2.3), TRAJAN’s Fréchet distance is the most temporally-sensitive among TRAJAN / I3D / VideoMAE-PT / VideoMAE-SSv2 / motion-histograms / RAFT-warp [§5.1, Fig. 4 left].
Paired-video latent distances: on a WALT video diffusion model trained on Kinetics-600 to extend 2 real conditioning frames, TRAJAN latent distance correlates least with PSNR (0.23) and SSIM (0.13) — lower than VideoMAE-PT (0.35 / 0.53) and I3D (0.24 / 0.26), meaning TRAJAN measures something distinct from pixel-level appearance error [Table 1].
Per-video reconstruction score: across EvalCrafter (11 generators) and VideoPhy (8 generators), TRAJAN’s reconstruction AJ leads Spearman correlation with human ratings on every dimension — Motion Consistency, Appearance Consistency, Realism, Interactions — beating VideoMAE-PT, I3D-Entropy, RAFT-Warp, and MooG-PSNR [Table 2].
The strongest individual gap is on Interactions — TRAJAN 0.19 vs RAFT-Warp 0.13 vs all appearance-based methods ≤ 0.05 on EvalCrafter — suggesting motion-only features capture object-interaction realism that appearance features and short-range optical flow miss [Table 2].
TRAJAN does not directly measure speed: TRAJAN-Len correlates with human “camera speed” / “object speed” ratings, but TRAJAN-Radii is decorrelated, indicating the latent compresses patterns of motion and not its magnitude — speed has to be read out from the unencoded tracks themselves [Table 3].
Inter-rater Spearman is ~0.46–0.53 across all four motion-judgment dimensions, so any per-video metric is bounded well below 1.0 — TRAJAN’s 0.29 (Motion Consistency, EvalCrafter) is closing roughly 60% of the inter-rater ceiling [Table 2].
Reconstruction quality (AJ) drops localizably both spatially (around the object whose appearance morphs) and temporally (at the frame where the morph happens), giving TRAJAN-AJ an interpretability axis no Fréchet-distance metric can provide [Fig. 1, project page].
A new human-evaluation dataset is collected with four motion-specific dimensions (Consistency / Interactions / Realism / Speed), z-scored per rater, with 3 raters per video drawn from a pool of 10 — these annotations are the supervision against which TRAJAN and baselines are compared [§4, §5.3].

Method

TRAJAN takes a video’s point tracks (estimated with BootsTAPIR) and trains an autoencoder to reconstruct them. The encoder embeds each track’s per-frame (x, y, occlusion) tuple with sinusoidal position encodings, adds a learnable “readout” token, runs self-attention with an occlusion-aware attention mask, and keeps only the readout token to obtain a per-track fixed-length vector. A Perceiver-style cross-attention then aggregates all per-track tokens into 128 latent tokens of size d. The decoder is the inverse: a query point (x, y) at frame t is up-projected to a token, attends to the 128 latent tokens, and is read out as the full trajectory through that query — including an occlusion logit trained with sigmoid cross-entropy and (x, y) coordinates trained with L2. Crucially, the decoder reconstructs held-out query tracks, so the latent must encode a dense field rather than memorize the input support set.

At evaluation time the model is used in three modes. For distributional comparison, latents are pooled per video and a Fréchet distance is computed between the empirical distributions of latents over real and generated sets. For paired-video comparison, the L2 distance between the two videos’ latent means is the metric. For single-video evaluation, the model reconstructs query tracks for the input video itself and Average Jaccard between predicted and ground-truth tracks (averaged over multiple distance thresholds) is the metric — lower AJ means worse motion plausibility. Comparison baselines span motion-based (motion histograms over point tracks following Liu et al. 2024a; warping error from a RAFT optical-flow model) and appearance-based (I3D logits, VideoMAE-v2 PT and SSv2 variants, MooG next-frame PSNR).

Results

Distributional sensitivity on UCF101 + elastic corruptions: TRAJAN’s spatiotemporal/spatial Fréchet ratio is the highest across all five corruption levels among the six tested feature spaces [§5.1, Fig. 4 left]. Per-video metrics averaged over the set show TRAJAN and RAFT-warp are most temporally sensitive, while VideoMAE-PT/SSv2 and motion histograms are insensitive [Fig. 4 right].
Paired latent vs pixel decorrelation (WALT, Kinetics-600 extension, 2364 high-motion samples): TRAJAN latent distance has the lowest correlation with PSNR (0.23) and SSIM (0.13), versus VideoMAE-PT (0.35 / 0.53) and I3D (0.24 / 0.26) [Table 1] — confirming TRAJAN measures motion separately from frame-level appearance.
Per-video human-rating Spearman (Table 2) — TRAJAN leads on every cell:
- EvalCrafter: Motion 0.29, Appearance 0.29, Realism 0.27, Interactions 0.19 (next-best RAFT-Warp 0.28 / 0.27 / 0.25 / 0.13; appearance-based all ≤ 0.05).
- VideoPhy: Motion 0.25, Appearance 0.32, Realism 0.29, Interactions 0.09 (next-best RAFT-Warp 0.20 / 0.26 / 0.18 / 0.03).
- Inter-rater bound 0.46–0.53 across dimensions; TRAJAN is closing roughly half the inter-rater gap.
Model-level realism ranking (EvalCrafter, 11 generators): TRAJAN’s average AJ tracks the order in which humans prefer the generators [Fig. 7], so the per-video signal aggregates to a usable model-level leaderboard without needing a reference distribution.
Speed metrics (Table 3): TRAJAN-Len and RAFT-magnitude correlate with object speed (TRAJAN-Len 0.58 EvalCrafter, RAFT 0.57); TRAJAN-Radii correlates with camera speed (0.42 EvalCrafter, 0.39 VideoPhy) but anti-correlates with object speed — speed is recoverable from the unencoded tracks rather than the latent itself.
Spatiotemporal localization: on a generated video where a glove’s appearance morphs mid-clip, AJ drops at the exact frame and at exactly the points covering the glove, giving a visual heatmap that pinpoints the failure [Fig. 1].

Why it’s interesting

TRAJAN is the cleanest filed instance of a purely motion-based video-quality signal — neither rule-based (HumanScore: Benchmarking Human Motions in Generated Videos via OpenSim biomechanics, The Pulse of Motion: Measuring Physical Frame Rate from Visual Dynamics via learned PhyFPS regression) nor VLM-judged (RISE-Video: Can Video Generators Decode Implicit World Rules?, VLM-as-Evaluator), but learned-on-real-motion and self-supervised. That places it in the same family as Pulse of Motion’s Visual Chronometer (task-specialized regressor that beats VLM judges) and HumanScore (procedural rule-based) on the “rule-/learned-based scoring beats VLM judges for motion” side of the A Very Big Video Reasoning Suite vs RISE-Video debate, but with a tighter scope: TRAJAN avoids semantic priors entirely by construction. For Luma it is directly usable as a free-per-call eval signal alongside or instead of FVD during training, and the AJ heatmap is a debugging primitive — when a generated video looks fine but rates poorly, AJ tells you which point at which frame is responsible, which neither FVD nor VLM judges can do. The latent itself is also a candidate reward signal for GRPO post-training of video models (the Flex / UnifiedReward slot in VLM-as-Evaluator), with the caveat that AJ doesn’t measure speed — pairing TRAJAN-AJ with PhyFPS would cover both axes Pulse of Motion shows are currently broken in frontier generators. The remaining open question is calibration headroom: inter-rater Spearman is only ~0.5, so any single per-video motion metric is fundamentally bounded — but the model-level Fig. 7 ranking suggests aggregating per-video AJ over a model’s outputs is reliable for ranking generators.