Generative Action Tell-Tales: Assessing Human Motion in Synthesized Videos

TL;DR

A learned-manifold metric for evaluating the plausibility of human actions in generated videos. The encoder fuses appearance-agnostic 3D skeletal features (SMPL pose / shape / orientation from TokenHMR), 2D keypoints (DWPose), and frozen ViT-H/16 appearance features, plus their first-order temporal derivatives, into a per-frame representation aggregated by a Transformer over short temporal windows. Two scores read out of the manifold: Action Consistency (distance from a per-class real-action centroid) and Temporal Coherence (smoothness of frame-trajectory in latent space). On the paper’s new TAG-Bench (10 UCF-101 action classes × 5 I2V generators, Mechanical-Turk-rated on two axes), the two metrics hit Spearman ρ 0.61 / 0.64 against humans — a +35.6% / +68.4% relative jump over the best prior baseline (GPT-5 at 0.45 / 0.38).

Key claims

Existing video-quality metrics correlate poorly with human ratings of action correctness — pixel/perceptual (PSNR, SSIM, LPIPS, FVD, CLIP-Score) and MLLM-fine-tuned (VideoScore, VideoPhy-2) scorers underperform because their feature spaces are appearance-biased and lack temporal sensitivity to anatomical implausibility [§1, §2, Table 1].
The proposed action manifold is built from five static feature sources — SMPL pose θ, body shape β, global orientation Φ, 2D joint keypoints J, and ViT appearance features V — plus their per-frame temporal derivatives, giving a “two-pathway” static + motion design inspired by SimonYan & Zisserman 2014 [§3.1, §3.2.1].
Bone-length / pose plausibility is captured by the SMPL features (which carry an anatomically-valid prior because SMPL is fit only to real human data); anomalies that escape the prior (elongated limbs, implausible joint configs common in generated video) are caught by the appearance-prior-free 2D keypoints [§3.1.1, §3.1.2].
The encoder is trained with a two-term objective: supervised contrastive loss across action classes for what action is being performed, plus a self-supervised temporal-distortion loss that pushes shuffled / frozen-first-frame / reversed windows away from their coherent counterparts for how the action progresses [§3.2.2, Eq. 3].
Two readouts: Action Consistency S_A = distance between a generated video’s mean window-embedding and the real-action class centroid; Temporal Coherence S_T = a smoothness term over consecutive frame embeddings within each window [§3.3, Eq. 4]. Lower = better on both.
TAG-Bench is the paper’s own benchmark: 10 UCF-101 actions selected for single-person full-body whole-body motion (BodyWeightSquats, HulaHoop, JumpingJack, PullUps, PushUps, Shotput, SoccerJuggling, TennisSwing, ThrowDiscus, WallPushups) × 5 I2V models (Wan 2.1, Wan 2.2, HunyuanVideo, OpenSora, Runway Gen-4), Mechanical-Turk rated on Action Consistency and Temporal Coherence with inter-rater Spearman 0.72 / 0.71 [§4].
The two proposed metrics hit Spearman ρ 0.61 / 0.45 (Action Consistency) and 0.53 / 0.64 (Temporal Coherence) — versus the best baseline (GPT-5 prompting at 0.45 / 0.38), VideoPhy-2 Physical Commonsense at 0.28 / 0.37, VBench-2.0 Human Anatomy at −0.40 / 0.02 [Table 1].
Generic MLLM prompting baselines vary widely: GPT-5 leads (0.45 / 0.38), Gemini-2.5-Flash next (0.40 / 0.25), then GPT-4o (0.34 / 0.31) and Qwen3-VL-8B (0.34 / 0.28); LLaVA-1.5/1.6 sit near zero or negative [Table 1].
Inter-rater Spearman bound is 0.72 / 0.71 — so the proposed metric closes most of the achievable gap on Action Consistency and nearly all of it on Temporal Coherence [Table 1].
The training corpus is the same 10 UCF101 actions at native 320×240 / 25fps, in 8-frame temporal windows with 4-frame overlap; videos used as first-frame seed for TAG-Bench are excluded, multi-person videos are filtered out [§5.1].

Method

The encoder operates per fixed-length temporal window (8 frames). For each frame, five appearance-, geometry-, and motion-features are extracted: SMPL pose θ, shape β, and global orientation Φ via TokenHMR (with Detectron2 person crops); 2D keypoints J via DWPose; ViT-H/16 appearance V from TokenHMR’s frozen backbone. Temporal derivatives of each are computed — relative rotations between adjacent frames for the rotation-matrix features (θ, Φ) and ℓ₂ Euclidean differences for the rest (β, J, V).

Each static feature and its motion counterpart is processed by its own 1D temporal-conv block (3 dilated conv layers, kernel size 3, dilations 1/2/4, residual). The static and motion encodings are summed element-wise (additive fusion, GENMO-style), giving five 256-d frame representations per input. A learned scaled-dot-product attention with one query vector fuses the five per-input embeddings into a single per-frame 256-d token. The sequence of 8 frame tokens plus a learnable [CLS] is processed by a 4-layer Transformer encoder (8 heads), yielding a [CLS] window-embedding E_win and per-frame embeddings E_per. Both are ℓ₂-normalized.

Training is from scratch on 10 UCF101 classes, 200 epochs, AdamW lr=1e-4 / wd=1e-2, batch 256, cosine schedule, single A100, with the two-term objective L = L_contrastive(E_win) + γ·L_temporal(E_win, shuffled/frozen/reversed) — γ is the loss weight. At evaluation, Action Consistency compares the mean of generated-video window embeddings against the corresponding action class’s centroid computed over real-video windows; Temporal Coherence averages a frame-trajectory smoothness term over the per-frame embeddings within each window.

Results

Spearman correlation with human ratings on TAG-Bench (higher is better):
- Action Consistency (ours) 0.61 vs GPT-5 0.45 vs VideoPhy-2 PC 0.28 vs Gemini-2.5-Flash 0.40 (next-best non-ours: GPT-5; +0.16 / +35.6% relative) [Table 1].
- Temporal Coherence (ours) 0.64 vs GPT-5 0.38 vs VideoPhy-2 PC 0.37 vs Qwen3-VL-8B 0.28 (next-best non-ours: GPT-5; +0.26 / +68.4% relative) [Table 1].
Both proposed metrics also correlate cross-axis (S_A → Temporal 0.45, S_T → Action 0.53), so each is useful as a soft estimate of the other axis [Table 1].
VBench-2.0’s Human Anatomy score is negatively correlated with human Action Consistency on TAG-Bench (−0.40) — the existing anatomy axis disagrees with human judgement on this task set [Table 1].
Frame-level appearance metrics (PIQUE, BRISQUE, SSIM-sim, MSE-dyn, X-CLIP-Score, TRAJAN) all sit in the −0.20 to +0.21 range, confirming the paper’s claim that appearance / scene-trajectory features are not enough for action realism [Table 1].
Inter-rater Spearman 0.72 / 0.71 puts a ceiling on any per-video metric — the proposed metrics close most of the gap from the GPT-5 baseline to that ceiling [Table 1].

Why it’s interesting

This is the third filed learned-or-rule-based motion-quality scorer that beats VLM-judge baselines on human action evaluation, joining HumanScore: Benchmarking Human Motions in Generated Videos (procedural rule-based OpenSim biomechanics) and Direct Motion Models for Assessing Generated Videos (TRAJAN, self-supervised point-track autoencoder). All three sit on the “specialized scorer beats LMM-as-judge” side of the VLM-as-Evaluator debate framed by A Very Big Video Reasoning Suite vs RISE-Video, but each picks a different substrate: HumanScore is fully procedural over a hand-designed anatomical/kinematic/kinetic rubric; TRAJAN learns a motion-only latent over generic point tracks with no human-specific prior; this paper learns a human-specific fused appearance + 3D-skeleton + 2D-keypoint manifold. For Luma the appeal is the same one Simran flagged — a free-per-call signal for assessing whether generated videos respect human motion semantics that GPT-5-prompting can’t currently match — and the SMPL+ViT recipe is more in line with what video-generation pipelines already produce as auxiliary signals than HumanScore’s OpenSim fit. The interesting open contrast with HumanScore is in coverage: HumanScore probes 51 motion types × 13 frontier closed+open generators with biomechanics-derived limits and reports Spearman ≈ 1.0 against AI/biomech researcher raters; this paper probes 10 UCF101 actions × 5 mostly-open generators with Mechanical-Turk raters at ρ ≈ 0.6. Whether the lower ρ is an inter-rater-noise effect or a real metric-quality gap is the question a side-by-side study would answer.