The Pulse of Motion: Measuring Physical Frame Rate from Visual Dynamics

TL;DR

Pulse of Motion names a previously-undocumented failure mode of frontier video generators — chronometric hallucination, where the speed of generated motion has no stable relationship to the nominal frame rate the video is saved at — and supplies the tool to measure it. The authors train Visual Chronometer, a query-pooled VideoVAE+-backed regressor that predicts Physical FPS (PhyFPS) directly from visual dynamics, and use it to audit 17 open- and closed-source video generators. The audit reveals pervasive PhyFPS misalignment: closed-source models (Sora-2, Veo-3.1-Fast, Kling-o3, Seedance, Grok-Imagine) average 28-58% percentage error on nominal-vs-physical frame rate, open-source models 31-126%, with intra- and inter-video coefficients of variation between 0.10-0.17 and 0.25-0.52 respectively. Frontier VLMs (Gemini-3.1-Pro, Seed-1.6, Qwen3.5+) fail catastrophically as PhyFPS judges, often mode-collapsing onto a single output. Re-timing generated videos to predicted PhyFPS is preferred by human raters over the originals in 44.2% of pairwise comparisons (vs 19.0% for originals).

Key claims

Modern video generators suffer from chronometric hallucination: their generated sequences have ambiguous, unstable, uncontrollable physical motion speeds, decoupled from the saved meta FPS [§1].
Visual Chronometer is trained on a curated 465,535-clip dataset spanning 18 PhyFPS values, built exclusively from sources where meta FPS verifiably matches physical sampling rate (Adobe240, BVI-VFI, UVG, NVIDIA/Honda sensor-synced AV data, biomechanical mocap, in-house verified) [§3.1].
The training pipeline upsamples sources to 240 FPS via RIFE, then synthesizes lower-rate variants via three exposure models — sharp capture, motion-blur (variable exposure window), and synthetic rolling shutter — forcing the model to learn intrinsic visual dynamics rather than semantic content priors [§3.2].
Architecture: VideoVAE+ backbone + learnable query embedding that cross-attends to latent tokens + MLP regressor, predicting log-PhyFPS; query-based pooling decouples regression from input frame count, enabling arbitrary-length inputs [§4.1].
Loss is MSE in log-space without the standard MSLE offset, since all target PhyFPS values are strictly positive — penalizing relative rather than absolute errors [§4.2].
Two model variants: VC-Wide trained on 18 frame rates (1.875–120 FPS), VC-Common narrowed to consumer/web range (16–60 FPS); both fine-tune the VideoVAE+ backbone end-to-end with the regression head [§4.3].
VC-Common reaches MAE 3.46 / MAPE 9% on PhyFPS-Bench-Real with strict cross-source train/val/test splits; VC-Wide reaches MAE 7.76 / MAPE 21% [Table 2].
All tested VLMs fail at PhyFPS prediction in both video-input and unrolled-image paradigms: Gemini-3.1-Pro (video) MAE 21.67 / MAPE 43%, Seed-1.6 MAE 20.40, Qwen3.5+ MAE 45.54 / MAPE 91% [Table 2].
Severe mode collapse in VLM judges: Seed-1.6-Flash predicts exactly 30 FPS for every input regardless of motion content [Table 2, §5.2].
PhyFPS-Bench-Gen audits 17 generators on 100 dynamic prompts balanced across entity / motion-type / camera / environment / scene axes, excluding speed-manipulation keywords [§5.1].
Open-source models: Wan2.1-1.3B is best aligned (Pct. Error 31%); InfinityStar worst (115-126%); LTX-Video has low CV (0.10 intra, 0.33 inter) but high absolute error (99%) — suggesting its meta FPS metadata is just systematically wrong [Table 1].
Closed-source models slightly outperform open-source on absolute alignment (Sora-2 at 28% Pct. Error best, others 35-58%) but show no advantage on temporal stability (Intra CV 0.13-0.17, Inter CV 0.25-0.34) — researchers prioritize visual fidelity over time-scale fidelity [Table 1].
The systematic bias is one-sided: across nearly all generators, predicted PhyFPS is higher than the nominal meta FPS — i.e. generated videos appear in slow motion relative to their saved frame rate, consistent with the “slow but smooth” phenomenon [§5.1].
User study (1,490 pairwise comparisons, Bradley-Terry preference): globally re-timing videos to predicted average PhyFPS wins 44.2% preference, dynamic per-clip re-timing 36.9%, untouched original 19.0% — global correction beats local correction (hypothesized: within-video frame-rate variation introduces jitter) [§5.1, Figure 4].

Method

Visual Chronometer treats time-scale perception as continuous regression rather than the binary classification used by prior work (SpeedNet, arrow-of-time detectors). Given an input clip of frames, a VideoVAE+ backbone extracts spatiotemporal latent tokens, a learnable query embedding cross-attends to these tokens to aggregate them into a clip-level feature, and an MLP regresses to a scalar log-PhyFPS prediction. The query-based pooling is the key architectural choice — it decouples the regression head from the input frame count, so the model handles arbitrary-length videos without padding hacks.

Training data is the more interesting half. The authors deliberately exclude any video whose nominal frame rate could disagree with its physical sampling rate, leaving only high-frame-rate academic datasets (Adobe240, BVI-VFI up to 120 Hz), uncompressed broadcast (UVG at 50/120 FPS), sensor-synchronized AV data (NVIDIA, Honda), biomechanical mocap, and in-house verified recordings. Each source is temporally upsampled to 240 FPS via RIFE, then resampled down to target rates via three exposure models that mimic real camera mechanics: (1) sharp capture by uniform subsampling (fast shutter), (2) motion blur by averaging windows of high-rate frames with varying exposure lengths, (3) synthetic rolling shutter by sampling pixels with progressively shifted time indices across spatial bands. This gives 465,535 clips across 18 PhyFPS values, all standardized to 128 frames.

At evaluation time on generated videos, overlapping clips of frames are extracted with a stride, per-clip PhyFPS is averaged to a video-level value, then averaged across videos for a model-level estimate. Three metrics are reported: (i) Meta-vs-PhyFPS alignment (absolute and percentage error against nominal meta FPS), (ii) intra-video coefficient of variation (CV across sliding windows in one clip), (iii) inter-video CV (CV across different generations from the same model).

Results

Generator audit (PhyFPS-Bench-Gen, 17 models): closed-source frontiers cluster at 28-58% Pct. Error (Sora-2 best at 28%, Veo-3.1-Fast 57%); open-source span 31-126% (Wan2.1-1.3B best at 31%, InfinityStar worst at 115-126%) [Table 1].
Temporal stability is uniformly poor across the entire frontier — closed-source Intra CV 0.13-0.17 is not meaningfully better than open-source 0.10-0.16, despite closed-source models having lower absolute alignment error. Visual fidelity has been optimized; temporal consistency has not [Table 1, §5.1].
LTX-2 and LTX-Video have the lowest Inter CV among open-source (0.34, 0.33) — internal temporal representations are consistent, the high absolute error is likely just bad metadata defaults; the authors suggest LTX-Video’s meta FPS should be adjusted from 24 to ~46.5 [§5.1].
Visual Chronometer accuracy (PhyFPS-Bench-Real, 4000 cross-source test clips): VC-Common MAE 3.46 / MAPE 9%; VC-Wide MAE 7.76 / MAPE 21% [Table 2].
VLM-as-judge fails catastrophically. Best video-mode VLM (Seed-1.6-Flash) MAE 20.00 / MAPE 40%; Gemini-3.1-Pro MAE 21.67 / MAPE 43%; Qwen3.5+ MAE 45.54 / MAPE 91% [Table 2]. Several VLMs (Gemini-3-Flash, Seed-1.6-Flash, Qwen3.5+) suffer mode collapse — always predicting the same scalar regardless of input.
Image-based paradigm (unrolling video into 128 separate image inputs) does not help — Gemini-3.1-Pro drops from MAE 21.67 (video) to a mode-collapsed Avg Pred of 5.15 (image-based), MAE 44.85 [Table 2].
User study: PhyFPS-guided re-timing improves human-perceived naturalness. Global re-timing (Pred) wins 44.2% of 1,490 pairwise comparisons, dynamic local re-timing (Pred Dyn) 36.9%, untouched original 19.0% [§5.1, Figure 4].

Why it’s interesting

This is a clean, narrowly-scoped contribution that pins down a specific failure mode that VBench / FVD / SSIM-style benchmarks simply cannot see — perceptual smoothness >90% can coexist with motion that runs 50-130% off its nominal speed. The strongest synergy is with Physion-Eval: Evaluating Physical Realism in Generated Video via Human Reasoning: Physion-Eval shows 83-94% of frontier-generated videos contain human-detectable physical glitches in object permanence / dynamics / interaction; Pulse of Motion shows that even within the smooth-looking outputs, the time axis itself is unreliable. The two benchmarks measure orthogonal failure axes of the world-model thesis. The VLM-as-judge result is the sharpest filed data point for the VLM-as-Evaluator open question — VLM judges don’t just underperform a specialized model, they mode-collapse onto a single output for an entire test set, directly extending the VBVR vs RISE-Video debate by showing rule-based (or task-specialized regression) wins where VLM judges fail outright. For Luma the practical takeaway is twofold: PhyFPS re-timing is a free post-processing improvement that a 44.2% vs 19.0% user preference shift makes hard to ignore, and the systematic “predicted > meta” bias suggests Luma’s own generators (and training data curation) should explicitly account for time-scale mismatch in source corpora rather than treating nominal FPS metadata as ground truth.