Video Analysis and Generation via a Semantic Progress Function

TL;DR

Generated video transformations (morphs, edits, style transitions) often evolve unevenly — long stretches with little change followed by abrupt semantic jumps. This paper introduces the Semantic Progress Function (SPF): a 1D curve derived from pairwise SigLIP embedding distances that captures cumulative semantic change across a sequence. The companion method, ReTime, linearizes this curve at inference time by warping the video diffusion transformer’s temporal RoPE positions according to the measured progress curve — allocating more temporal capacity to semantically dense regions and less to stable ones. Demonstrated on Wan2.2 and LTX-2 (where the audio track aligns naturally with the linearized visual content), with no retraining or manual annotation. SIGGRAPH ‘26.

Key claims

The Semantic Progress Function reduces a video to a 1D curve fitted by weighted least squares to pairwise SigLIP angular-distance constraints, with Gaussian temporal weighting (scale σ_t) emphasizing local pairs — its slope is the instantaneous rate of semantic change [§3.1, §3.2, Eq. 4–6].
SigLIP outperforms OpenCLIP, DINO, and pixel-L2 as the SPF embedder: pixel-L2 fails to register a man→gorilla transition; SigLIP captures a local peak corresponding to onset of anger that other embedders miss [Fig. 10 top, §5.4].
ReTime warps RoPE temporal positions via inversion of the normalized SPF: output frame n should exhibit progress n/N, so warped position τ(n) = ψ_φ⁻¹(n/N) stretches semantically dense regions and compresses stable ones [§4.1, Eq. 7].
Uniform RoPE warping across all frequency bands destabilizes generation; frequency-aware warping with band-dependent strength α_d decaying exponentially from low to high frequencies gives the best linearization without local jitter [§4.1, Eq. 9–10, Fig. 4].
A timestep-dependent multiplier β(t) = exp(-λ_t · t) concentrates warping early in denoising (structure formation) and lets late steps refine details naturally [§4.1, Eq. 11].
Three iterations of refinement (Eq. 12–13) are empirically sufficient to drive the SPF close to linear [§4.1].
For first-and-last-frame-only models like Wan2.2, the SPF is partitioned by segmented least squares into approximately-linear segments; each segment is then regenerated as a clip whose duration is proportional to its semantic span and the clips are concatenated [§4.2].
For LTX-2 (joint audio-video), retiming the visual sequence to linear semantic progression makes the generated audio synchronize naturally with the linearized visual content because the model anchors cross-modal attention to linear temporal coordinates [§4.1 “Extension to Audio-Video Models”].
Retimed Wan2.2 and LTX-2 outputs stay within one standard deviation of the original on VBench Aesthetic Quality, Motion Smoothness, and Temporal Flicker — visual fidelity preserved despite manipulating internal embeddings [Table 1].
A user study reports 88% preference for the retimed videos on semantic pacing while visual quality is maintained [§5.5, supplementary §E].
On synthetic Keenan’s-Spot rotations with known angular velocity profiles (constant, rising exponential, falling exponential), the SPF dotted curves closely track the ground-truth angular position — validating that the SPF measures semantic pacing rather than pixel motion [Fig. 9, §5.3].

Method

A frozen pretrained semantic encoder (SigLIP) embeds every frame; pairwise angular distances d_{ij} between L2-normalized embeddings are computed for frames whose temporal distance |i−j| ≤ k. The SPF f ∈ R^N is then estimated by solving a regularized weighted least-squares system: minimize ‖W^{1/2}(Df − d)‖² + λ‖f‖² where D is a sparse pair-difference matrix and W is diagonal with Gaussian-in-temporal-distance weights w_{ij} ∝ exp(−|i−j|²/(2σ_t²)). The closed form is f* = (DᵀWD + λI)⁻¹ DᵀWd.

For inference-time retiming of a video diffusion transformer (Wan2.2, LTX-2), the normalized SPF ψ_φ(n) ∈ [0,1] is inverted to give warped temporal positions τ(n) = ψ_φ⁻¹(n/N). These positions are substituted into the temporal RoPE rotation R(τ, ω_d) = [[cos ωτ, −sin ωτ], [sin ωτ, cos ωτ]] per frequency band d. Critical refinements: (a) frequency-aware blending τ̃(n, d) = (1−α_d)·n + α_d·τ(n) with α_d decaying from low to high frequencies, (b) a timestep multiplier β(t) = exp(−λ_t t) concentrating correction at high noise, (c) three iterations of the warping pass, (d) interpolation to the model’s 4×-compressed latent grid. For closed-source videos where Wan can only condition on first/last frames, the SPF is partitioned by segmented least-squares into piecewise-linear segments and each segment is regenerated with clip length proportional to its semantic span.

Results

VBench metrics (Aesthetic Quality, Motion Smoothness, Temporal Flicker) on Wan2.2 and LTX-2 fall within 1σ of the original baseline after retiming [Table 1].
User study: 88% of participants prefer the retimed version on semantic pacing while reporting comparable visual quality [§5.5].
Linear pixelwise interpolation between key frames produces ghosting on an abrupt strawberry→bird transition; LTX-2 in keyframe-interpolation mode is bottlenecked by its own generative constraints; ReTime on the host model produces a more coherent transition [Fig. 7, §5.1].
Real cinematic transformation sequence (Stranger Things — Vecna transformation) regenerated via SPF-segmented Wan2.2 redistributes a previously lighting-cue-masked metamorphosis into a continuous human→monster evolution [Fig. 5–6, §5.2].
Exponential target SPFs (rising/falling) successfully accelerate/decelerate a sunrise sequence at controlled rates, demonstrating the framework supports arbitrary target pacing, not only linearization [Fig. 8, §5.3].
Synthetic angular-velocity validation: SPF curves track ground-truth rotation angle for constant, rising-exp, and falling-exp profiles on a single rotating object on white background [Fig. 9].
For Wan2.2, distance power γ = 0.7 yields better segmentation for existing-video regeneration than the γ = 1 default [§5.4 bottom, Fig. 10 bottom].

Why it’s interesting

The SPF treats semantic pacing as a first-class, measurable axis of video — orthogonal to camera trajectory, object motion, or visual fidelity. Methodologically the inference-time fix is closely related to Camera-Controlled Video Diffusion recipes that manipulate temporal RoPE — SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time, which adds an animation-time embedder E_ani to disentangle space and time on Wan2.1, and OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer, whose per-task RoPE offset injection picks the spatial-width axis for temporal tasks — but instead of conditioning RoPE on a control signal, it warps RoPE based on a measured property of the model’s own output. That makes it a diagnostic + corrective loop rather than a control adapter. The LTX-2 finding that retiming the visual track automatically realigns the audio is a free corollary worth noting for Joint audio-video generation work.