FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generation

TL;DR

FlowMo is a training-free, inference-time guidance method for text-to-video flow-matching diffusion models that improves motion coherence using only the model’s own per-step predictions — no external motion signal, no fine-tuning, no architecture change. The core observation: temporally coherent video corresponds to low patch-wise variance over time in an appearance-debiased latent representation (per-frame L1 distance between consecutive latent frames). FlowMo extracts this signal at early-to-mid denoising steps (1–12 of 50) and adds a gradient step on the input latent to minimize the maximum patch-wise variance, on top of the standard CFG step. Demonstrated on Wan2.1-1.3B and CogVideoX-5B, with a +6.20% and +5.26% VBench Final Score lift respectively, and a ≈2.36× inference-time cost.

Key claims

In an appearance-debiased latent representation $\Phi(v_\theta) = \lVert v_\theta(t{+}1) - v_\theta(t) \rVert_1$ over consecutive predicted latent frames, temporally incoherent videos exhibit strictly higher patch-wise variance than coherent ones — validated by a user study on hundreds of generated videos, with the separation becoming prominent from approximately the 5th denoising step onward [§3.2, Fig. 2].
Motion information emerges only in mid-stage denoising steps (≈ steps 4–8 of the trajectory) while coarse appearance is set in the first few steps; this is confirmed both quantitatively (variance separation) and qualitatively by visualizing $\hat{z}_1$ across timesteps [§3.2, Fig. 3].
The FlowMo loss is the maximum patch-wise temporal variance (averaged across channels), not the mean — using the mean ablates the effect because most patches are static and dilute the gradient signal [§3.3, Eq. 8; §4.3 Fig. 6, 3rd row].
The guidance is applied as a gradient step on the input latent $z_t$ (not on the velocity output), with Adam at learning rate 0.075, only in the first 12 denoising steps; applying it across all steps introduces high-frequency artifacts because motion is already determined [§3.3, Eq. 9; §4.3 Fig. 6, 5th row].
On VBench, FlowMo lifts Wan2.1-1.3B’s Final Score 75.14% → 81.34% (+6.20) and CogVideoX-5B’s 65.43% → 70.69% (+5.26), with Motion Smoothness up +2.13 and +2.28 respectively, and Dynamic Degree dropping only <1.5% (vs. −16.20% for FreeInit) [Table 1].
In a forced-choice user study on the VideoJAM benchmark, annotators prefer FlowMo for Motion Coherence in 44.3% / 43.0% of pairs vs. 16.2% / 17.6% for the Wan2.1 / CogVideoX baselines; aesthetic quality and text-video alignment also improve [Fig. 5].
Three design choices are individually necessary: max-variance loss (not mean), the appearance-debiasing operator (raw velocities are appearance-biased and dilute the motion signal), and restricting guidance to the first 12 steps [§4.3, Fig. 6].
Compute overhead is real: 234.3s vs. 99.27s per video on H100, i.e. ≈2.36× slowdown, attributed to the gradient backprop through the denoising step [§4.4].

Method

FlowMo plugs into a standard Flow Matching denoising loop on a DiT-style video generator. At each denoising step where the FlowMo schedule fires (steps 1–12), the model first runs CFG-guided velocity prediction $v_\theta$ as usual. It then constructs the appearance-debiased temporal representation $\Phi(v_\theta) \in \mathbb{R}^{(F-1) \times N \times C}$ by taking pairwise L1 distances between consecutive predicted latent frames. The per-patch temporal variance tensor is computed across the $F-1$ time axis, averaged across channels to give a spatial coherence map, and reduced to a scalar by taking the maximum — this targets the single most incoherent patch, which empirically gives non-zero gradient where the mean would not. A single Adam gradient step (lr 0.075) is then taken on the input latent $z_t$ to reduce this max variance, the velocity is re-predicted on the updated $z_t$ , and the FM step proceeds normally to produce $z_{t-1}$ .

The schedule and operator are motivated by an empirical analysis (§3.2): coherence-vs-variance separation only becomes prominent at step ≈5 and motion emerges in the latent only between steps 4–8, so guidance is restricted to the early-to-mid window. Outside this window the standard CFG step is unmodified. The debiasing operator $\Phi$ is critical — predicted velocities are appearance-biased per Chefer et al. 2025 (VideoJAM), so measuring variance on raw $v_\theta$ buries motion in scene-content statistics.

Results

Headline numbers are on two open-source FM video models at native settings: Wan2.1-1.3B at 480×832 and CogVideoX-5B at 480×720, both at 81 frames / 16 fps / 5 seconds. On VBench’s aggregated Final Score, FlowMo gains +6.20% on Wan2.1 (75.14 → 81.34) and +5.26% on CogVideoX (65.43 → 70.69). Quality Score (the appearance-quality aggregate) improves +8.0 / +11.28; Semantic Score (text alignment) improves +4.41 on Wan2.1 and decreases negligibly (−0.77) on CogVideoX. Motion Smoothness goes up +2.13 / +2.28 (key claim is motion coherence). Dynamic Degree, which credits amount of motion, decreases only <1.5% — an important comparison because the closest prior method (FreeInit, adapted to DiTs) collapses Dynamic Degree by 16.20% to achieve similar coherence gains. Human evaluation on the VideoJAM motion benchmark shows FlowMo winning motion-coherence comparisons by 44.3 to 16.2 (Wan2.1) and 43.0 to 17.6 (CogVideoX). Cost: 234.3s vs. 99.27s per video on H100, a 2.36× slowdown from backprop through the denoising step; partially mitigatable by training-time integration but not demonstrated.

Why it’s interesting

FlowMo is a clean instance of two themes the wiki is tracking. First, it’s a CFG variant (Classifier-Free Guidance variants) — but unlike the four prior axes filed there (training-time guidance via SoFlow, bimodal CFG in LTX-2, noise-augmentation in Ostris, reference-vs-zero in SparkVSR, generalization-fix in CAFM), FlowMo’s “second branch” is not a conditioning channel at all but a self-supervised loss on the model’s own intermediate prediction. CFG with strength 5 still runs as the first step; FlowMo’s variance loss then refines the input latent. That makes it the first filed “guidance on top of CFG via the model’s own latent statistics” — and the right comparator for Random noise augmentation during diffusion training eliminates the need for CFG‘s claim that CFG patches a noise miscalibration, because FlowMo’s “patch” is at a different layer (model-internal motion statistics, not the noise distribution).

Second, it’s another datapoint for Inference-Time Scaling on the generation side. The wiki’s existing generation-side entry, Self-Refining Video Sampling (Self-Refining Video Sampling), iterates Predict-and-Perturb at each fixed noise level to pull the latent toward the data manifold; FlowMo instead does a gradient step on the latent to reduce a hand-designed motion-coherence loss. Both spend roughly 2× NFEs for noticeable per-frame improvement; both depend on the trained model’s own predictions as the “verifier” rather than an external VLM judge. The contrast is at the loss design: Self-Refining uses Predict-and-Perturb as a generic manifold-pull, FlowMo targets a specific failure mode (temporal incoherence) with a specific statistic. This is the right kind of paper to compare against if Luma wants to ask “what’s the best per-step refinement loss for video?”.

The other interesting connection is to the same authors’ TokenTrim: Inference-Time Token Pruning for Autoregressive Long Video Generation (TokenTrim), filed yesterday. FlowMo and TokenTrim are both training-free, inference-time, latent-space-only interventions on text-to-video — FlowMo on a single video’s denoising trajectory, TokenTrim on an autoregressive video’s KV cache over time. The Shaulov / Chefer group is staking out a research line: “extract usable temporal signal from the pre-trained model’s own latents, then act on it at inference.” Worth watching.