Improving Motion in Image-to-Video Models via Adaptive Low-Pass Guidance

TL;DR

I2V models fine-tuned from T2V backbones systematically produce more static videos than their T2V counterparts (e.g. −18.6% Dynamic Degree for Wan 2.1, −16.6% for CogVideoX on VBench). ALG diagnoses this as the I2V denoiser “locking in” on the reference image’s high-frequency details within the very first denoising step — a shortcut trajectory that overfits the static appearance. The fix is a training-free inference-time tweak: low-pass filter (bilinear down-then-up or Gaussian blur) the conditioning image during the first ~6% of timesteps, then expose the original image. Adapting the CFG formula so only the conditional and one unconditional term use the filtered image (the other unconditional term keeps the original) preserves image fidelity. Yields an average +36% Dynamic Degree across CogVideoX, Wan 2.1, HunyuanVideo, and LTX-Video with negligible quality regression and minimal overhead.

Key claims

I2V models built by fine-tuning T2V backbones lose ~17-19% Dynamic Degree relative to the T2V parent on VBench, while subject consistency, aesthetic quality, imaging quality, motion smoothness, and temporal flickering remain comparable — isolating the I2V conditioning mechanism as the cause [Table 1, §3.1].
PCA visualizations of the Wan 2.1 denoiser’s intermediate feature maps show the model “locks onto” the reference image’s fine details after just one denoising step (at t=0.02 of 50 steps), confining the trajectory before motion can develop [§3.1, Fig. 2].
Applying a low-pass filter (downsampling or Gaussian blur) to the input image monotonically increases Dynamic Degree as filter strength grows, but trades off image fidelity in proportion [§3.1, Fig. 3].
ALG uses an adaptive low-pass filter strength κ(t) that is high at early timesteps (t≈1) and decays to zero at later timesteps (t→0); both a step function (κ=κ₀ for t≥τ, 0 otherwise) and an exponentially decaying κ work, provided high strength is concentrated at early steps [§3.2, Eq. 4].
The CFG formula is modified asymmetrically — only the conditional term and one unconditional term use the filtered image x̃; the other unconditional term keeps the original x — which algebraically decomposes into (a) standard CFG over the filtered image (promotes motion) plus (b) a high-frequency restoration term (preserves fidelity) [§3.2, Eq. 3]. Using x̃ in all three terms produces distorted visuals and unstable generations [§3.2, Appendix C.1.4].
Default config: bilinear down-up filter with downsampling factor 2.5, step-function schedule with transition τ=0.06 (filter active for the first 6% of timesteps) [§4, “ALG implementation details”].
ALG produces an average +36% Dynamic Degree improvement across four I2V models on VBench-I2V: CogVideoX 64.2→82.5, Wan 2.1 28.9→41.5, HunyuanVideo 88.2→92.7, LTX-Video 12.6→21.1 [Table 2].
Image Subject Consistency drops only 0.2-1.4 points across models; the small drop partly reflects the fact that more-dynamic videos legitimately move the subject out of frame, which the metric penalizes [Table 2, §4 qualitative].
The compute overhead is small because the filtered-image branch is only evaluated during the first few denoising steps; total inference cost is comparable to standard CFG [§4, Appendix 1].

Method

ALG sits inside the I2V sampler’s CFG step. Let v_θ(z, c_x, c_T) be the I2V velocity model conditioned on image latent c_x and text c_T. Standard I2V CFG mixes a conditional prediction with one or two unconditional predictions (text-null and/or image-null). ALG modifies this mix using two image conditions per step: the original latent image x and a low-pass-filtered version x̃_t = LPF(x, κ(t)) whose filter strength κ(t) is a decreasing function of timestep.

Concretely, ALG asymmetrically substitutes x̃_t into Eq. 3: the conditional term and one unconditional term use x̃_t, but the remaining unconditional term keeps the original x. Rearranged, this is the sum of (a) ordinary CFG executed on the filtered image (which is what relieves the high-frequency lock-in and recovers T2V-level motion) and (b) a difference term that re-injects high-frequency information from the original image (which is what preserves fidelity). At later timesteps, κ(t)→0, so x̃_t → x and the sampler smoothly converges to standard I2V CFG.

The default LPF is bilinear downsample-then-upsample at factor 2.5; Gaussian blur is an alternative. The default κ(t) is a step function with transition point τ=0.06 — i.e. full filter strength for the first 6% of denoising steps, then off — which the ablation shows is enough to disrupt the shortcut without sacrificing fidelity [§3.2, §5, Table 3].

Results

Headline (Table 2): +36% average Dynamic Degree across 4 I2V backbones on VBench-I2V. Per-model Dynamic Degree (CFG → ALG): CogVideoX 64.2 → 82.5 (+28%); Wan 2.1 28.9 → 41.5 (+44%); HunyuanVideo 88.2 → 92.7 (+5%); LTX-Video 12.6 → 21.1 (+67%).
Average top-5% optical flow (raw motion magnitude, not just the binary “dynamic” classifier): CogVideoX 8.9 → 12.6; Wan 2.1 4.3 → 5.4; HunyuanVideo 10.7 → 11.8; LTX-Video 2.9 → 4.1 [Table 2].
Fidelity: Image Subject Consistency drops 0.2-1.4 points (max −1.4 for CogVideoX, where the motion gain is largest); Imaging Quality drops 0.6-4.0 points; other VBench quality metrics largely flat [Table 2].
Schedule ablation (Table 3, CogVideoX): step function τ=0.06 yields Dynamic Degree 82.5; exponential decay with rate λ=10 yields 82.9 (slightly higher motion, slightly lower consistency); larger λ (sharper decay) gives less motion. All choices that concentrate filter strength early outperform the CFG baseline of 64.2.
Transition time sweep (Fig. 6a): τ=0.04 already gives +22% Dynamic Degree; pushing τ higher continues to improve motion but degrades Image Subject Consistency faster.

Why it’s interesting

ALG is the first filed example in the wiki of a CFG variant that reframes the condition itself as a per-timestep schedulable signal rather than the guidance scalar — complementary to Classifier-Free Guidance variants‘s existing axes (training-time CFG folding via SoFlow: Solution Flow Models for One-Step Generative Modeling, adversarial post-training via Continuous Adversarial Flow Models, attention-space negative guidance via Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models, reference-vs-zero branching via SparkVSR: Interactive Video Super-Resolution via Sparse Keyframe Propagation). Where SparkVSR redefines the unconditional branch, ALG schedules the conditional branch’s content. The diagnostic that I2V conditioning collapses the trajectory within one denoising step is a concrete, mechanistic explanation of “I2V looks more static than T2V” — a complaint that has floated around for two years without a clean root cause. For Luma, the practical implication is direct: any I2V product built on a T2V parent (Wan-derived, Hunyuan-derived, CogVideoX-derived) likely leaves substantial motion on the table that ALG can recover for free at inference time.