Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoising

TL;DR

Time-to-Move (TTM) is a training-free, backbone-agnostic add-on that gives any image-to-video diffusion model precise motion and camera control at inference time. The user provides a crude reference animation — cut-and-drag for objects, or a depth-reprojected warp for camera trajectories — plus a mask of the controlled region; TTM initializes denoising from a noised version of that warp and applies dual-clock denoising (low noise inside the mask to enforce motion, high noise outside to let the scene evolve freely). It claims parity or better than training-based baselines such as Go-with-the-Flow on object-control and camera-control benchmarks, with no fine-tuning and no extra runtime cost. Demos run on Wan 2.2.

Key claims

• Crude “cut-and-drag” reference animations (or depth-based reprojection for camera control) are sufficient motion cues for an I2V diffusion model when properly noised and combined with a clean-image conditioning — no learned trajectory adapter is required [project page §1, §3].
• Dual-clock denoising — different per-region SDEdit-style noise levels, low inside the user mask (enforce commanded motion) and high outside (allow free dynamics) — outperforms both vanilla SDEdit and RePaint at the same task: SDEdit either suppresses unmasked dynamics (low noise) or drifts from the commanded motion (high noise), while RePaint introduces artifacts in uncontrolled regions and rigid motion in controlled ones [project page §3, Fig. dual-clock comparison].
• The same single recipe handles three control modes: user-specified object motion (drag a masked region along a 2D trajectory), camera motion (depth-reproject the input image along a user camera path and nearest-neighbor-fill the resulting holes), and appearance editing (insert/modify objects in the warp so the diffusion model integrates them into the generated motion) [project page §3, §4].
• TTM is reported as comparable to or better than training-based Go-with-the-Flow on both realism and motion fidelity in the object-control comparison, with zero additional training [project page §5, Fig. GWTF comparison].
• The method has no additional runtime cost over a single I2V sampling pass — it only changes initialization and per-region noise scheduling [project page §1].

Method

TTM has two inputs: (i) a coarse warped reference video built by either dragging masked image regions along user trajectories or by depth-reprojecting a single image under a user camera path with nearest-neighbor hole filling; (ii) a binary mask marking the region the user wants to control. The I2V diffusion backbone is conditioned on the clean input image and initialized from a noised version of the warped reference (SDEdit-style anchoring of appearance, injection of motion).

During sampling, a region-dependent noise schedule is applied: pixels inside the mask are initialized at lower noise (so the commanded motion is preserved through more of the denoising trajectory), pixels outside the mask are initialized at higher noise (so the prior over natural dynamics dominates and the scene can evolve organically). The same primitive — region-specific noise levels acting as a per-pixel “denoising clock” — is what gives the method its name.

For camera control specifically: depth is estimated from the input frame, the frame is reprojected to each new viewpoint along the user camera path, holes are filled by nearest-neighbor color, and the resulting coarse video is fed through the same TTM pipeline. For appearance edits, the warp simply contains the modified object before TTM is run on top.

All demos are produced by integrating TTM into Wan 2.2, but the framework is presented as backbone-agnostic — the only requirements are an I2V diffusion model and access to its per-step noised latents.

Results

• Object-control comparison against the training-based Go-with-the-Flow (GWTF) baseline reports comparable or superior realism and motion fidelity on user-specified object-control tasks, without any fine-tuning of the backbone [project page §5, Fig. GWTF comparison].
• Dual-clock denoising is qualitatively shown to outperform SDEdit (which trades off motion fidelity vs unmasked-region freezing as a function of the global noise level) and RePaint (which forces motion via foreground overwriting and introduces artifacts) on the same controlled-vs-uncontrolled-region task [project page §3, Fig. dual-clock comparison].
• Quantitative numbers (FVD, ObjMC, RotErr, VBench) are not stated on the project page; the arxiv PDF likely contains them.
• Camera-control demos cover smooth dolly / orbit / pan trajectories on stills, with TTM-on-Wan-2.2 generating dynamic backgrounds and parallax that the warp alone cannot supply.

Why it’s interesting

Direct relevance to Paul’s question: TTM is explicitly a training-free, plug-and-play recipe — if Ray 3’s I2V interface exposes the noised latent at each step (or even just the first one), the dual-clock initialization can in principle be applied without touching weights. The hard part is whether Ray 3’s specific noise schedule and conditioning structure are compatible with SDEdit-style re-initialization; that’s an integration question, not a training one.

The cluster framing for Camera-Controlled Video Diffusion has so far been dominated by training-based adapter recipes on a frozen Wan backbone — VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control (GeoAdapter blocks every 5th DiT layer), SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time (animation-time embedder), OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer (reference branch with RoPE offsets) — all of which require building a synthetic training set (CamTime, VerseControl4D, internal reference-pair data) and a non-trivial post-training run. TTM is the first datapoint in that cluster that is zero-training: no adapter weights, no synthetic dataset, no fine-tune. The trade-off it implicitly bets on is that the warp+mask conveys enough information that a strong I2V prior (Wan 2.2 here) can fill in the rest from the noise schedule alone.

It also contrasts with Video Analysis and Generation via a Semantic Progress Function (ReTime/SPF), which is also inference-time and also operates on Wan 2.2 — but ReTime warps temporal RoPE positions globally toward uniform semantic velocity, while TTM warps the noise schedule spatially per-region. The two are likely composable.

See also

• Camera-Controlled Video Diffusion — TTM is the first training-free entry in this cluster; all other filed papers train an adapter on a frozen Wan backbone.
• OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer — OmniTransfer also avoids explicit trajectory parameterization, but does so by conditioning on a clean reference video through a learned reference branch; TTM avoids both training and the reference-video dependency by using a crude user-built warp.
• Video Analysis and Generation via a Semantic Progress Function — ReTime/SPF is another inference-time control primitive on Wan 2.2, but warps temporal RoPE rather than the noise schedule; likely composable with TTM.
• SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time — SpaceTimePilot manufactures a synthetic CamTime dataset and trains an E_ani embedder for the same camera+animation control TTM gets for free at inference time.
• VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control — VerseCrafter’s GeoAdapter requires a 35k-clip training set; TTM is the training-free counterpoint for the same object-trajectory control task.