Generative Video Motion Editing with 3D Point Tracks

TL;DR

Edit-by-Track is a track-conditioned video-to-video framework from Adobe Research that enables joint editing of camera and object motion in a real video by conditioning a pretrained Wan2.1-T2V-1.3B on the full source video plus paired source/target 3D point tracks. A learnable sampling-and-splatting cross-attention conditioner (Tracktention-style) uses the projected 3D tracks to gather source visual context and splat it into the target frame space; explicit depth tokens give the model 3D awareness for occlusion and depth-order reasoning. Two-stage LoRA-64 fine-tuning — synthetic Mixamo+Kubric pairs to bootstrap, then real monocular non-contiguous-clip pairs (24K stock + DL3DV + object-removal pairs) — yields a 1.3B model that beats ATI (14B), DaS, GEN3C, TrajectoryCrafter, ReVideo, and TrajAttn on DyCheck and MiraData while supporting motion transfer, shape deformation, and object removal/duplication.

Key claims

Existing track-conditioned I2V models (DaS, ATI, PaC, TrajAttn) generate motion from a single frame and lose the rest of the source context, so they hallucinate appearance under joint camera+object edits; the V2V formulation that conditions on the full input video plus paired 3D tracks fixes this [§1, §2].
The 3D-track conditioner uses cross-attention sampling-and-splatting in 2D screen space (Tracktention-style) rather than direct nearest-neighbor sampling (TrajAttn-style), and omits Tracktention’s attention-bias term — bias is reported as sensitive to noisy 3D tracks [§3.1, Eq. 1–2].
3D depth is injected as an additive positional-encoding token on the sampled features (not via the splatting branch), giving explicit depth cues for occlusion handling without splatting depth values directly [§3.1].
Visibility labels from 3D track estimators are not used — all source tracks are fed in regardless of occlusion, and the model is expected to implicitly reason about visibility (because edited-track visibility is ambiguous after manipulation) [§3.1].
Two-stage training: Stage 1 bootstraps motion control on Blender pairs (Mixamo humans + Kubric backgrounds, same base scene, varied animation+camera, GT 3D tracks from mesh vertices); Stage 2 fine-tunes on 24K monocular stock videos using non-contiguous clip pairs (1–5 s gap) with 2K estimated 3D tracks each, plus DL3DV static scenes and an object-removal video-pair set [§3.2].
On DyCheck iPhone (12 dynamic scenes, joint camera+object editing): Edit-by-Track reaches PSNR 14.80 / SSIM 0.424 / LPIPS 0.406 (masked: 15.99 / 0.747 / 0.247), beating the strongest privileged baseline TrajAttn+NVS-Solver (13.94 / 0.416 / 0.549) and all V2V-inpaint methods (GEN3C, TrajCrafter) [Table 1].
On 100 in-the-wild MiraData pairs: the 1.3B Edit-by-Track outperforms the 14B ATI on PSNR (19.55 vs 19.07), SSIM (.657 vs .635), LPIPS (.236 vs .244), and especially End-Point Error (6.12 vs 11.44 — almost 2× better track control), giving up only ~38 FVD points (306 vs 269) to the 14B model [Table 2].
Inference: 81-frame 480×832 video in 4.5 minutes on a single A100 with 50 steps and CFG=5; trained with LoRA rank 64 on Wan2.1-T2V-1.3B at total batch 64 on 16 A100s for 4K + 8K iterations [§3.2, §3.3].
Application set unlocked by the 3D-track interface: (a) independent camera-only or object-only editing by editing only one track group [§3.3], (b) human motion transfer by swapping SMPL-X local pose θ between source/target [§3.3, Fig. 7a], (c) non-rigid shape deformation via point-group selection + Linear-Blend-Skinning interpolation [§3.3, Fig. 6b], (d) object removal by moving track points outside frame bounds — uniquely supports simultaneous viewpoint change [§3.3, Fig. 7c], (e) partial-track input — user only specifies a few key tracks (e.g. goat body) and the model synthesizes plausible motion for unspecified parts (legs) [§3.3, Fig. 8].

Method

The model is Wan2.1-T2V-1.3B with LoRA-64 fine-tuning and an additional 3D-track conditioner trained from scratch. The DiT block input is the concatenation of (noisy target video tokens) + (clean source video tokens from a VAE-encoded source). The 3D-track conditioner takes source and target 3D tracks Ps, Pt ∈ R^{T×N×3}, projects them through their respective camera parameters to 2D screen-space disparity coordinates ps, pt, temporally downsamples to match the latent’s T’, and then runs a learnable sample-then-splat:

Sample: positional-encode source coordinates as query; key/values come from per-frame source video tokens at a 16×16 grid; cross-attention gathers a context vector per track per frame, followed by per-track temporal Transformer aggregation.
Depth inject: add positional-encoded z values from the 3D tracks (additive token on the sampled features).
Splat: positional-encode source coordinates and target coordinates as separate queries; cross-attention splats the per-track sampled features back into source and target frame token spaces.

The resulting paired track tokens are element-wise added to the corresponding source/target video tokens (no explicit attention bias), then concatenated and fed to the DiT. Training is rectified-flow with Wan2.1’s original objective; the Stage 1 → Stage 2 split is crucial because real video pairs with controlled motion variation do not exist at scale, and synthetic data alone domain-gaps to real video.

Results

DyCheck joint motion: PSNR 14.80, SSIM .424, LPIPS .406 (full frame); masked LPIPS .247 (next best privileged baseline TrajAttn+[126]: .351) [Table 1].
MiraData in-the-wild: PSNR 19.55, SSIM .657, LPIPS .236, FVD 306.44, EPE 6.12 vs ATI 14B’s 19.07 / .635 / .244 / 268.80 / 11.44 [Table 2]. EPE almost halved — the strongest claim that the 3D-track conditioner actually transfers motion rather than just hallucinating in-distribution video.
Parameter efficiency: 1.3B base + LoRA-64 beats a 14B base (ATI) on three of five metrics; the lift is in motion-fidelity (EPE) and context-preservation (PSNR/SSIM/LPIPS).
Qualitative editing wins [§4.3, Fig. 9]: I2V baselines (ATI, DaS) lose source context; inpainting V2V (GEN3C, TrajectoryCrafter) cannot synthesize the unwarped object motion even when fed an optical-flow-warped video. Edit-by-Track uniquely composes object removal with viewpoint change, which inpainting methods cannot.

Why it’s interesting

This is the first filed paper in the wiki to make 3D point tracks (not 2D tracks, not 3D Gaussians, not skeletons, not bounding boxes) the unified control signal for joint camera+object editing on a real source video. It sits directly between VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control (which argued 3D Gaussian trajectories beat point tracks 2.51 vs 6.90 on ObjMC, but only for generation, not editing) and Vista4D: Video Reshooting with 4D Point Clouds (which uses 4D point clouds for video reshooting — camera-only). The V2V-with-full-source-context formulation also resolves the failure mode Shape-for-Motion: Precise and Consistent Video Editing with 3D Proxy identified in I2V proxy-based editing, but with a much lighter user interface (no 3D mesh proxy). And the Tracktention-style cross-attention splatter is a third design point in the Wan-adapter family alongside VerseCrafter’s GeoAdapter and OmniTransfer’s RoPE-offset reference branch (Camera-Controlled Video Diffusion).