Shape-for-Motion: Precise and Consistent Video Editing with 3D Proxy

TL;DR

Shape-for-Motion is an object-centric video editing framework that lifts the target object to a time-consistent 3D mesh (the “proxy”), lets the user edit that proxy once on a single frame, then propagates the edit to all other frames and re-renders via a decoupled video diffusion model conditioned on geometry (normal map) and texture renderings. The mesh is built by deformable 3D Gaussian Splatting plus DPSR + Marching Cubes, with multi-view diffusion-generated novel views and a balanced view-sampling rule to compensate for monocular-input limitations. A Dual-Propagation Strategy uses canonical Gaussians for geometry edits and the canonical mesh for texture edits to avoid the vertex-misalignment failure modes of a naive single-path propagation. The final video diffusion model (an SVD I2V backbone + ControlNet) is trained with a two-stage mixed-training recipe that first learns geometry-following from augmented data, then learns texture enhancement with stochastic dropout of the texture stream.

Key claims

3D-mesh proxies — not text, not first-frame edits, not 2D point drags — are the right interaction primitive for precise object-level video edits like “rotate the panda 20° left” or “stretch the car’s roof”, because the underlying transformation is quantifiable and consistent across frames by construction [§1, §4.1.1, Fig. 1, Fig. 6].
Per-frame independent mesh reconstruction breaks frame-to-frame consistency; a canonical mesh + time-varying deformation field (deformable 3DGS-based) is needed to expose the cross-frame correspondences edits need to ride [§3.1].
Monocular-only deformable-3DGS reconstruction is incomplete; mixing in multi-view diffusion-generated novel views, plus a Balanced View Sampling that caps the aggregate sampling probability of all novel views at the per-frame observed view’s probability, is what makes the reconstruction work without amplifying synthetic-view artifacts [§3.1, Table 2: PSNR 20.35 → 22.91 with novel views + BVS].
A naive single-path “edit canonical mesh → apply deformation field” fails because the deformation field was trained for Gaussian points, not mesh vertices, and accumulates positional errors on mesh vertices [§3.2, Fig. 8 Variant-1].
Dual-Propagation Strategy: use the canonical Gaussian deformation field to propagate the geometry offset (via nearest-neighbor vertex→Gaussian mapping), and use the canonical mesh + a backward deformation field to propagate texture (avoids spatial colour shifts) [§3.2, Eqs. 2-3].
The decoupled diffusion conditioning (geometry as ControlNet normal map, texture as a coarse texture rendering into the base SVD model) prevents the model from degrading into a pure 2D enhancement that ignores the geometry stream [§3.3].
Two-stage mixed training is what enables view-changing edits: Stage 1 freezes the SVD base and trains only the geometry controller with the texture stream replaced by background; Stage 2 fine-tunes the base with the geometry controller frozen and the texture stream supplied with 20% probability — single-stage training fails on rotations the augmentation can’t simulate [§3.3, Table 3: Frame-Acc 0.786 → 0.970 with Stage 2].
On V3DBench (22 videos × 6 edit categories: pose, rotation, scaling, translation, texture, composition), Shape-for-Motion wins on all reported metrics — Frame-Acc 0.970, CLAP Score 0.917, plus user-study Editing-Quality and Semantic-Consistency rankings — over Tune-A-Video, Pix2Video, I2V-Edit (= Image-Sculpting + I2V-Edit), and DragVideo [Table 1].
The same 3D-proxy pipeline doubles as an image-to-video animator: reconstruct → rig → animate the mesh → re-render with the decoupled diffusion model [§5, Fig. 10].

Method

The pipeline has three stages. (1) Consistent 3D proxy reconstruction: deformable 3DGS jointly optimizes canonical Gaussians, a deformation MLP, a canonical-color MLP, and a backward deformation MLP. DPSR + Marching Cubes converts deformed Gaussians into per-frame meshes; the backward deformation lets each deformed-mesh vertex query its canonical-frame color. Stable Video 3D (Voleti et al. 2025) generates auxiliary novel views per frame; Balanced View Sampling caps the cumulative selection probability of all generated views at the observed-view probability so synthetic-view inconsistencies don’t dominate. A scale-invariant depth loss + L1 + SSIM + mesh-image L1 form the training objective.

(2) Dual-Propagation editing: the user edits the canonical mesh to produce an offset $\Delta P_c^M$ . Geometry path: build a nearest-neighbor vertex→Gaussian map, transfer the offset to the canonical Gaussian set, apply the deformation field at frame $t$ , integrate to get the Gaussian-propagated edited mesh. Texture path: the mesh-propagated edited mesh (obtained by directly applying the deformation field to the canonical mesh) carries the correct colour from $C_{can}$ ; an additional nearest-neighbor mapping between the mesh-propagated and Gaussian-propagated edited meshes transfers colour from the former to the latter. Both maps are computed once per frame.

(3) Decoupled video diffusion: Stable Video Diffusion (SVD) I2V is the base; a ControlNet-style branch consumes the normal map (geometry). Training pairs are synthesized from the input video itself — the target object is segmented and degraded (random scale/shift/rotate + SLIC + median blur + down-up sample) to mimic the pre/post-edit pair. Stage 1 trains only the geometry controller with the texture stream replaced by background + noise in the object region — forcing geometry-following. Stage 2 freezes the geometry controller and fine-tunes the base; the texture stream is supplied with probability 20% and replaced by background with probability 80%, so the model learns to use texture when available without becoming dependent on it.

Results

V3DBench (new): 22 videos across pose editing, rotation, scaling, translation, texture modification, object composition.
Editing metrics [Table 1]: Frame-Acc 0.970, Tem-Con 0.988, CLAP Score 0.917 — beating Tune-A-Video (0.559 / 0.474), Pix2Video (0.851 / 0.690), I2V-Edit with Image-Sculpting first-frame edit (0.887 / 0.829), and DragVideo (0.918 / 0.856). User study mean ranks: Editing Quality 1.16, Semantic Consistency 1.23 — first place on both.
Reconstruction ablation [Table 2]: monocular-only baseline 20.35 PSNR / 0.903 SSIM / 0.060 LPIPS / 0.375 DINO; adding novel views jumps to 22.87 / 0.922 / 0.057 / 0.396; adding Balanced View Sampling and depth loss reaches 23.00 / 0.923 / 0.055 / 0.397.
Training-strategy ablation [Table 3]: Stage-1-only reaches Frame-Acc 0.786 / CLAP 0.746; Stage-1+Stage-2 reaches 0.970 / 0.917.
Qualitative coverage: pose editing, rotation, scaling, translation, texture modification, and 3D-to-2D composition all demonstrated in the supplementary [§5, Fig. 6, Fig. 10].

Why it’s interesting

Shape-for-Motion is a 2025 object-centric precursor to the 2026 4D-control cluster: where VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control exposes a scene-level editable state (background point cloud + per-object Gaussian trajectories) and ActionMesh: Animated 3D Mesh Generation with Temporal 3D Diffusion generates topology-consistent animated meshes from scratch, Shape-for-Motion takes the opposite path — lift an existing video object to a topology-consistent mesh, edit, re-render. It’s the cleanest filed example of the explicit-state philosophy The flavor of the bitter lesson for computer vision critiques: the 3D mesh is the user-facing interface and the deformable-3DGS reconstruction is the load-bearing piece. The Dual-Propagation Strategy is a worked example of a non-obvious failure mode for any “edit canonical → propagate via deformation field” pipeline: the deformation field is faithful for the points it was trained on (Gaussians) but accumulates error on a different point set (mesh vertices), and the fix is to route geometry and texture through their respective faithful primitives. This also predates the Vista4D: Video Reshooting with 4D Point Clouds / Reshoot-Anything: A Self-Supervised Model for In-the-Wild Video Reshooting reshoot-anything camera-control wave, but tackles a complementary task — object manipulation rather than novel-view rendering — with a thematically related “3D proxy + diffusion cleanup” recipe.