Reshoot-Anything: A Self-Supervised Model for In-the-Wild Video Reshooting

TL;DR

Reshoot-Anything is a video-reshooting method that re-renders a monocular input video along a user-specified camera trajectory, trained entirely on monocular videos via self-supervision. The trick is a pseudo multi-view triplet: two distinct smooth random-walk crop trajectories from the same source clip become source/target views, and the anchor is synthesized by forward-warping the source’s first frame with a dense 2D tracking field — exactly the kind of distorted, hole-ridden signal a 4D-reconstruction anchor produces at inference. A Wan 2.2-14B MoE I2V backbone is adapted with rank-512 LoRA, token-concatenated source + anchor conditioning, and an Offset RoPE (constant temporal offset of 50 on source tokens). The model is the closest paired comparison point to Vista4D: Video Reshooting with 4D Point Clouds (published the same day with the same baselines) and beats TrajectoryCrafter / EX-4D / ReCamMaster on view synchronization and temporal consistency on 100 Opensora-mixkit clips.

Key claims

Pseudo multi-view triplets can be synthesized from a single monocular video by taking two independent smooth random-walk crop trajectories as source/target and forward-warping the source’s first frame (via AllTracker dense tracks + crop-offset flow + softmax splatting) into an anchor [§3.1.1, Fig. 1, Fig. 3].
Because the source and target crops are not spatially aligned, the model cannot just copy the corresponding source frame to fill the target — it is forced to attend to other frames of the source to retrieve textures for dis-occluded regions, which the authors describe as an “information erasure mechanism” that teaches implicit 4D reconstruction [§3.1.2, Fig. 4].
Forward warping (rather than backward warping) is deliberately chosen so the anchor exhibits the same artifacts, occlusions, and dis-occlusions that 4D-point-cloud anchors show at inference [§3.1.1 Eq. 1].
Self-attention token concatenation of source + anchor outperforms a dedicated cross-attention path for source conditioning on all camera/view-sync/quality metrics [§5.3, Table 2 — w/ cross-attn row regresses on every metric].
Offset RoPE — a constant 50-frame temporal offset added to the source tokens’ 3D-RoPE temporal coordinate — decouples source positional context from the target denoising trajectory; the offset exceeds the maximum 21 latent frames so source and target never overlap in RoPE space [§3.2.2, §4.2].
A source-token reconstruction auxiliary L1 loss (λ = 0.1) on the source-stream outputs encourages the source pathway to actually preserve high-fidelity content rather than degrade [§3.2.2, §4.3].
3D-aware noise injection — adding Gaussian noise (per-channel U[0, 0.5] on normalized images) to the source’s reference frame before forward warping, so noise moves coherently with structure — prevents the model from copying anchor textures and improves RotErr from 3.27 → 2.49 [§3.2.3, Table 2].
A 15% mixture of synthetic ReCamMaster multi-view data is needed for extreme out-of-distribution camera trajectories (e.g. 120° orbits); monocular-only training generalizes well to moderate motion but breaks on extreme paths [§4.1, Fig. 7, Table 2 — Ours 2.76/4.23 vs Ours + Syn 3.36/3.66 RotErr/TransErr trade].
Rank-512 LoRA on attention + FFN layers + full training of the patchify layer is enough to adapt the 14B MoE backbone; training is 2K steps, batch 24, lr 1e-5 AdamW [§4.3].
High-noise and low-noise MoE experts are trained independently; the high-noise expert dominates camera-motion control (early denoising timesteps) so anchor-video ablations are all done on the high-noise side [§4.3].
On 100 Opensora-mixkit five-second 480p clips, Reshoot-Anything beats TrajectoryCrafter, EX-4D, and ReCamMaster on VBench Aesthetic, Imaging, Flickering, Subject, Background, CLIP-F, FVD-V, and CLIP-V, with competitive (not best) RotErr/TransErr [Table 1].
The “Ours w/o Source” variant exhibits the same texture-drift failure mode as anchor-only EX-4D, confirming the source-video branch is what preserves fine details [§5.2, Fig. 8].
A fluorescent-pink background in masked anchor regions (instead of black) provides a clearer boundary signal especially in dark scenes [§3.2.3, §5.3].

Method

The setup adapts a Wan 2.2-14B MoE I2V DiT. Both source video $V_s$ and anchor video $V_a$ are encoded by the pre-trained 3D VAE. The anchor stream concatenates $V_a$ ‘s latent with a noise latent and a downsampled binary validity mask; the source stream replaces the noise channels with a duplicated source latent + all-ones mask. The two streams are temporally concatenated, producing a 42-frame token sequence (twice the standard 21). Offset RoPE adds +50 to the source tokens’ temporal RoPE coordinate so the two streams cannot share temporal positions. Adaptation is rank-512 LoRA on attention + FFN layers plus full training of the patchify layer.

The self-supervised data pipeline is the load-bearing piece. From a single in-the-wild monocular video, two independent smooth-random-walk crop trajectories produce source and target clips. AllTracker on the original uncropped video provides dense 2D tracks; combined with each clip’s crop-offset flow, the resulting offset-aware flow field is used to forward-warp (softmax splatting) the source’s first frame to every target time, yielding the anchor. The dataset is ~100k clips from ~30k high-resolution monocular videos plus a 15% mixture of synthetic ReCamMaster multi-view pairs for extreme-trajectory coverage. Training augmentations include 3D-aware noise on the reference frame before warping, fluorescent anchor background, random anchor reference-frame selection, and an L1 source-reconstruction auxiliary loss (λ = 0.1).

At inference, the source video is converted to a 4D point cloud, the user-specified target trajectory renders a (distorted) anchor video, and the model produces the new target. The same Offset-RoPE token-concat path is used unchanged.

Results

100 five-second 480p videos from Opensora-mixkit, balanced for semantic concept, camera motion diversity, and occlusion/dis-occlusion mix [§5.1].
Camera accuracy vs view synchronization vs VBench quality [Table 1, default 81-frame inference]: Reshoot-Anything posts the best Aesthetic 52.85, Imaging 58.64, CLIP-F 99.03, FVD-V 586.24, CLIP-V 93.16 among ReCamMaster / EX-4D / Reshoot-Anything; on RotErr 2.76 it beats ReCamMaster 11.29 and EX-4D 3.94, and on TransErr 4.23 it beats ReCamMaster 19.59 (EX-4D 4.21 is essentially tied).
Head-to-head with TrajectoryCrafter at TC’s native 49-frame setting [Table 1]: Reshoot-Anything wins on Flickering 97.43, Smoothness 99.24, Subject 95.09, Background 95.62, CLIP-F 99.01, FVD-V 488.22, CLIP-V 94.96; TC wins on RotErr 2.26 vs 2.61. Aesthetic essentially tied (52.69 vs 52.72).
Ablations [Table 2]: removing the source-video branch collapses Mat. Pix from 2636 → 226 and CLIP-V from 92.94 → 89.97 (the texture-drift failure mode); adding 3D noise in the anchor instead of in the latent improves RotErr 2.81 → 2.49 at same TransErr; cross-attention conditioning regresses Mat. Pix 2636 → 1766; LoRA is competitive with full fine-tuning at a fraction of cost.
Ours + Syn vs Ours: the 15% synthetic mixture trades 0.6 RotErr for 0.57 TransErr improvement and is the configuration that handles extreme orbits without losing complex real-world dynamics.

Why it’s interesting

Reshoot-Anything is the natural contrast partner to Vista4D: Video Reshooting with 4D Point Clouds: same task (video reshooting), same backbone family (Wan 2.x), same baselines (TrajectoryCrafter / EX-4D / ReCamMaster), submitted the same day. The two papers disagree philosophically about what the training data should look like: Vista4D argues you must train on noisy 4D-reconstructed multiview so the model learns geometry correction; Reshoot-Anything argues you can sidestep multi-view entirely by manufacturing misalignment via independent crops on monocular video — the model is then forced into the same “implicit 4D routing” regime by an information-erasure bottleneck rather than by reconstruction noise. The Offset-RoPE token-concat scheme is also a much lighter conditioning recipe than the GeoAdapter-every-5th-layer pattern in VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control or the dedicated control branches in NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos. For the Camera-Controlled Video Diffusion cluster this is the strongest filed evidence yet that “frozen Wan + LoRA + clever data” can compete with custom-architecture camera-control recipes; it also extends Parameter-Efficient Finetuning‘s evidence to large-scale (14B MoE) video DiT adaptation with a single rank-512 LoRA.