VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control

TL;DR

VerseCrafter is a video world model that exposes an explicit, editable 4D world state — a static background point cloud plus per-object 3D Gaussian trajectories in a shared world frame — and renders it into per-frame conditioning maps for a frozen Wan2.1-14B diffusion backbone via a lightweight GeoAdapter. The Gaussian-trajectory representation gives soft, category-agnostic control over object position, extent, and orientation, sidestepping the rigidity of 3D bounding boxes and the category-specificity of SMPL-X. Training uses VerseControl4D, a new 35k-clip in-the-wild dataset with automatically derived camera + multi-object 3D Gaussian trajectory annotations, built from Sekai-Real-HQ and SpatialVID-HQ. The model achieves joint camera + multi-object control with substantially lower trajectory error than Perception-as-Control, Yume, Uni3C, ViewCrafter, Voyager, and FlashWorld.

Key claims

A 4D Geometric Control state composed of a static background point cloud and per-object 3D Gaussian trajectories in a shared world frame is sufficient to drive unified, disentangled camera and multi-object motion control through a frozen video diffusion backbone [§3.1, §3.2].
3D Gaussian trajectories beat both 3D bounding boxes and 3D point trajectories as object motion controls: Overall VBench score 88.10 vs 85.45 (3D box) vs 85.57 (3D point), and ObjMC 2.51 vs 4.52 vs 6.90 [Table 3].
Decoupled background and foreground control streams outperform a single merged control stream, with ObjMC dropping from 3.73 (merged) to 2.51 (decoupled) [Table 3].
Adding a depth channel to RGB control maps is necessary for correct foreground/background ordering and parallax — removing depth raises RotErr from 0.89 to 1.18 and ObjMC from 2.51 to 4.93 [Table 3, §5.3].
On joint camera+object control on VerseControl4D, VerseCrafter beats Perception-as-Control, Yume and Uni3C across VBench-I2V quality metrics and reduces RotErr to 0.890 (best baseline 1.361), TransErr to 3.103 (best 7.731), ObjMC to 2.507 (best 5.883) [Table 1].
On camera-only control on the static-scene split, VerseCrafter beats ViewCrafter, Voyager and FlashWorld with RotErr 0.650 vs best baseline 2.101 and TransErr 2.587 vs best 3.880 [Table 2].
The backbone (Wan2.1-14B) stays frozen; only GeoAdapter (initialized from its paired DiT block, attached every 5th DiT layer) and a small projection are trained, with 2,500 + 2,500 iterations over two resolutions on 16×96 GB GPUs (~380 GPU-hours) [§5 Implementation Details].
Annotation is fully automatic: MegaSAM with MoGe-2 + UniDepth V2 for depth, Grounded-SAM2 for instance masks, Qwen2.5-VL-72B for captions, then per-frame Gaussian fitting yields the 4D supervision needed for both training and evaluation [§4 Data annotation].
Known limitation: ellipsoid-level Gaussians give principal-axis orientation only, not full 6D pose; near-isotropic shapes (human-like single ellipsoids) can yield facing-direction ambiguities [Project page Limitations].

Method

VerseCrafter has two pieces: the 4D Geometric Control representation and the GeoAdapter that injects it into a frozen video diffusion backbone. From an input frame the system estimates monocular depth with MoGe-2 and instance masks with Grounded SAM2, back-projects per-pixel points into a world-frame point cloud, partitions it into a static background cloud and per-object clouds, and fits a full-covariance 3D Gaussian to each object. A 3D Gaussian trajectory is the sequence of these per-frame Gaussians — means trace a 3D motion path, covariances encode shape, size, and orientation evolution. Users edit trajectories by manipulating ellipsoid meshes in Blender; the underlying Gaussians are what the model actually conditions on. At each target frame the system renders five per-frame control maps in the target camera view: background RGB, background depth, 3D Gaussian trajectory RGB, 3D Gaussian trajectory depth, and a soft control mask (inverted background visibility merged with the projected Gaussian footprints). The first frame is replaced with the input image and its mask zeroed so it is preserved.

GeoAdapter is a lightweight DiT-style branch matching Wan2.1’s token dimensionality but with far fewer layers. The four RGB/depth maps are encoded by the same 3D VAE as the video latent; the mask is reshaped to latent resolution. The five streams are concatenated channel-wise into a geometry tensor, and every 5th Wan-DiT block is paired with a GeoAdapter block whose output is linearly projected and added as a residual modulation. Text conditioning (umT5) is shared by both. Background and Gaussian-trajectory channels are kept decoupled by design so camera edits only affect the background stream and object edits only affect the Gaussian stream, enabling geometry-consistent independent control. At inference the same model handles camera-only (zero out Gaussian channels), object-only (fix camera, render static background), and joint control.

VerseControl4D is built in four stages from Sekai-Real-HQ (~26%) and SpatialVID-HQ (~74%): PySceneDetect for shot extraction (81-frame clips matching Wan2.1’s temporal length), Grounded-SAM2-based filtering on controllable object count, mask area (<60%), and human aspect ratios, then aesthetic/luminance filtering. Annotation uses Qwen2.5-VL-72B for captions and a modified MegaSAM pipeline (MoGe-2 + UniDepth V2 for depth) plus per-frame Gaussian fitting. Result: 35k training clips + 1k val/test clips, with 20% static scenes for camera-only evaluation.

Results

Joint camera + multi-object control on VerseControl4D [Table 1]: Overall VBench-I2V 88.10 (Perception-as-Control 83.66, Yume 85.47, Uni3C 83.55). 3D control metrics — RotErr 0.890 / TransErr 3.103 / ObjMC 2.507, with the closest baseline (Uni3C) at 1.361 / 7.731 / 5.883.

Camera-only control on the static-scene split [Table 2]: Overall 86.80 (ViewCrafter 84.04, Voyager 78.12, FlashWorld 81.80). RotErr 0.650 / TransErr 2.587, with the closest baseline (ViewCrafter) at 2.101 / 9.868.

Ablations on representation and architecture [Table 3]: 3D Gaussian vs 3D bounding box vs 3D point trajectory — Overall 88.10 / 85.45 / 85.57; ObjMC 2.51 / 4.52 / 6.90. Removing depth degrades RotErr (0.89 → 1.18) and ObjMC (2.51 → 4.93). Merging background and foreground control streams raises ObjMC to 3.73.

Inference cost: 81-frame, P-resolution clip generation on 8×96 GB GPUs takes ~1152 s, peak memory ~90 GB [§5 Implementation].

Why it’s interesting

VerseCrafter completes a 3-paper cluster on camera-controlled video diffusion (SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time, NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos) and on 4D scene generation, but takes a different stance on the control representation: rather than disentangling time from camera (SpaceTimePilot) or feeding the diffusion model a reconstructed-then-degraded 4DGS rendering (NeoVerse), it exposes object dynamics as editable 3D Gaussian trajectories in a shared world frame. This is a clean middle ground — softer than 3D bounding boxes, more general than SMPL-X, more compact than per-point trajectories — and the ablations directly support that all three alternatives are worse on object-motion accuracy. The Wan2.1-14B-frozen + GeoAdapter recipe also rhymes with SpaceTimePilot (Wan2.1 T2V-1.3B + ReCamMaster-style camera embedder) and NeoVerse (Wan-T2V-14B + control branch); together these three papers establish “frozen Wan + lightweight geometric adapter” as a default architecture for 4D-controllable video.