NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos

TL;DR

NeoVerse is a 4D world model that pairs a feed-forward, pose-free 4D Gaussian Splatting reconstructor (a “Gaussianized” VGGT with bidirectional motion features) with a Wan-T2V-14B-based novel-trajectory video generator, trained end-to-end on ~1M in-the-wild monocular video clips. The headline trick is online, on-the-fly reconstruction from sparse key frames plus three principled monocular degradation simulators (visibility-based Gaussian culling, average-geometry filter, larger-kernel filter), which manufacture the degraded-input / clean-target training pairs that monocular video alone cannot. The result is a single model that does reconstruction, novel-trajectory video generation, 3D tracking, editing, stabilization, and super-resolution while scaling its training data far beyond the multi-view or pre-processed corpora prior 4D world models required.

Key claims

The scalability bottleneck in current 4D world models is either limited data scalability (dependence on hard-to-capture multi-view dynamic video, as in SynCamMaster / ReCamMaster) or limited training scalability (cumbersome offline reconstruction/depth pre-processing, as in TrajectoryCrafter / FreeSim) [§1].
“Gaussianizing” VGGT — i.e. predicting per-pixel 4D Gaussian attributes (position from depth back-projection, plus opacity, rotation, scale, SH, life-span, and bidirectional linear/angular velocities) on top of VGGT’s frame features — yields a feed-forward, pose-free 4DGS reconstructor in a single forward pass [§3.1].
Bidirectional motion modelling (separate forward and backward velocity heads from frame features split along the temporal axis) is what enables clean Gaussian interpolation between adjacent timestamps, which is the mechanism that makes sparse-keyframe training work [§3.1, §3.2 Eq. 3-5].
Online monocular degradation simulation manufactures realistic novel-trajectory degraded renderings from monocular video alone via three principled effects: visibility-based Gaussian culling (occlusion), small-kernel average filter on rendered depth (flying-edge pixels), and larger-kernel filter (broader distortion) [§3.2, Fig. 3].
The control branch is the only thing trained in stage 2; the Wan-T2V-14B base is frozen, which is presented both as a training-efficiency move and as the reason NeoVerse remains compatible with off-the-shelf distillation LoRAs [§3.2, §3.3].
Static reconstruction beats prior feed-forward pose-free models — VRNeRF (16 views) PSNR 20.73 vs AnySplat 18.02, NoPoSplat 11.27; ScanNet++ (32 views) PSNR 25.34 vs AnySplat 22.79 [Table 1].
Dynamic reconstruction beats MonST3R and posed-input 4DGT on ADT (PSNR 32.56 vs 4DGT’s 30.09) and DyCheck (PSNR 11.56 vs 4DGT’s 9.94), without requiring camera poses as input [Table 2].
Sparse-keyframe inference is dramatically faster than the closest hybrid baseline: 11 key frames → total 20 s vs TrajectoryCrafter’s 159 s and ReCamMaster’s 168 s, while matching or beating their VBench scores (Subj.Consist. 88.43 / Back.Consist. 92.27 / Imag.Quality 59.75) [Table 3].
VBench scores monotonically improve from 11 → 21 → 41 → full key frames (PSNR-proxy ImgQ 59.75 → 60.01 → 60.37 → 61.51) at increasing reconstruction cost, providing an explicit quality–compute knob at inference [Table 3].
Both bidirectional motion modelling and opacity regularization are necessary: removing the latter drops PSNR to 10.86, removing the former to 11.27 vs. the full reconstruction’s 11.56 on DyCheck [Table 4].
A trade-off observed across baselines — purely generative methods (ReCamMaster) have visual quality but weak trajectory control; reconstruction-based methods (TrajectoryCrafter) have trajectory control but inferior generation; NeoVerse claims to push out both Pareto axes via its scalable monocular training [§4.3, Fig. 4].
Global motion tracking (visibility-weighted maximum projected velocity per Gaussian across the whole clip) is needed to correctly classify objects that have both static and dynamic phases as dynamic; instantaneous velocity thresholding alone misclassifies them [§3.4, Fig. 8].

Method

The architecture is two stages glued by online reconstruction. Stage 1 is a feed-forward 4DGS reconstructor built on VGGT’s DINOv2 + alternating-attention backbone. For each frame, VGGT’s frame features are duplicated and sliced along time to produce queries / keys-values for two cross-attention heads, giving forward (t → t+1) and backward (t → t−1) motion features. Pixel-level heads then predict 4D-Gaussian attributes — position is the back-projection of predicted depth using predicted camera parameters, while opacity / rotation / scale / SH / life-span come from frame features, and forward/backward linear and angular velocities come from the motion features. Gaussian interpolation at a non-key timestamp t_q transfers a nearest-keyframe Gaussian forward in time using linear position update and quaternion-from-axis-angle rotation update, with opacity decayed by a sigmoid-controlled normalized temporal distance (§3.2, Eq. 3-5). Stage 1 is supervised by photometric (L1 + LPIPS), camera, depth, motion (against GT bidirectional velocities from dynamic datasets like Kubric / PointOdyssey / Waymo / VirtualKITTI2), and opacity-regularization losses (§3.3).

Stage 2 is a control branch on top of a frozen Wan-T2V-14B rectified-flow video generator. For each training video, the stage-1 model runs on-the-fly on sparse key frames (with K = 11 / 21 / 41), producing a 4DGS scene. A novel trajectory is synthesized by random transforms of the predicted source trajectory (constrained to point at scene center). Three online degradations are then applied to the rendered novel-view condition: (a) cull Gaussians whose visibility from the novel viewpoint is zero, then render at the original views (occlusion artifacts); (b) render depth at novel views, smooth with an average filter, push each Gaussian’s center to that filtered depth, re-render at original views (flying-edge pattern); (c) the same with a larger kernel (broader distortion). The resulting degraded RGB + depth + opacity-derived masks + Plücker embeddings of the original camera trajectory are concatenated and injected through the trainable control branch into the frozen Wan base. Only the control branch and the small E_cam-style projection are trained; the base diffusion stays frozen, preserving compatibility with distillation LoRAs (e.g. LightX2V) for inference acceleration. At inference, an optional global motion tracking step (visibility-weighted projected-velocity maximum across the full clip) separates static from dynamic Gaussians so the static set can be aggregated across all frames while the dynamic set is aggregated only locally to avoid motion drift.

Training runs on 32× A800 GPUs: stage 1 for 150K iterations, stage 2 for 50K iterations, on 18 public datasets (ARKitScenes, DL3DV, PointOdyssey, Kubric, Waymo, SpatialVID, GFIE, …) plus a self-curated ~1M-clip in-the-wild monocular video corpus.

Results

Static reconstruction (Table 1): VRNeRF 16-view PSNR 20.73 / SSIM 0.766 / LPIPS 0.352 beats AnySplat (18.02 / 0.705 / 0.366) and NoPoSplat (11.27 / 0.408 / 0.620); ScanNet++ 32-view PSNR 25.34 / SSIM 0.834 / LPIPS 0.195 vs AnySplat 22.79 / 0.773 / 0.217.

Dynamic reconstruction (Table 2): ADT PSNR 32.56 / SSIM 0.927 / LPIPS 0.120 beats 4DGT (30.09 / 0.909 / 0.178) and MonST3R (17.42 / 0.554 / 0.534); DyCheck PSNR 11.56 / SSIM 0.293 / LPIPS 0.558 beats 4DGT (9.94 / 0.208 / 0.639) and MonST3R (9.32 / 0.103 / 0.710). 4DGT consumes ground-truth camera poses at inference; NeoVerse does not.

Novel-trajectory generation (Table 3, 81-frame outputs): NeoVerse at 11 key frames takes 2 s reconstruction + 18 s generation = 20 s vs TrajectoryCrafter’s 38 + 121 = 159 s and ReCamMaster’s 168 s. VBench Subject Consistency 88.43 / Background Consistency 92.27 / Temporal Flicker 96.77 / Motion Smoothness 98.80 / Aesthetic Quality 44.55 / Imaging Quality 59.75. Scaling key frames to full input pushes Subject Consistency to 89.42, Imaging Quality to 61.51 at total inference 28 s — still 6× faster than the baselines.

Ablations (Table 4 on DyCheck): full reconstruction PSNR 11.56 → 11.27 without bidirectional motion → 10.86 without opacity regularization. Adding the generation stage on top of reconstruction lifts PSNR to 14.59, confirming that the diffusion generator does meaningful work beyond cleaning up reconstruction.

Why it’s interesting

NeoVerse and SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time are converging on the same problem (camera-and-time-controlled video generation from monocular inputs) from opposite ends — SpaceTimePilot pushes everything through a diffusion model with a clean space/time disentanglement embedder and a synthetic full-grid dataset, while NeoVerse hands the geometry to a feed-forward 4DGS reconstructor and uses the diffusion model only to clean up the resulting renderings. Both circumvent the “no real dataset covers this control axis” problem with a synthetic-supervision trick: SpaceTimePilot via temporal-warping augmentation and Blender-rendered CamTime, NeoVerse via online geometric-degradation simulation applied to monocular video. The 1M-clip training corpus and the explicit reconstruction-time inference knob (Table 3, 11→full keyframes) are also notable — this is the first hybrid 4D world model in the wiki that is explicitly designed to scale on in-the-wild data and to expose a quality/compute Pareto at inference.