4DAnyone: Create Anyone in 4D from a Casual Monocular Video

TL;DR

4DAnyone reconstructs 4D humans (3D + time) from a single uncalibrated monocular video by generating dense multiview-consistent target videos with a camera-controlled DiT and lifting them into 4D Gaussian Splatting. The core diagnosis is that when the number of target views exceeds a single DiT forward pass, splitting into groups exposes two coupled bottlenecks: reference-context grows as O(N) and disjoint target groups cannot exchange information. Reference Context Packing (RCP) compresses reference views into a fixed-length mixed-resolution context (O(1)) and Target Context Routing (TCR) rotates target groupings across denoising steps so information propagates across the full view set. Trained on a purpose-built game-engine multiview dataset (MVGameHuman) combined with light-stage and in-the-wild data. Ships with open code, weights, and project page.

Key claims

Scaling camera-controlled video diffusion to the tens of views required for 4DGS is a bounded-attention-context problem, not a modelling one: single-DiT passes cannot fit the required target views, and naive grouping causes both weakened reference guidance (O(N) growth) and global structural drift across groups [Abstract].
Reference Context Packing compresses a growing set of previously generated reference views into a fixed-length, mixed-resolution context, yielding O(1) reference-context complexity independent of the number of already-generated groups [Abstract].
Target Context Routing rotates target-view groupings during denoising — sharing context across groups at high-noise steps for global structure and stabilizing details at low-noise steps within local groups [Abstract].
The two designs are complementary, not substitutable: RCP addresses the reference-side bottleneck and TCR addresses the target-side bottleneck, and both are needed for full multiview consistency [Abstract].
4DAnyone outperforms prior methods on DNA-Rendering and DyMVHumans in both novel-view video quality and downstream 4DGS reconstruction, and generalises robustly to in-the-wild casual monocular video [Abstract].

Method

Given a monocular video of a person, 4DAnyone generates a large set of multiview-consistent target-view videos with a camera-controlled DiT, then lifts them into a 4D Gaussian Splatting representation for free-viewpoint rendering across time. Because the number of target views needed for reconstruction-grade 4DGS exceeds one DiT forward pass, generation runs in groups. RCP maintains a fixed-length mixed-resolution reference context — previously generated views are packed rather than concatenated at native resolution — so appearance guidance from all prior groups is preserved without O(N) attention cost. TCR alternates which target views are grouped together across denoising steps: at high noise levels, groupings mix so that global structure is negotiated across the full view set; at low noise, groupings stabilize so that within-group detail is not disrupted.

Training data is a mixture of MVGameHuman (built with an in-house game engine, presumably synthetic multiview human sequences), light-stage captures, and in-the-wild monocular video. Evaluation is on DNA-Rendering and DyMVHumans, covering both novel-view video quality and downstream 4DGS reconstruction quality.

Results

Headline: outperforms prior methods on DNA-Rendering and DyMVHumans across novel-view video quality and downstream 4DGS reconstruction; robust in-the-wild generalization from casual monocular capture [Abstract]. Specific metric deltas are not disclosed in the abstract on file at ingest time.

Why it’s interesting

Explicit diagnosis of the group-boundary problem in camera-controlled video diffusion at reconstruction scale. Existing filed papers (NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos, VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control, SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time) all attack camera control on a frozen Wan-family DiT, but they operate in the single-forward-pass regime; 4DAnyone is the first filed paper to name and address the out-of-pass regime that 4DGS lifting demands.
RCP is a novel operating point in the Context Length / Quality Trade-off in Video Generation cluster: instead of evicting or sinking old tokens, it packs them into a fixed-length mixed-resolution context specifically to keep O(1) cost while preserving cross-group appearance guidance. Contrasts with SCoPE’s (SCoPE: Sightline-Coordinate Positional Encoding for Video Diffusion Transformers) additive-Plücker approach to consistency by construction — 4DAnyone’s consistency comes from context management, not from geometric positional encoding.
Complements the reconstruct-then-generate branch (NeoVerse) and the control-first branch (VerseCrafter, SpaceTimePilot) with a generate-then-reconstruct recipe explicitly tuned for lifting into 4DGS. Sits alongside Lift4D: Harmonizing Single-View 3D Estimation for 4D Reconstruction In-the-Wild as an object-centric (here: human-centric) monocular-video-to-4D method, but distinct in that 4DAnyone’s contribution is on the multiview generator’s context scaling rather than on the reconstruction-side canonical model.
Continues the pattern of synthetic supervision being load-bearing in this cluster — MVGameHuman joins CamTime, VerseControl4D, and NeoVerse’s online degradation in the Synthetic Training Data roster, this time from a game engine specifically for multiview human sequences.