Video World Models with Long-term Spatial Memory

TL;DR

SPMem augments an autoregressive video world model with a geometry-grounded long-term spatial memory: an incrementally fused static point cloud (via TSDF-Fusion over per-frame static depth maps) plus a sparse set of representative historical keyframes (episodic memory) plus the standard recent-frame context window (working memory). The memory is injected into a frozen CogVideoX-5B-I2V backbone (initialized from DiffusionAsShader) via a ControlNet-style condition DiT that consumes static-point-cloud renderings of the target trajectory, plus a historical cross-attention head over keyframe tokens. On a 500-clip MiraData test set with reversed-trajectory view-recall evaluation, SPMem reaches PSNR 19.10 / SSIM 0.6471 / LPIPS 0.3069 versus TrajectoryCrafter’s 11.71 / 0.4380 / 0.5996 and Wan2.1-Inpainting’s 12.16 / 0.4506 / 0.5875.

Key claims

Existing video world models forget previously-visited environments on revisit because their attention windows are short and their memory is purely image-based, lacking persistent 3D understanding [§1].
A geometry-grounded long-term memory cast as three human-memory-inspired components — spatial (TSDF-fused static point cloud), working (recent frames), episodic (sparse keyframes) — is sufficient to enforce long-horizon scene consistency on autoregressive video generation [§3.2].
TSDF-Fusion inherently filters out dynamic objects: their depth observations are temporally inconsistent across frames, so their voxels accumulate low-confidence noisy SDF values and are naturally suppressed in the final fused volume [§3.2, Eq. 2].
At inference time CUT3R is used instead of Mega-SaM for autoregressive 4D reconstruction because it incrementally updates a persistent state and avoids the CUDA-memory growth of NDC-coordinate reconstruction over long videos [Appendix A].
The episodic memory is populated by an unknown-region-visibility criterion — when the mask of newly revealed unseen pixels in a generated frame exceeds a threshold, that frame is added to the keyframe set [§3.2].
Static point-cloud renderings of the target trajectory are encoded by the pre-trained CogVideoX 3DVAE and fed through a copy of the first 18 DiT blocks (the “condition DiT”) with zero-initialized linear projections back into the main DiT — the ControlNet recipe applied to a video DiT [§3.3].
The last five source frames are concatenated with the target tokens along the frame dimension as dynamic-context guidance; an additional historical cross-attention layer treats target tokens as queries and keyframe tokens as keys/values [§3.3].
The training set is built from MiraData by segmenting each video into 97-frame clips (49 source + 48 target + 1 shared transition frame), running Mega-SaM for camera intrinsics/extrinsics/depth, fusing source frames into a TSDF, and rendering target-camera visibility masks and static reconstructions as paired training data — 90k samples total [§3.4].
On reversed-trajectory view-recall metrics SPMem nearly doubles PSNR over the strongest geometry-grounded baseline (19.10 vs 12.16 for Wan2.1-Inpainting; vs 12.01 for DaS) and cuts LPIPS roughly in half [Table 1].
Per-axis user-study Average Human Rankings (1–4, higher better) place SPMem first on camera accuracy (3.63), static consistency (3.38), and dynamic plausibility (3.40), versus DaS as best baseline at 2.57 / 2.44 / 2.70 [Table 1].
On VBench general video quality SPMem leads on Aesthetic Quality (0.5835), Temporal Flickering (0.7580), Motion Smoothness (0.9886), and Subject Consistency (0.9359); Wan2.1-Inpainting wins Imaging Quality (0.6788 vs 0.6701) because its inpainting prior tends to generate static scenes that trivially maximize background consistency [Table 2, §4.1].
Ablation shows all three memory types contribute: removing working memory drops Imaging Quality 0.6701→0.6384 and Temporal Flickering 0.7580→0.6740; removing episodic memory drops Aesthetic 0.5835→0.5603 and Motion Smoothness 0.9886→0.9870 [Table 3].
The spatial-memory mechanism specifically targets spatial consistency; frame-packing approaches like FramePack-F1 primarily target character/identity consistency — combining them remains future work [§5].
Stated limitations: TSDF-Fusion introduces artifacts when revisits come from very different camera poses than the original observations, and the system does not address generation drift (image-quality degradation due to error accumulation over time) [§5].

Method

The system runs an autoregressive next-clip diffusion model with three memory streams updated at each step. The short-term working memory is the standard sliding window of the most recent 5 frames concatenated along the frame dimension with the target latents. The long-term spatial memory is a TSDF voxel volume that is updated each autoregressive step by running CUT3R on newly generated frames to extract a static point map and fusing it into the volume; for each new clip, the volume is rendered along the upcoming camera trajectory (with un-occupied regions set to black), 3DVAE-encoded, and fed through a CogVideoX-block condition DiT that adds zero-initialized residuals into every main DiT block (ControlNet recipe). The sparse episodic memory is a growing list of historical keyframes selected by visibility-mask thresholding; their 3DVAE tokens act as keys/values in a dedicated historical cross-attention layer where the target video tokens are queries.

The training corpus is 90k synthetic source-target pairs derived from MiraData. Source/target are 49+48 frames sharing one transition frame; Mega-SaM gives intrinsics, extrinsics, and per-frame depth on the source; TSDF-Fusion suppresses dynamic content; rendering through the target camera pose yields the static-region guidance and visibility mask for that target. Training is 6k iterations at 480×720, batch size 8, learning rate 1e-5 on 8×A100, with CogVideoX-5B-I2V (DiffusionAsShader-initialized) frozen except for the condition DiT, the historical cross-attention layer, and the LoRA-style zero-init residual projections.

Results

Headline view-recall (forward then reversed trajectory, 500 held-out MiraData clips): PSNR 19.10 vs 12.16 (Wan2.1-Inpainting), SSIM 0.6471 vs 0.4506, LPIPS 0.3069 vs 0.5875; DaS and TrajectoryCrafter both worse than Wan2.1-Inpainting on PSNR. User study (20 video/3D/4D-experienced raters, 14 cases, Average Human Ranking 1–4): Camera-Accuracy 3.63, Static-Consistency 3.38, Dynamic-Plausibility 3.40 — first place on all three, beating DaS (the strongest baseline) by margins of 1.06 / 0.94 / 0.70. VBench general-quality numbers cluster within 1–4 points of Wan2.1-Inpainting; SPMem leads on motion-relevant metrics (Motion Smoothness, Temporal Flickering, Subject Consistency, Aesthetic Quality) while Wan2.1’s inpainting prior wins Imaging Quality due to its tendency toward static scenes. Ablations remove one memory at a time and find each removal hurts on a different axis: working memory drives short-term motion coherence (drops Temporal Flickering by 0.084), episodic memory preserves distant detail (drops Subject Consistency by 0.003 and Background Consistency by 0.005), and spatial memory yields the largest combined headroom (full model is best on all six VBench dimensions).

Why it’s interesting

This is the first filed paper in the wiki that handles the autoregressive-video forgetting–drifting dilemma with an external explicit 3D state rather than a learned representation. The existing AR-video long-context cluster (see Autoregressive Video Generation) covers four learned approaches — partitioned KV caches in Context Forcing: Consistent Autoregressive Video Generation with Long Context, explicit memory compression in Pretraining Frame Preservation in Autoregressive Video Memory Compression, partition-plus-mid-compression in PackForcing: Short Video Training Suffices for Long Video Sampling and Long Context Inference, and a fully recurrent memory vector in MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation — all of which keep memory inside the network’s representational space. SPMem instead grounds memory in an explicit point-cloud TSDF that can be inspected, edited, and queried geometrically, at the cost of needing CUT3R/Mega-SaM in the pipeline. The result complements the position-essay critique on file from The flavor of the bitter lesson for computer vision (which argues explicit 3D is a hand-crafted intermediate the Bitter Lesson should dissolve): SPMem is the strongest current existence proof that the explicit-3D recipe pays off on quantitative view-recall metrics, exactly the metric that pure rollout-based world models struggle on. The design also overlaps the 4D-scene-generation cluster — the static-point-cloud-as-ControlNet-condition recipe is essentially the same control surface as VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control and NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos, but applied to temporal extension rather than to camera/object trajectory control.