Latent Spatial Memory for Video World Models

TL;DR

Mirage replaces the standard RGB-space point-cloud “spatial memory” used in long-horizon video world models with a latent spatial memory: scene content is back-projected directly from diffusion latent tokens into a persistent 3D cache, and novel views are synthesized by warping latent tokens back into screen space — no VAE round-trip per query. Built on top of a CogVideoX-like 3DVAE-DiT backbone, Mirage reports 10.57× end-to-end speedup and 55× memory-footprint reduction versus explicit-3D baselines, while reaching 70.36 average WorldScore and strong RealEstate10K reconstruction. It is the direct successor to SPMem (Video World Models with Long-term Spatial Memory): same problem (long-horizon scene consistency in autoregressive video WFMs), same external-cache strategy, but operating on the latent manifold rather than RGB.

Key claims

Existing video world models that maintain 3D spatial consistency rely on explicit point-cloud memory in RGB space, which forces repeated rendering and VAE re-encoding round-trips and discards latent-space feature information [Abstract].
A latent spatial memory — a persistent 3D cache of diffusion latent tokens — eliminates both the information loss of pixel-space reconstruction and the recurring encode/decode compute cost [Abstract].
The cache is constructed by lifting latent tokens into 3D via depth-guided back-projection [Abstract, project page architecture diagram].
Queries are answered by synthesizing novel views through direct latent-space warping into the target view, with no VAE decode step in the inner loop [Abstract].
The unified latent-in / latent-out formulation enables 10.57× end-to-end video-generation speedup versus explicit-3D baselines [Abstract].
The same formulation reduces 3D-cache memory footprint by 55× relative to explicit-3D baselines [Abstract].
Mirage attains state-of-the-art performance on WorldScore (70.36 average reported on project page) and strong reconstruction quality on RealEstate10K [Abstract, project page].
The approach inherits the diffusion model’s geometric prior — depth-guided back-projection and latent warping are conditioned on the same latent feature distribution the generator was trained on, rather than going through an external 3D representation [Abstract].

Method

Mirage is an autoregressive video world model with three operations on a persistent latent spatial memory: initialize, read, and update. Initialization back-projects the current clip’s diffusion latent tokens into 3D using depth estimates, producing a sparse cache of (3D position → latent token) entries that encode static scene content directly in the generator’s own representational space. Reading the cache for a new target camera trajectory warps the stored latent tokens into the target view via depth-guided projection, producing a partial latent canvas that conditions the next clip’s denoising — no VAE decode + render + VAE encode round-trip is required. The cache is updated as new frames are generated: new latent tokens are back-projected and merged into the existing 3D store. The project page describes the system as a unified pipeline over a CogVideoX-style 3DVAE backbone; the static-versus-dynamic separation (a major design dimension of the SPMem predecessor) is implicit in how latent tokens persist across rollouts.

Results

Headline efficiency: 10.57× faster end-to-end video generation and 55× smaller 3D cache memory footprint versus explicit-3D baselines [Abstract]. Headline quality: 70.36 average WorldScore [project page] and “strong reconstruction quality” on RealEstate10K [Abstract] — no per-axis WorldScore breakdown, no head-to-head PSNR/SSIM/LPIPS table against SPMem or other explicit-3D baselines, and no ablation numbers are quoted in the abstract or project page at filing time. The paper claims state-of-the-art on WorldScore but the comparison table is not in the abstract.

Why it’s interesting

This is the wiki’s first paper that takes the SPMem recipe (Video World Models with Long-term Spatial Memory — TSDF-fused static point cloud + sparse keyframes + recent-frame context for autoregressive video world models) and moves the entire spatial-memory representation into latent space. SPMem’s main cost was the per-query pipeline: render the point cloud, encode through the 3DVAE, condition the DiT. Mirage’s claim is that the encode/decode round-trip was both a compute bottleneck and an information bottleneck — and that a latent-native 3D cache solves both, at 10.57×/55× margins. This positions the explicit-cache branch of AR-video memory directly against the in-network branches (Context Forcing: Consistent Autoregressive Video Generation with Long Context partitioned KV, Pretraining Frame Preservation in Autoregressive Video Memory Compression memory compression, MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation recurrent memory vector, TokenTrim: Inference-Time Token Pruning for Autoregressive Long Video Generation token pruning) and against pure rollout-based world models. It also softens the Sitzmann critique on file (The flavor of the bitter lesson for computer vision) that explicit 3D is a hand-crafted intermediate: Mirage’s 3D index is a layout for learned latents, not for RGB. Replication awaits the promised MIT-licensed code release.