Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrieval

TL;DR

Scene-consistent interactive long-video generation by treating all previously generated frames as memory and retrieving relevant ones via camera-FOV overlap. Two design choices: (1) store frames directly (no 3D reconstruction, no learned compression); (2) condition by concatenating retrieved context frames with predicted frames along the frame dimension at DiT input, using a separate positional-encoding range. The Memory Retrieval module reduces hundreds of candidates to ~20 by computing FOV co-visibility from known camera poses, then deduplicating adjacent frames. Trained on a 100-video UE5 dataset with precise camera annotations; generalizes to open-domain images at inference.

Key claims

“All historical generated frames as memory” is the underlying recipe for scene-consistent long video — alternatives like fixed-window Diffusion Forcing or hierarchically-compressed FramePack either ignore distant frames or lose information through exponential decay [§1, §2.2].
Context can be injected by frame-dimension concatenation with separate RoPE positions for the noisy predicted latents vs. the clean context latents, keeping the base model’s positional encoding intact for the prediction window [§3.2].
A camera-FOV-overlap rule is sufficient to determine co-visibility between two frames given their camera poses, by intersecting the four left/right rays of the two FOV fan-shapes restricted to the XY plane [§3.3, Fig. 4].
Adjacent-frame redundancy is a real problem at the retrieval stage — taking only one frame per group of consecutive candidates (“Non-adj”) improves PSNR from 19.17 to 20.11 on Ground Truth Comparison [Table 3].
A “Far-st” tiebreaker (selecting spatially/temporally distant frames among candidates) gives a small additional bump on LPIPS but is mostly redundant with “Non-adj” [Table 3].
Context-as-Memory beats DFoT (fixed-window) and FramePack (hierarchical compression) on PSNR / LPIPS / FID / FVD on both Ground Truth Comparison and the harder History Context Comparison protocol, on a 1B-parameter Wan-style DiT [Table 1].
More context helps but with diminishing returns past 20 frames: PSNR 20.22 at size 20 vs 20.31 at size 30, while inference FPS drops from 0.97 to 0.79 [Table 2].
Random retrieval beats DFoT and FramePack on average — accessing distant context (even at random) matters more than ordering many recent frames [Table 1].
The History Context Comparison protocol (newly generated frames vs. earlier generated frames at the same camera pose) shows much wider method gaps than Ground Truth Comparison, isolating memory capability from one-shot generation quality [Table 1, §4.1].
Open-domain generalization is demonstrated qualitatively: trained on UE5 scenes, the model maintains memory consistency on internet images of unseen styles when run on the “rotate away and back” trajectory [§4.4, Fig. 6].
Known limitations: static scenes only; FOV-overlap fails under occlusions (e.g. interconnected indoor rooms); long-video error accumulation persists [§5].

Method

The base model is a 1B-parameter text-to-video Diffusion Transformer with a causal 3D VAE (temporal compression factor 4), built on a Wan-style stack of spatial → spatial-temporal → cross-attention → FFN blocks. Camera control is injected ReCamMaster-style: per-frame camera poses are mapped through a small MLP camera encoder to feature-channel dimension and added to the output of spatial attention before 3D attention.

Context Learning. During training a long ground-truth video (thousands of frames) is sampled; a 21-frame segment becomes the prediction window, and Memory Retrieval picks up to 20 context frames from the remainder. These context frames are individually encoded through the causal 3D VAE (yielding 20 latent frames) and concatenated with the noisy predicted latents along the frame dimension. The clean context latents participate equally in DiT attention but only the noisy latents receive the predicted noise output. The predicted latents keep the base model’s original RoPE positions; context latents get new positions appended.

Memory Retrieval. Given the camera trajectory of all generated context frames and the (target) camera trajectory of the prediction window:

FOV co-visibility filter. For each (context, prediction) frame pair, intersect the four boundary rays of the two FOV fan-shapes in the XY plane. Filter out pairs with no intersection or with origin-to-intersection distance above a threshold.
Adjacent deduplication (“Non-adj”). Among surviving candidates, take one random frame per run of consecutive context frames.
Far-st tiebreaker (optional). If candidates still exceed the budget, prefer frames temporally/spatially farther from the prediction window.
The most-recent frame is always added unconditionally to ensure continuity, and the first prediction frame is included as an extra context frame.

During training, with some probability only the recent frame is used (no context), simulating the cold-start of long-video generation.

Dataset. 100 Unreal Engine 5 videos of 7,601 frames each, 12 scene styles, multimodal-LLM captions every 77 frames, camera trajectories restricted to XY-plane translation + Z-axis rotation. Context size 20 → ~5 RGB frames → ~5 latent frames after VAE encoding.

Results

On the UE5 holdout (20% of dataset, “rotate forward then back” trajectories for memory evaluation):

Ground Truth Comparison (predicted vs. ground-truth, context from GT frames): PSNR 20.22 / LPIPS 0.3003 / FID 107.18 / FVD 821.37 vs. DFoT 17.63 / 0.4528 / 112.96 / 897.87 and FramePack 17.20 / 0.4757 / 121.87 / 901.58 [Table 1].
History Context Comparison (newly generated vs. previously generated at same camera pose): PSNR 18.11 / LPIPS 0.3414 / FID 113.22 / FVD 859.42 vs. DFoT 15.70 / 0.5102 / 121.18 / 919.75 and FramePack 15.65 / 0.4947 / 131.59 / 974.52 [Table 1].
Context-size ablation. PSNR scaling 15.72 (size 1) → 17.37 (5) → 19.14 (10) → 20.22 (20) → 20.31 (30), with FPS dropping from 1.60 to 0.79 [Table 2].
Retrieval-strategy ablation. Random 17.70 → FOV+Random 19.17 → FOV+Non-adj 20.11 → FOV+Non-adj+Far-st 20.22 (Ground Truth PSNR) [Table 3].
50 sampling steps, classifier-free guidance on the text prompt, 384×384 / 21-frame generations.

Why it’s interesting

Slots cleanly into the autoregressive-long-video Pareto laid out in Context Length / Quality Trade-off in Video Generation as a sixth operating point: retrieve, don’t compress. The four prior filed operating points all do some form of compression — PFP (Pretraining Frame Preservation in Autoregressive Video Memory Compression) compresses through a learned encoder that bypasses the VAE bottleneck; Context Forcing (Context Forcing: Consistent Autoregressive Video Generation with Long Context) partitions the cache and drops the mid-region; PackForcing (PackForcing: Short Video Training Suffices for Long Video Sampling and Long Context Inference) partitions and learnedly compresses the mid; MALT (MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation) collapses everything into one recurrent vector. CaM keeps the full history uncompressed in storage and instead uses cheap rule-based retrieval — interesting because its compression strategy is “select few”, not “average many”. The accompanying tweet frames this as the long-term memory analogue to Google DeepMind’s Genie 3: A new frontier for world models / Project Genie: Experimenting with infinite, interactive worlds.

The FOV-overlap retrieval also makes the paper a clean datapoint for Camera-Controlled Video Diffusion — it inherits the ReCamMaster camera-encoder lineage and exploits known-camera-poses-from-control as the retrieval key, which is structurally cheaper than the 3D-reconstruction-based memory of WorldMem and the explicit-3DGS state of NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos / VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control.