Captain Safari: A World Engine with Pose-Aligned 3D Memory

TL;DR

Captain Safari is a pose-conditioned video “world engine” that adds an explicit, retrievable world memory to a frozen Wan2.2-Fun-5B-Control-Camera DiT. A pose-aware retriever fetches 3D-aware memory tokens (StreamVGGT features) keyed by the target camera pose from a local time window, and injects them via a dedicated memory cross-attention at every DiT layer. The paper also releases OpenSafari, an in-the-wild FPV drone dataset with verified 6-DoF camera trajectories (multi-stage SfM + kinematic-check pipeline) tailored to stress-test long-horizon 3D consistency under aggressive maneuvers. Against Geometry Forcing, Real-CamI2V, and Wan2.2-5B-Control-Camera, Captain Safari leads on 3D consistency and trajectory following while winning 67.4% of votes in a 50-participant 5-way human study.

Key claims

A pose-indexed, retrievable world memory beats both implicit clip-bound memory and explicit 3D scene reconstruction on the FPV-drone regime, with the memory ablation showing it is the load-bearing component for 3D consistency and trajectory following [Table 1, §5.6].
Memory is implemented by encoding (pose-token, StreamVGGT memory-tokens) pairs across a local time window, then cross-attending learnable queries — concatenated with the target pose embedding — to retrieve a pose-aligned token set per query time-step [§3.1, §3.2, Eq. 6–9].
Retrieved world tokens are injected into the DiT through a dedicated memory cross-attention layer that runs after self-attention and before the existing text-conditioning cross-attention, with weights initialized from the context cross-attention [§3.2, Eq. 11].
The memory window is a sliding, clip-aligned local window (≤W_max seconds before the clip entrance, with end overlapping the clip) rather than the full bank — keeping conditioning compact while letting neighboring clips share overlapping entries [§3.1, Eq. 1].
Pose features and 3D-aware memory features come from a pretrained StreamVGGT; four layers of StreamVGGT features (256 tokens each, 1024 total per frame) are used [§5.1].
Training is two-stage: warm up the retriever on pose-aligned memory reconstruction, then jointly train retriever + DiT with the DiT updated via LoRA [§5.1].
On OpenSafari, Captain Safari reports FVD 1023.46 (vs. Wan2.2-5B-Control-Camera 1387.75, Real-CamI2V 1585.61, Geometry Forcing 2662.75), MEt3R 0.3690 (vs. 0.3703 / 0.3932 / 0.4834), reconstruction rate 0.968 (vs. 0.923 / 0.767 / 0.877), AUC@30 0.200 (vs. 0.181 / 0.174 / 0.168), and pose-CosSim 0.563 (vs. 0.420 / 0.296 / 0.429) [Table 1].
The memory ablation (“Captain Safari w/o Mem.”) matches or beats baselines on FVD/LPIPS but is markedly worse on 3D consistency and trajectory following (reconstruction 0.912 → 0.968, CosSim 0.508 → 0.563, AUC@30 0.193 → 0.200) and loses 65–69% of human-study votes to the full model [Table 1, Table 2, §5.6].
5-way anonymized 50-participant human study (1500 preference votes) prefers Captain Safari 67.4% on video quality, 65.6% on 3D consistency, 69.0% on trajectory following — the ablated variant is a distant second at ~23%, and all three external baselines stay in single digits [Table 2].
OpenSafari is curated from AirVuz + YouTube FPV drone videos, 720p / 24fps with center crop, single-shot segmentation, RAFT-based motion filtering, COLMAP-style SfM via Hierarchical Localization, and a three-stage (database / geometric / kinematic) trajectory-verification pipeline with targeted SLERP-based fixups [§4.1, §4.2].
The released artifact includes a non-overlapping test set of clips; clips are 5 s at 24 fps from 30 s source videos, with poses and memory features sampled at 4 fps [§5.1].

Method

A target video clip at time-step n is generated by a frozen Wan2.2-Fun-5B-Control-Camera DiT conditioned on (a) a Qwen2.5-VL-7B caption and (b) a set of pose-aligned world tokens emitted by a learned retriever. The retriever operates over a local memory window M_n built from a global bank of (pose-token, 3D-aware memory tokens) pairs extracted by a pretrained StreamVGGT geometry encoder. The window is constrained to start at most W_max seconds before the clip entrance and to end at or after it, so adjacent clips share memory entries and stay coupled to a shared world.

Inside the retriever, pose and memory tokens at each time-step in the window are stacked into a joint sequence, encoded by transformer blocks ψ_local with 3D-aware positional encoding, and concatenated into an encoded local memory M̃. For target time-step k, the query pose token e_p^k is concatenated with R learnable query tokens, processed by a sibling encoder ψ_query, and cross-attended against M̃; the rows of the output corresponding to the learnable queries form the retrieved world tokens W_k. These tokens are mapped into the DiT hidden space by an MLP and injected via a dedicated memory cross-attention block placed after self-attention at every DiT layer (initialized from the context cross-attention). The clip-level world tokens are reused as keys/values across layers, so the prior is shared along the whole denoising stack rather than re-attended per layer.

Training is two-stage: first warm up the retriever with a reconstruction loss in the original memory-token space at the query pose, then jointly fine-tune the retriever and the DiT (DiT via LoRA) on the OpenSafari clip set.

Results

OpenSafari benchmark (Table 1) — Captain Safari beats the strongest external baseline (Wan2.2-5B-Control-Camera) on every axis: FVD 1023.46 vs 1387.75, reconstruction rate 0.968 vs 0.767, AUC@30 0.200 vs 0.181, pose-CosSim 0.563 vs 0.420. MEt3R is essentially tied with Real-CamI2V (0.3690 vs 0.3703).

Ablation (Table 1) — Removing the memory keeps FVD slightly better (998.47) but degrades reconstruction (0.912), AUC@30 (0.193), and CosSim (0.508). The memory is the explicit lever for 3D consistency and trajectory adherence; without it, the model converges to a stronger Wan-finetune that still drifts.

Human study (Table 2, 50 participants × 10 cases × 3 criteria = 1500 votes) — Captain Safari wins 67.4% / 65.6% / 69.0% on VQ / 3D consistency / trajectory following respectively; the w/o-Mem variant wins ~23% across axes; Real-CamI2V, Wan2.2-5B-Control-Camera, and Geometry Forcing each pick up single-digit shares.

Why it’s interesting

This is the first filed paper in the camera-controlled-video-diffusion cluster that solves the trajectory-following problem with an explicit, retrievable, pose-indexed world memory rather than (a) per-block GeoAdapter conditioning on rendered RGB+depth control maps (VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control), (b) a feed-forward 4DGS reconstructor (NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos), or (c) source-aware camera embedders (SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time). The mechanism is closer to retrieval-augmented LM inference than to ControlNet-style conditioning, and the memory operates over StreamVGGT features rather than rendered images — neatly bridging camera-controlled-video-diffusion and world-foundation-models (StreamVGGT is itself a perception-side WFM in the V-JEPA / VGGT lineage). Captain Safari also directly addresses the open question on the WFM concept page about how closed-flagship generative-rollout WFMs compare against open Wan-based recipes: it does not go closed, and it does not drop 3D — it instead stuffs the 3D signal inside an attention-retrieval module rather than a rendered control channel.