AlayaWorld: Interactive Long-Horizon World Modeling

TL;DR

AlayaWorld is a full-stack open-source interactive video world model from Alaya Lab (Shanda AI) built on a ~13B video DiT (LTX-2.3 base with the audio module removed), generating 540p/720p video at 24 fps autoregressively in latent chunks of K=4 frames under continuous camera-trajectory control and switchable per-chunk text prompts. Its bounded visual context is a fixed prefix of four clean conditioning streams — a pinned “sink” identity frame, a compressed temporal history (Frame-Preservation-style H_ϕ over the last 6 latent frames), a geometry-aligned spatial memory (GEN3C-style depth-projected cache of past views into the current camera), and the single most-recent frame as I2V continuity — attended jointly with the noised target chunk under a single non-causal self-attention DiT. Training runs in three stages: (1) full-parameter bidirectional adaptation from LTX-2.3 on a mixed real+synthetic corpus (222K clips across Sekai/SpatialVid/RealEstate10K/DL3DV/MUGEN + GameVerse/GenEvent), (2) autoregressive fine-tuning with a LoRA-warmed history encoder, AdaLN camera control, spatial memory, and next-forcing auxiliary head, stabilised by Helios drift simulation + an error bank that replays the model’s own residuals, and (3) discrete AR distillation combining DMD, Self-Forcing++, and consistency distillation to reduce inference from ~30 to 4 steps per chunk. Reported best-in-class on iWorld-Bench; code, weights, and technical report are all released.

Key claims

The four capabilities interactive world models must jointly satisfy — interaction, spatiotemporal consistency, long-horizon stability, and efficiency — cannot be addressed independently: broader interaction makes consistency harder, longer rollouts amplify residual errors, and aggressive acceleration compromises visual stability [§1].
AlayaWorld generates video autoregressively in latent chunks of K=4 latent frames under (a) an absolute per-frame camera trajectory π_i and (b) an optional chunk-level text prompt y_i that can be switched at chunk boundaries to drive prompt-driven actions such as combat or spell-casting [§3.1].
The per-chunk prefix S_i = [sink ; temporal-memory ; spatial-memory ; nearby-frame ; target] is processed by full non-causal self-attention, then sliced off after the last transformer block so only the target segment is denoised [§3.1, Eq. 2].
The sink is a single clean latent frame pinned at RoPE temporal position 0 and held fixed across all chunks as a global identity/appearance anchor; at training it is drawn as a remote frame (≥8 latent frames from the target) so the model cannot extrapolate the next chunk from it and must instead rely on the camera-control signal [§3.1].
Temporal memory h_i = H_ϕ(w_i) compresses a sliding window w_i of the last L=6 latent frames via a Frame-Preservation-style encoder recomputed every chunk; its tokens bypass the patch embedder and are injected directly [§3.1].
Spatial memory follows GEN3C: an explicit cache B = {(I_j, D_j, π_j)} stores each past generated frame with its Depth-Anything-3 monocular depth and camera pose; up to 10 frames are greedily selected by maximum-coverage under a z-buffer with occlusion tolerance δ=0.1, forward-splatted into the target view, VAE-encoded to g_i, and injected with a self-attention key bias so uncovered regions are ignored [§3.1].
Per-chunk compute is constant because the context is a bounded rolling window (fixed sink + 6-frame history + capped 10-frame cache), so the horizon N is in principle unbounded [§3.1].
Camera control is applied per-frame as a Fourier-embedded per-axis relative-pose increment concatenated over 6 pose components and mapped through an MLP into an AdaLN modulation of the timestep embedding: c_cam = MLP(⊕_k PE(∆π_k)), e ← e + c_cam [§3.3, Eq. 5].
Stage 1 (bidirectional pretraining) is a full-parameter fine-tune of the LTX-2.3-derived ~13B video DiT on a mixed 24-fps 540p/720p ≤20s corpus under a mixture of image/video/text-conditioned objectives, with an adaptive sigma-shift schedule scaling the flow-matching timestep shift with clip length and a short low-σ refinement pass at the end [§3.2].
Stage 2a (history pretraining) keeps the backbone frozen and trains only a LoRA-adapted history-compression module H_ϕ under a flow-matching objective, reconstructing a target window z_Ω conditioned on a masked-history embedding with per-frame noise σ_i ∼ U(0.2, 1) [§3.3, Eq. 4].
Stage 2b (full-stack fine-tune) unfreezes the backbone and jointly trains it with the history-compression module, the AdaLN camera-control module, and a next-forcing auxiliary head that supervises features hooked from several backbone layers on the next chunk at a shifted noise level τ̃ = 10τ/(1+9τ); total loss L = L_flow + 0.5·L_nf [§3.3, Eq. 6].
Anti-drift training uses two exclusive mechanisms on the temporal/spatial/nearby context tokens (sink kept clean): Helios drift simulation applies additive noise + downsample-blur + saturation-shift artefacts, and an error bank buffers the model’s own residuals δ = ẑ_0 − z_0 bucketed by chunk length and noise level and replays them additively into context and target; the error bank warms up first with Helios only, then takes priority [§3.3].
Stage 3 (post-training) distils the ~30-step teacher into a 4-step student by combining Distribution-Matching Distillation (DMD), Self-Forcing++, and consistency distillation, preserving the complete control and memory stack [§1, §3.4].
Training data mixes real captures (Sekai-Real 21,561; SpatialVid 23,210; RealEstate10K 17,429; DL3DV 7,905; MUGEN 21,436) with synthetic renderings (GameVerse 124,116; GenEvent 6,490), totalling 222,147 clips [Table 1].
Poses for real-world sources without camera metadata are recovered per-frame with ViPE [§2.1].
The unified filtering pipeline enforces six gate classes (technical validation ≥720p / ≥3s / 24–65 fps / H.264-HEVC-AV1; photometric with border-ratio ≤0.10; shot-boundary via OmniShotCut; motion analysis with bucketed camera-motion labels; text/UI suppression via EasyOCR + pixel-stability mask ≤0.04 overlay; person control via YOLO11) plus an optional pose-stability gate for sources with estimated trajectories [§2.2].
Captions are annotated at two levels — a small clip-level attribute vocabulary (weather, time-of-day, location type, camera perspective, camera motion, video style — 26 values total, reduced from 59) and segment-level tracks separating subject motion / environmental dynamics / static scene / camera description, fused into a 5–9-sentence full_prompt plus a 15–45-word short_prompt for dropout/augmentation, generated by a VLM (default Kimi-K2.6) over 1–2 fps timestamped frames [Table 2, §2.3].
AlayaWorld achieves the best overall performance on iWorld-Bench across generation quality, trajectory following, and memory ability [§4] (numerical breakdowns cut off by the 30KB fetch truncation).

Method

AlayaWorld’s backbone is LTX-2.3 with the audio module removed, leaving a ~13B video diffusion transformer that generates in the causal video VAE’s latent space. Generation is autoregressive over K=4-frame latent chunks under two conditioning modalities that enter by different routes: the camera trajectory becomes a compact per-frame AdaLN modulation from Fourier-embedded relative-pose increments, and the visual past enters as a four-stream in-context prefix — a pinned sink frame, a compressed temporal-memory embedding H_ϕ(w_i) over the last 6 latent frames, a geometry-aligned spatial-memory render g_i built from a GEN3C-style depth cache, and the single most-recent latent frame as I2V continuity — followed by the noised target. The whole prefix is processed by full non-causal self-attention and sliced off after the last transformer block; only the target segment is denoised.

The spatial memory is where the geometry lives. Every generated frame is decoded to pixels, monocular depth estimated by Depth-Anything-3, and (I_j, D_j, π_j) appended to a cache B. To render the cache into chunk i, up to 10 frames are chosen by greedy maximum-coverage selection under a z-buffer with occlusion tolerance δ=0.1; each is unprojected to a 3D point cloud via its depth and camera, reprojected into the target camera π_i by forward splatting (Eq. 3), and fused across sources by nearest-depth. The result is VAE-encoded to g_i and placed at the target’s RoPE coordinates; the coverage mask M_i becomes a self-attention key bias so never-observed regions are ignored rather than trusted. Because the sink + 6-frame history + 10-frame cache prefix is bounded, per-chunk compute is constant and the horizon is in principle unbounded.

Training runs in three stages (Fig. 3). Stage 1 full-parameter-adapts LTX-2.3 to the world-model domain under a bidirectional flow-matching objective. Stage 2a warms up only H_ϕ via LoRA under a masked-history reconstruction objective (Eq. 4). Stage 2b unfreezes the backbone and jointly trains it with H_ϕ, an AdaLN camera-control head (Eq. 5), and a next-forcing auxiliary head that predicts the next chunk at a shifted noise level τ̃ = 10τ/(1+9τ), supervised at loss weight 0.5 (Eq. 6). Two anti-drift mechanisms operate on the temporal/spatial/nearby streams (never the sink): Helios drift simulation injects additive-noise / down-and-up-sample-blur / saturation-shift artefacts, and an error bank buffers per-chunk-length + per-noise-level residuals δ = ẑ_0 − z_0 and replays them additively into context and target. Stage 3 distils the ~30-step teacher into a 4-step student via DMD + Self-Forcing++ + consistency distillation, keeping the full control + memory stack.

Results

Reported headline: AlayaWorld achieves the best overall performance on iWorld-Bench across generation quality, trajectory following, and memory ability, with qualitative demonstrations of controllable navigation, consistent revisitation of previously seen places, prompt-driven actions (combat, spell casting), and stable long-horizon rollouts across diverse scenes [§4]. The 30KB PDF fetch truncated before the numeric leaderboard, ablation, and per-axis breakdown tables, so specific metric numbers are not extractable from the fetched body at filing time. All weights (merged_infer.safetensors on HuggingFace AlayaLab/AlayaWorld), inference code, and the full technical report are released; training code and partial training data are on the roadmap but not yet released. Model runs on a single CUDA GPU with PyTorch ≥ 2.6 and can be scaled to multi-GPU via Ulysses context parallel.

Why it’s interesting

AlayaWorld is the most complete filed instantiation of the composite-context recipe for interactive video world models: it stacks nearly every design choice the wiki has been tracking in isolation into one system. The four-stream prefix (sink + Frame-Preservation temporal memory + GEN3C spatial memory + I2V nearby frame) unifies the Pretraining Frame Preservation in Autoregressive Video Memory Compression memory-compression axis with the Latent Spatial Memory for Video World Models and Video World Models with Long-term Spatial Memory external-cache axis and the Context Forcing: Consistent Autoregressive Video Generation with Long Context slow-fast/attention-sink axis, and adds a spatial-memory design that pushes the RGB point cloud back into the DiT’s own latent space via a VAE encode (unlike SPMem’s RGB rendering + separate encoder). Compared to RELIC: Interactive Video World Models with Long-Horizon Memory — which annotates a compressed KV memory with absolute camera poses so retrieval is implicitly 3D-aware — AlayaWorld keeps an explicit 3D reprojection cache while still tying it to the model’s own coordinate system through the AdaLN camera signal, giving a directly-testable answer to the open “implicit-camera-annotation vs explicit-3D-cache” question the two systems bracket.

The Alaya Lab / Shanda team is the same group behind WildWorld: A Large-Scale Dataset for Dynamic World Modeling with Actions and Explicit State toward Generative ARPG, so this is the natural next step after their data-side contribution: the same authors instrumented Monster Hunter: Wilds to produce paired action–state–observation data at 108M frames, and now use game-engine synthetic data (GameVerse 124K clips) plus a smaller GenEvent set alongside real captures to train a full-stack interactive model. AlayaWorld’s Stage-2b anti-drift recipe — Helios drift simulation + error-bank residual replay — is a concrete stacking of two of the four injection-site variants the Autoregressive Video Generation concept page identified as complementary but never combined (Context Forcing’s Error-Recycling Fine-Tuning, MALT’s noise-augmented memory, LongVie 2’s degradation-aware input training, FlowAct-R1’s generated-GT memory swap). The Stage-3 distillation combining DMD + Self-Forcing++ + consistency distillation is likewise the first filed system to combine all three, addressing Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generation‘s stated DMD-vs-CD split (CD for initialization, DMD for polish) with an explicit multi-objective recipe.

Compared to closed flagships Project Genie: Experimenting with infinite, interactive worlds (Genie 3) and Runway GWM-1 (Runway GWM-1 launch broadcast — world model, avatars, audio, multi-shot editing), AlayaWorld is an open-weights + open-code + open-tech-report alternative in the same real-time interactive category, and next to open peers Advancing Open-source World Models (LingBot-World) (LingBot-World) and HY-World 1.5 (WorldPlay): A Systematic Framework for Interactive World Modeling with Real-Time Latency and Geometric Consistency (HY-World 1.5 / WorldPlay), it is the most explicit about combining an explicit geometry cache with a compressed temporal memory in one DiT prefix. The main missing datapoint is the numeric iWorld-Bench table — kept as a re-resolve target once the full PDF renders.