Advancing Open-source World Models (LingBot-World)

TL;DR

LingBot-World is an open-source action-conditioned world model from Robbyant (Ant Group’s embodied-AI unit) that turns Wan2.2 into an interactive world simulator. The system predicts the conditional distribution of future video tokens given past frames, language prompts, camera trajectories, and discrete keyboard actions, and autoregressively rolls out streams up to ~10 minutes with emergent long-term memory. The release ships a 28B-total / 14B-active MoE DiT base model (Cam and Act variants) plus LingBot-World-Fast, a distilled block-causal student that runs at 16 FPS / 480p on a single 8-GPU node with sub-1-second latency. Code, weights, and a project page are public under Apache-2.0; positioned as one of the first fully open systems claiming the Genie-3-class envelope (general domain coverage, multi-minute horizon, high dynamic degree, 720p, real-time).

Key claims

Most text-to-video models produce passive clips and do not model how actions change the environment over time; LingBot-World instead learns transition dynamics so keyboard/mouse and camera inputs drive the evolution of future frames [§1].
The data engine combines three sources — web videos (first- and third-person), game RGB strictly paired with W/A/S/D + camera parameters, and Unreal Engine synthetic trajectories with known intrinsics/extrinsics and object layouts [§Data Engine].
Hierarchical captioning produces three caption levels (whole-trajectory narrative including camera motion, scene-static layout without motion, dense temporal captions for short windows) to disentangle static structure from motion patterns [§Data Engine].
The backbone is a 2-expert MoE DiT initialized from Wan2.2 — 14B parameters per expert, 28B total, only 1 expert active per denoising step, so inference cost matches a dense 14B model while capacity is expanded [§Architecture].
A training curriculum extends sequences from 5 → 60 seconds and skews the noise schedule toward high-noise timesteps, which stabilizes global layouts over long contexts and reduces mode collapse for long rollouts [§Architecture].
Actions are injected into transformer blocks: camera rotations encoded as Plücker embeddings, keyboard actions as multi-hot vectors over keys (W/A/S/D), fused and routed through adaptive layer-norm modulation [§Architecture].
Only action-adapter layers are fine-tuned; the visual backbone remains frozen, preserving Wan2.2’s visual prior while learning action responsiveness from a smaller interactive dataset [§Architecture].
LingBot-World-Fast initializes from the high-noise expert, replaces full temporal attention with block-causal attention (bidirectional inside a block, causal across blocks), enabling KV-cache-based streaming autoregressive generation [§Fast].
Distillation combines diffusion forcing on a small set of target timesteps (including t=0 so the student sees both noisy and clean latents) with Distribution Matching Distillation and an adversarial discriminator head — adversarial loss updates only the discriminator, student is updated by the distillation loss [§Fast].
LingBot-World-Fast reaches 16 FPS at 480p on a single GPU node with sub-1-second end-to-end interaction latency [§Fast].
Emergent long-term memory is reported without explicit 3D representations (no Gaussian splatting): landmarks like Stonehenge reappear with consistent geometry after ~60s off-camera; a car leaving and re-entering the frame appears at a physically plausible location [§Long Horizon].
On VBench over 100 curated >30s videos, LingBot-World reports the highest imaging quality, aesthetic quality, and dynamic degree vs Yume-1.5 and HY-World-1.5 — dynamic degree 0.8857 vs 0.7612 / 0.7217 [§VBench].
Camera-pose control signals are computed from input video via the ViPE pose estimator; control inputs are intrinsics.npy [num_frames, 4] (fx, fy, cx, cy) and poses.npy [num_frames, 4, 4] in OpenCV coordinates [README].
The release ships three model variants under Apache-2.0 — Base (Cam) at 480P/720P, Base (Act) for keyboard-style action strings (e.g. w-10,a-10,d-10,iw-15,none-10,j-10,l-10,s-15), and Fast — plus a community-contributed 4-bit (NF4) quantized variant for low-VRAM inference [README].

Method

LingBot-World is built on Wan2.2 (a 14B image-to-video DiT) and extended into a 2-expert MoE DiT with 28B total parameters, 14B active per denoising step. Three classes of conditioning are fed into the transformer: (i) past frame latents, (ii) text prompts via the original Wan T5 path, and (iii) interactive signals — Plücker-encoded camera rotation + multi-hot keyboard vectors — fused and applied to the DiT hidden states via adaptive LayerNorm modulation. Only the action-adapter layers train; the visual backbone is frozen. Training proceeds along a curriculum that grows sequences from 5s to 60s and skews the diffusion noise schedule toward high-noise timesteps to keep long-rollout global layout stable.

The Fast variant is a distilled student initialized from the MoE high-noise expert. It replaces full temporal attention with block-causal attention (bidirectional within a temporal block, causal across blocks) so that a KV cache can be reused across blocks and frames stream out autoregressively. Distillation uses diffusion forcing on a small target-timestep set including t=0 (the student sees both noisy and clean latents in training), combined with Distribution Matching Distillation and an adversarial discriminator head trained only via the adversarial loss while the student updates only via the distillation objective.

The training corpus is built by a three-source data engine — diverse web videos, game logs with strict action↔frame pairing, and Unreal Engine renders with known camera parameters — followed by a profiling/curation pass (resolution/duration filters, clip segmentation, missing-pose estimation, VLM quality scoring) and hierarchical captioning at narrative / scene-static / dense-temporal granularity. The same backbone is trained on both image-to-video and video-to-video continuation tasks, yielding a transition function that can start from arbitrary time points.

Results

Long-horizon rollout: training sequences reach 60s; inference autoregressively extends coherent video streams up to ~10 minutes with stable layout [§Long Horizon, §VBench setup].
VBench (100 videos, each >30s) vs Yume-1.5 / HY-World-1.5: highest imaging quality, aesthetic quality, and dynamic degree (0.8857 vs 0.7612 vs 0.7217). Motion smoothness and temporal flicker comparable to the best baseline; best overall consistency [§VBench].
Real-time interactivity: LingBot-World-Fast achieves 16 FPS at 480p on a single GPU node with sub-1-second end-to-end interaction latency [§Fast].
Inference scripts: i2v-A14B Base (Cam) at 480×832 / 720×1280, default 161 frames; can be scaled to 961 frames (≈1-minute video at 16 FPS) given enough CUDA memory; FSDP-wrapped DiT + T5 with Ulysses sequence parallel size 8 across 8 GPUs [README].
Release surface: arXiv tech report, GitHub code, HF + ModelScope weights (3 model variants), online demo (Reactor), Apache-2.0 license, community 4-bit NF4 inference checkpoint already exists [README].
Comparison axes vs Matrix-Game-2.0 / Mirage-2 / Genie-3 (qualitative): the blog summary positions LingBot-World as one of the few fully open systems combining general domain coverage, long generation horizon, high dynamic degree, 720p, and real-time capability [§Comparison].

Why it’s interesting

For Luma this is the closest open-source datapoint yet to a Genie-3-class interactive world model — and the recipe is the now-familiar “frozen Wan + adapters” pattern that already shows up across the wiki (Camera-Controlled Video Diffusion cluster), but pushed to (a) MoE scale (28B-total / 14B-active), (b) multi-minute autoregressive rollout with emergent memory rather than explicit 3D/KV-cache structure, and (c) two qualitatively different control modalities (Plücker camera + W/A/S/D action vectors) sharing a single AdaLN-modulated adapter. It is a useful sibling reference for both the Autoregressive Video Generation cluster (PFP, Context Forcing, FlowAct-R1, LongVie 2 — all of which engineer explicit cache/memory structures) and the World Foundation Models cluster (NVIDIA Cosmos as a closed-source bundle vs LingBot-World as a single open release). The Fast variant is also a clean third datapoint on chunkwise-AR streaming distillation alongside FlowAct-R1 (Seedance, 3-NFE) and Context Forcing (Wan2.1-T2V-1.3B teacher–student), making the 3-paper threshold for a future streaming-ar-distillation concept page worth watching.