Yan: Foundational Interactive Video Generation

TL;DR

Yan is a Tencent framework for end-to-end interactive video generation built on Stable Diffusion, packaging three modules trained on a shared ~400M-frame dataset from the 3D game Yuan Meng Star: Yan-Sim (1080P / 60FPS real-time game simulator), Yan-Gen (multi-modal action-controllable generator with hierarchical captioning), and Yan-Edit (text-driven structure + style editing via a depth-bridged hybrid model). The headline pitches are (a) 1080P/60FPS interactive simulation through a 32×32×2 VAE plus KV-cache shift-window denoising at 4 DDIM steps with FP8 quantization, and (b) explicit disentanglement of interactive mechanics from visual rendering, enabling on-the-fly editing during interaction.

Key claims

Yan-Sim simulates a modern 3D game at 1080P / 60FPS with 0.11s latency, beating Genie 3 (720P / 24FPS), Matrix-Game (720P / 16FPS), and PlayGen (128P / 20FPS) on the resolution+FPS frontier [Table 2].
The 3D-VAE pushes spatial downsampling from 8× to 32× and adds 2× temporal compression by concatenating consecutive frames along the channel dim, raising the compression ratio from 4×8×8 to 2×32×32 with latent channels expanded to 16 [§5.1.1].
The diffusion model uses causal 1D temporal attention plus per-frame action-cross-attention with a 768-d action token from an MLP, trained via Diffusion Forcing with staged per-frame noise schedules so frame N can only attend to frames <N [§5.1.1, §5.1.2].
Shift-window denoising inference processes a window of frames at varying noise levels per step, concatenating a pure-noise latent to the input and using KV cache for historical states; this reduces average per-frame latency vs. full per-frame denoising [§5.1.3, Fig. 4].
FP8 weight+activation quantization across all GEMM operations plus structural UNet pruning gives ~2× inference speedup over FP16 with minimal quality loss; the entire pipeline is captured as a single CUDA graph and uses torch.compile max-autotune [§5.1.3].
The training dataset is over 400M frames at 1080P / 30FPS from a modern 3D game (Yuan Meng Star), across 90+ scenarios with an 8-dim action space — the largest interactive-video dataset compared (next is The Matrix at 792M frames at 720P / 60FPS) [Table 1].
Data collection uses a hybrid random + PPO-RL exploration agent for breadth+depth, with three filters (visual color-variance, anomaly frame-count, rule consistency) and a balanced sampler over engine-recorded attributes (xyz, alive, collision) [§4.1].
Yan-Gen uses a multimodal DiT (umt5-xxl text encoder, ViT-H-14 visual encoder) with task-specific cross-attention layers and a hierarchical autoregressive caption method that injects game-specific knowledge into an open-domain video diffusion backbone — claimed to prevent long-term semantic drift via stable global context + detailed local descriptions [§5.2].
Yan-Gen demonstrates cross-domain style+mechanics blending: when textual and visual prompts come from different domains, the model composes them flexibly under user prompts (open-domain generalization claim, qualitative only) [Abstract, §5.2].
Yan-Edit disentangles interactive mechanics from visual rendering by using a depth map as the intermediate state: a Yan-Sim-based mechanics simulator (with text cross-attention layers for structure editing) feeds a depth map into a Yan-Gen-based visual renderer (via ControlNet) for style editing [§5.3, Project page].

Method

Yan combines three modules trained on the same in-house dataset. Yan-Sim is a Stable-Diffusion-based causal video diffusion model with three core architectural modifications: a much higher-compression VAE (4×8×8 → 2×32×32, latent channels expanded to 16, lightweight pruned decoder with pixel-shuffle for symmetric upsampling); a causal-attention diffusion UNet with per-frame action-MLP injection via cross-attention; and an inference pipeline that combines DDIM-4 sampling, KV-cached shift-window denoising (each step processes a window of frames at varying noise levels, with a noise latent appended per step), FP8 quantization, structural pruning, and CUDA-graph capture of the full pipeline. Training uses Diffusion Forcing with a staged per-frame noise schedule and a clean first-frame condition. Yan-Gen extends an open-domain MM-DiT (umt5-xxl + ViT-H-14) with hierarchical autoregressive captions, frame-wise action conditioning, and a multi-stage training pipeline to address long-term semantic drift in AR rollouts. Yan-Edit is a hybrid model that explicitly routes interaction through a depth-map intermediate: Yan-Sim handles structure editing (text-conditioned simulation of mechanics, depth-map output), and Yan-Gen handles style editing (ControlNet conditioned on the depth map, text-driven appearance). The shared training dataset is built by a random + PPO-RL exploration agent inside a modern 3D game, with timestamp-aligned screenshots+actions, three quality filters, and balanced sampling.

Results

Yan-Sim leads the interactive-simulator comparison on the resolution+FPS frontier: 1080P / 60FPS / 0.11s latency / infinite-length vs. Genie 3 (720P / 24FPS / few minutes), Matrix-Game (720P / 16FPS), GameFactory (640P), and PlayGen (128P / 20FPS / 0.05s) [Table 2]. The training dataset compares favorably to prior interactive-video datasets on the (resolution × FPS × scale × frame-wise control) frontier: 1080P / 30FPS / 8-action / 400M frames, vs. The Matrix 792M frames at 720P / 60FPS / 5-action, GameGenX 192M frames at 720P–4K / 1–24FPS, Matrix-Game 50M frames at 720P / 16FPS, and GameFactory 4M frames at 360P / 16FPS [Table 1]. Qualitative results are presented for visual quality (8 levels of the same game rendered at high resolution), motion consistency (correct frame-level action responses), accurate mechanism simulation (water slides preserve inertia, electric fences arrest motion, trampolines amplify jumps), long video generation (consistency across multi-minute rollouts), and cross-domain style+mechanics blending for Yan-Gen. No quantitative comparisons against prior systems on a shared benchmark are reported in the excerpt fetched.

Why it’s interesting

Yan is the first filed paper that ships a complete interactive-video stack — simulation + generation + editing — under one framework with a single shared dataset, contrasting with the wiki’s current world-foundation-models cluster where each entry handles one slice: Genie 3: A new frontier for world models and Project Genie: Experimenting with infinite, interactive worlds cover the closed-flagship generative-rollout side at 720P / 24FPS, Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory covers the open game-simulator side at 720P / 16FPS, and Hunyuan-GameCraft: High-dynamic Interactive Game Video Generation with Hybrid History Condition covers game-video generation with hybrid history conditioning. Yan’s 1080P/60FPS claim pushes the resolution+FPS frontier and the 32×32×2 VAE is the most aggressive spatial compression filed for this regime — comparable to DC-AE 1.5: Accelerating Diffusion Model Convergence with Structured Latent Space‘s structured-latent-space approach but routed through aggressive decoder pruning + FP8 + CUDA graphs rather than latent-space restructuring. Yan-Sim’s diffusion-forcing + causal-attention + KV-cache combination is structurally adjacent to Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation and Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generation but explicitly trades the distillation-step argument for raw quantization + pipeline engineering — a different point in the streaming-AR design space. The Yan-Edit depth-map disentanglement is a notable architectural choice that doesn’t appear elsewhere in the filed interactive-video stack and may matter for the broader “editable world model” thesis.