Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory

TL;DR

Matrix-Game 3.0 is Skywork’s open-source (MIT) memory-augmented interactive world model: a Diffusion Transformer that generates 720p @ 40 fps real-time action-conditioned video with minute-long spatiotemporal consistency, released at 5B (and scaling to a 2×14B MoE). It unifies three stages — an industrial Video–Pose–Action–Prompt data engine combining Unreal Engine synthetic scenes, automated AAA-game collection and real-world video augmentation; a memory-augmented DiT trained with prediction-residual frame re-injection and camera-aware memory for long-horizon consistency; and an inference stack of few-step sampling + INT8 quantization + VAE-decoder distillation reaching the headline 40 fps. The model card credits Self-Forcing, GameFactory’s action-control module, LightX2V quantization, Wan2.2 as base model, and LingBot-World’s context-parallel framework.

Key claims

The release targets 720p real-time long-form video generation with a 5B model, framed as a “memory-augmented interactive world model” with action conditioning [Overview].
The data engine combines three sources at industrial scale — Unreal Engine synthetic scenes, automated AAA game collection, and real-world video augmentation — emitting Video–Pose–Action–Prompt quadruplets [Framework Overview, Feature 1].
Long-horizon consistency is achieved via two mechanisms named on the card: prediction residuals and frame re-injection for self-correction, plus camera-aware memory for long-term spatiotemporal consistency [Feature 2].
Real-time inference at 40 fps / 720p comes from a multi-segment autoregressive distillation strategy based on Distribution Matching Distillation (DMD), combined with model quantization (INT8) and VAE-decoder distillation, claimed stable over minute-long sequences [Feature 3].
Scaling further to a 2×14B (28B) MoE is reported to improve generation quality, dynamics and generalization, though no benchmark numbers are disclosed on the card [Feature 3 / 28B-MoE].
The release is open-weight under MIT, distributed on HuggingFace with a Diffusers integration, and inference is reproducible via a public GitHub repo (SkyworkAI/Matrix-Game-3.0) [Installation, Inference].

Method

The model card is product-framed: the architecture is described as a “memory-augmented DiT” with three explicit components but no parameter counts beyond the headline 5B / 28B-MoE numbers. The inference recipe is reproducible from the provided command — torchrun ... generate.py --size 704*1280 --fa_version 3 --use_int8 --num_iterations 12 --num_inference_steps 3 --compile_vae --lightvae_pruning_rate 0.5 --vae_type mg_lightvae — which exposes the design choices: FlashAttention 3, INT8 quantization, three sampling steps per chunk under a “light VAE” (50% pruning of the VAE decoder), with num_iterations controlling how many 40-frame extensions get appended to the initial 57-frame clip. With multiple GPUs an asynchronous VAE pipeline is supported (--use_async_vae). A --use_base_model flag falls back to a 50-step non-distilled mode, and --interactive switches from pre-specified random actions to user-input actions.

The acknowledgements section is the most informative architectural footnote: Matrix-Game 3.0 is built on Wan2.2 as the base model, uses Self-Forcing as a streaming-AR recipe, takes its action-control module from GameFactory, its quantization framework from LightX2V, and its context-parallel framework from lingbot-world — placing it inside the same Wan-base, Self-Forcing-style, DMD-distilled lineage that Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation and Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generation sit in. No technical report is yet released; the citation block points to a forthcoming PDF at the SkyworkAI/Matrix-Game GitHub.

Results

The card reports no benchmark numbers (no VBench, no PhysicsIQ, no user-study). Headline operating points are:

720p (704×1280) @ 40 fps real-time inference with a 5B model [Feature 3].
Minute-long sequence stability claimed [Feature 3]; concretely, the example command’s 57 + (num_iterations − 1) × 40 formula at num_iterations=12 gives 497 frames ≈ 12.4 s at 40 fps per run, with the system designed to extend further.
28B (2×14B) MoE variant claimed to further improve quality, dynamics, generalization [Feature 3 / 28B-MoE].
3 inference steps per chunk under INT8 + LightVAE [Inference command].
The model card reports 348 downloads in the last month as of fetch [HF Downloads].

No comparison numbers vs. Self-Forcing, Self-Forcing++, LongLive, Rolling Forcing, FramePack-F1, Wan-baseline, or other interactive WFMs are provided.

Why it’s interesting

Matrix-Game 3.0 is the first filed open-weight (MIT) entry in the cluster of real-time interactive world models — sitting opposite Project Genie: Experimenting with infinite, interactive worlds (Project Genie / Genie 3, closed Google Labs prototype) on the open-vs-closed axis, and complementing Advancing Open-source World Models (LingBot-World) (LingBot-World) and FlowAct-R1: Towards Interactive Humanoid Video Generation as another Wan-based streaming AR system that productizes the Self-Forcing distillation lineage. The 28B-MoE scaling claim is a datapoint on whether MoE buys quality at the WFM-rollout regime, where neither Genie 3 nor the wiki’s open lineage has previously been MoE. The recipe ingredients — Wan2.2 base, Self-Forcing forcing scheme, GameFactory action-control, LightX2V INT8, lingbot-world context parallelism — read like a survey of the cluster the wiki has been filing all week, assembled into one stack with a clean inference budget (40 fps, 3 NFEs, INT8).