Solaris: Building a Multiplayer Video World Model in Minecraft

TL;DR

Solaris is a multiplayer video world model trained on coordinated Minecraft gameplay — it jointly predicts first-person observation streams for two players from their past observations and actions. The release has three pieces: SolarisEngine, a Mineflayer-based pipeline for collecting paired multiplayer gameplay (9,240 episodes / 12.64M frames); a multiplayer DiT built on MatrixGame 2.0 with per-player 3D RoPE and player-ID embeddings, trained via flow matching + diffusion forcing with per-player noise levels; and Solaris Eval, a VLM-judged benchmark for five multiplayer capabilities (movement, grounding, memory, building, consistency). The key training contribution is Checkpointed Self Forcing — a memory-efficient reformulation of Self Forcing that splits the autoregressive rollout into a gradient-disabled cache pass and a parallelized recompute pass, making it tractable to backpropagate through a long-context bidirectional teacher and through KV-cache representations that the original Self Forcing stops gradients on.

Key claims

The world-modeling thesis behind Solaris is that world state is global while rendering is local: with more than one agent, the only scaling-relevant quantity is the shared representation beneath individual views [project page intro / tweet OP].
SolarisEngine builds on Mineflayer (composable primitives — pathfinding, block placement, combat) plus a custom bot-to-bot coordination layer and higher-level building/scaffolding/tool-use/navigation primitives; controller bots are paired with headless Minecraft Java client camera bots synchronized via a server plugin to mirror position/orientation/animations at 20 FPS, with actions and observations aligned in post-processing by timestamp [§ Solaris Engine].
The dataset is 9,240 episodes / 12.64M combined frames across building (houses/walls/towers/bridges), combat (PvP+PvE), movement (chasing/navigation/exploration), and mining, split evenly between Superflat and Normal world types with varied biomes/weather/time-of-day; episode lengths 128–512 frames (6.4–25.6 s @ 20 fps); claimed to be the first action-annotated multiplayer Minecraft dataset suitable for world-model training [§ Dataset, Fig. 3].
The model is a multiplayer DiT built on MatrixGame 2.0 with three changes: 3D RoPE applied independently per player, learned player-ID embeddings to distinguish streams, and cross-attention / FFN / action-conditioning modules applied independently per player; training uses flow matching combined with diffusion forcing where independent noise levels are sampled per player and per timestep, so each stream denoises while staying cross-player consistent [§ Model, Fig. 4].
The architecture nominally generalizes to N players despite training with two [§ Model].
Checkpointed Self Forcing removes the prohibitive memory cost of combining long-context bidirectional teachers with Self Forcing’s autoregressive rollout: phase 1 runs the AR rollout forward with gradients disabled and caches clean estimates + noisy inputs per step; phase 2 recomputes the generator’s outputs in a single parallelized forward pass using a custom attention mask that reproduces the sliding-window causal dependencies [§ Checkpointed Self Forcing, Fig. 6].
The memory savings further enable backpropagating through the KV-cache representations — gradients that the original Self Forcing implementation blocks via stop-gradient — and enabling those gradients further improves generation quality [§ Checkpointed Self Forcing].
The generator uses a rolling KV cache with a sliding window of 6 latent frames while the teacher operates over the full sequence; this asymmetry is what makes the original (uncheckpointed) Self Forcing rollout infeasible at long teacher context [§ Checkpointed Self Forcing].
Solaris Eval probes five multiplayer capabilities — Movement, Grounding (turn-away-and-turn-back across agents), Memory (return to original orientations), Consistency (co-visible regions render identically across players), and Building (observer sees the structure built by the other agent) — using a VLM-as-judge over 7 held-out ground-truth episodes [§ Solaris Eval].
On Solaris Eval, the full Solaris model beats a Frame-concat baseline and a Solaris-without-pretrain ablation on Grounding, Memory, Building and Consistency (VLM ↑ and FID ↓), at the cost of slightly lower Movement-VLM than Frame-concat (68.23 vs 77.08) — but Frame-concat exhibits action hallucinations under no-op actions [Table 1].
The Self Forcing ablation finds that (a) straightforward causal fine-tuning replaces the original CausVid ODE-regression init, (b) few-step ability can be learned jointly with stable autoregressive generation rather than needing a pre-distilled few-step model, and (c) enabling KV backpropagation gives the best FID across all categories while trading a small amount of action-following in some categories [Table 2].
Capabilities demonstrated qualitatively include inventory tracking, simulating weather, placing torches, generating animations, and simulating PvP [§ Solaris Model Capabilities].

Method

SolarisEngine packages controller bots, camera bots, and the Minecraft server as Docker containers orchestrated by Docker Compose, with Python scripts launching parallel workers; bots sample and execute episodes from a library, teleport to a random location at the start of each episode, and a safety mechanism detects failures and resets state to allow continuous unattended collection. All actions are annotated as semantic game events in VPT-compatible format covering movement, camera, digging, placing, and attacking.

The model jointly denoises both players’ observation streams. Architecturally it inherits MatrixGame 2.0’s DiT but applies 3D RoPE per player and adds learned player-ID embeddings; cross-attention for first-frame conditioning, feed-forward layers, and action conditioning are applied independently per player. Training uses flow matching with diffusion forcing where the noise levels for each player and each timestep are sampled independently, so the model learns to denoise each player’s stream conditional on (possibly noisy) views of the other.

The key training-time contribution, Checkpointed Self Forcing, addresses a concrete problem: combining Self Forcing’s autoregressive student rollout with a long-context bidirectional teacher makes the backward graph grow with sequence length, blowing past GPU memory. The two-phase recomputation reformulation collapses the sequential rollout into a single parallel forward pass for the backward, slashing peak memory and additionally unlocking gradients through the KV cache that Self Forcing previously had to stop.

Results

On Solaris Eval (Table 1, VLM-judge ↑ / FID ↓):

Solaris: Movement 68.23 / 38.48, Grounding 62.50 / 38.03, Memory 37.50 / 55.13, Building 20.83 / 83.58, Consistency 71.35 / 99.40.
Solaris w/o pretrain: 69.27 / 42.53, 29.17 / 49.85, 18.75 / 67.80, 0.00 / 86.60, 49.48 / 121.39.
Frame-concat baseline: 77.08 / 68.87, 53.13 / 66.57, 37.50 / 74.44, 0.00 / 103.24, 53.11 / 129.38.

Solaris dominates on FID across all categories and on VLM-score for Grounding / Building / Consistency; Frame-concat wins on Movement VLM-score but FID is much worse and the project page notes Frame-concat hallucinates actions under no-ops. Solaris-without-pretrain is the clear worst across most categories, suggesting the MatrixGame 2.0 pretraining base is doing meaningful work.

Self Forcing ablations (Table 2) compare initialization strategy {ODE-regression, causal fine-tuning} × {with/without pre-DMD} × {with/without KV backprop}. Best FID across categories is causal FT + no pre-DMD + KV-BP, the Solaris default; it also gives the strongest Building VLM (20.83) and Consistency VLM (71.35). Causal FT + Pre-DMD + KV-BP gives slightly higher Movement / Memory VLM but worse FID.

Why it’s interesting

Solaris adds a multi-agent axis to the wiki’s growing autoregressive-video-generation cluster: it complements Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation and Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generation, which fix the initialization of the Self Forcing pipeline for single-stream AR generation, by attacking a different bottleneck of the same pipeline — memory cost when the teacher is long-context — and the two lines should compose (Checkpointed Self Forcing’s two-phase recompute is orthogonal to whether the init is causal CD or causal ODE distillation). It also extends the design choices catalogued in Context Forcing: Consistent Autoregressive Video Generation with Long Context (Context Forcing’s long-context teacher recipe) — Solaris is in some sense Context Forcing’s long-context teacher made affordable in the joint-distribution regime. From the world-models angle, Solaris is the first filed paper to operationalize the shared-state / local-rendering framing as a concrete training objective rather than an evaluation criterion, and it sharpens the World Foundation Models cluster’s “WFM = predictive latents under coordinated agent behavior” thesis. It contrasts with the closed-flagship Project Genie: Experimenting with infinite, interactive worlds / The Waymo World Model: A New Frontier For Autonomous Driving Simulation product framings by being open, single-domain (Minecraft), and explicitly multi-agent. The VLM-as-judge eval is also another datapoint for VLM-as-Evaluator applied to physics/world-rule judgment rather than aesthetics.