SIMA 2: A Generalist Embodied Agent for Virtual Worlds

TL;DR

SIMA 2 is DeepMind’s generalist embodied VLA agent built on a Gemini Flash-Lite backbone, trained to play a portfolio of 3D commercial games (Goat Simulator 3, Hydroneer, No Man’s Sky, Satisfactory, Space Engineers, Valheim, Wobbly Life) plus research environments by reading 720p RGB frames and emitting keyboard-and-mouse actions as structured text. Unlike SIMA 1’s short-instruction-following, SIMA 2 reasons about high-level goals, converses with the user, generalizes to held-out games (ASKA, Minecraft via MineDojo, The Gunk) and to entirely synthetic photorealistic worlds generated on-the-fly by Genie 3, and demonstrates open-ended self-improvement: Gemini sets tasks, scores trajectories, and the agent autonomously learns new skills from scratch in previously unseen environments. The training mixture is SFT on human gameplay spans + Gemini-synthesized “bridge” reasoning+dialogue data + online RL from verifiable rewards.

Key claims

A Gemini Flash-Lite base finetuned on a mixture of gameplay trajectories + Gemini-pretraining data + Gemini-synthesized reasoning/dialogue (“bridge”) data substantially closes the gap to human performance across the training game portfolio while retaining the base model’s vision, dialogue, and reasoning capabilities [§1, §3.3].
Action output is structured text parsed deterministically into 96 keyboard keys + mouse clicks + discretized relative mouse motion — not a custom action-token head — which lets a single SFT objective cover keystrokes, dialogue, and internal reasoning in one decode [§3.2].
The agent operates from the same modalities as a human player: 720p RGB frames and keyboard+mouse only, with no privileged environment state, and explicitly contrasts this with prior work that reads underlying game state [§3.2].
Bridge data — a small, high-quality set of trajectories where Gemini Pro retroactively annotates causally-consistent internal reasoning and dialogue interleaved with frames and actions — is the mechanism that fuses Gemini’s text capabilities with embodied action [§3.3.2].
Online RL from verifiable rewards is layered on top of SFT, using a curated set of (initial state, text instruction, verifier) tuples sourced from human suggestions plus verifier-mined goal-completion points in human trajectories, with dialogue tasks generated from screenshot + human-suggested QA pairs [§3.3.3].
SIMA 2 generalizes to entirely held-out commercial games (ASKA Viking survival, The Gunk action-adventure) and a 50-task subset of the MineDojo Minecraft benchmark covering Combat, Harvest, and Tech Tree categories with 15 random seeds each [§3.1.1].
SIMA 2 generalizes to photorealistic worlds it has never seen by running inside Genie 3, which generates the environment frame-by-frame in response to the agent’s keyboard+mouse actions — the agent’s training distribution is video-game footage but the test distribution is generative-rollout pixels [§3.1.1, §1].
Open-ended self-improvement loop uses three foundation models — a task-setter (Gemini), the agent (SIMA 2), and a reward model (Gemini) — plus a general world model (Genie 3) as the environment, enabling autonomous skill acquisition in unseen environments [§2 Open-Ended Self-Improvement, §1].
Human data was collected via two complementary protocols: free-form single-player annotated post-hoc, and a two-player “Setter-Solver” protocol in which language instructions causally precede actions, producing tighter language↔action alignment than retrospective annotation [§3.3.1].
Trajectories are reorganized into “spans” — a single task instruction paired with the corresponding sub-sequence of frames and actions — which is the granularity used during SFT [§3.3.1].

Method

SIMA 2 is a Gemini Flash-Lite that emits structured text combining low-level keyboard/mouse commands with optional internal reasoning and user-facing dialogue. The history fed to each forward pass is a stream of past 720p RGB frames plus the natural-language inputs and the model’s own prior reasoning and replies; the output is a chunk of actions that the harness parses and dispatches into the running game (Figure 3). There is no policy head, no learned action embedding, and no privileged game state — the interface is deliberately the same modality envelope a human player has.

Training is three-stage. Stage 1 is supervised finetuning over a mixture of (a) “human data” — gameplay spans with text instructions and frame+action sequences, collected via the free-form-annotation and Setter-Solver protocols on the licensed games — and (b) “bridge data” — a smaller high-quality set where Gemini Pro retroactively annotates the trajectories with internal reasoning and dialogue that are causally consistent with the visible frames and the executed actions. The bridge corpus deliberately covers error-correction, instruction chaining, visual QA, memory reliance, long-horizon behavior, and explicit no-ops so the agent learns to stop after task completion. Mixing in non-gameplay Gemini pretraining data is reported as crucial to preserve base capabilities. Stage 2 is online RL from verifiable rewards (RLVR): for each curated (initial state, instruction, verifier) task the agent rolls out, gets a binary reward for embodied success or correct grounded-QA answer, plus a few shaped rewards for instruction-following. Stage 3 is the open-ended self-improvement loop, where a Gemini task-setter proposes new goals in an unseen environment, the agent attempts them, and a separate Gemini reward model scores trajectories — closing the loop without human supervision and (in the photorealistic case) without a fixed game environment, since Genie 3 generates the world itself.

Results

The paper claims SIMA 2 “substantially closes the gap with human performance” across the training game portfolio and “demonstrates robust generalization to previously unseen environments” including ASKA, MineDojo Minecraft, The Gunk, and Genie 3 photorealistic worlds [Abstract, §1]. It further claims open-ended self-improvement enables autonomous acquisition of new skills from scratch in new environments via the task-setter / agent / reward-model triple [Abstract, §1]. Quantitative per-environment success rates, the comparison curves against SIMA 1 and against Gemini Flash-Lite prompted as a VLA, and the MineDojo-subset numbers are in sections of the paper beyond the 30 KB fetch window — not reproduced here at filing time.

Why it’s interesting

SIMA 2 is the strongest filed datapoint that the closed-flagship WFM stack (Genie 3: A new frontier for world models) is being deliberately co-developed with an embodied agent that uses it both as a test environment and as a source of unbounded curriculum, instantiating the AI-GA / generative-curriculum thesis the Genie-3 post only gestured at. The agent-acting-inside-Genie-3 demo in the Genie 3 announcement is now backed by a full agent paper, which closes the loop on Sitzmann’s “video generative pre-training as the embodied-AI substrate” position (The flavor of the bitter lesson for computer vision) and contrasts with the rollout-as-action-source minimal pipeline of Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations — RIGVid skips the policy entirely and reads actions out of generated pixels, SIMA 2 trains a policy that acts inside generated pixels. The self-improvement triple (task-setter + agent + reward-model, all Gemini-derived) is also the clearest filed extension of the reasoning-RL recipe from Reasoning RL — particularly the InfTool/Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing closed-data-loop pattern — into the embodied-agent regime, with the foundation model playing all three roles. Finally, the structured-text-as-actions design is a counterpoint to the discrete-action-token VLAs (NVIDIA Unveils New Open Models, Data and Tools to Advance AI Across Every Industry‘s Cosmos+GR00T) — DeepMind’s bet is that emitting actions as text preserves Gemini’s reasoning/dialogue surface in the same decode.