Lumine: An Open Recipe for Building Generalist Agents in 3D Open Worlds

TL;DR

Lumine is a generalist VLA agent built on Qwen2-VL-7B-Base that plays Genshin Impact — and zero-shot transfers to Honkai: Star Rail and Wuthering Waves — by reading 720p raw pixels at 5 Hz and emitting textual keyboard-and-mouse actions at 30 Hz via action chunking, with a hybrid thinking mode that emits internal reasoning only when needed. The training recipe is a three-stage curriculum: 1731 h of human gameplay for visuomotor pre-training, 200 h of language-grounded instruction following, and 15 h of reasoning data for adaptive monologue. After end-to-end inference optimization (25.3× latency reduction), Lumine completes Mondstadt’s full five-hour main storyline at expert-human efficiency despite training reasoning data on Act I only, and the authors claim it is the first agent able to complete hours-long missions in real time within a commercial AAA open-world game.

Key claims

Lumine identifies six core challenges for 3D-open-world generalist agents — scalable environments, multimodal perception, high-level planning, low-level control, multi-timescale memory, and real-time inference — and is positioned as an integrated recipe addressing all of them, in contrast to prior single-axis work [§1].
The agent uses a human-equivalent I/O envelope (720p screen pixels in, 96-key keyboard + mouse motion + clicks out) and explicitly contrasts with prior game agents that read API-exposed game state [§1, Table 1].
Action representation is structured text emitted autoregressively with action chunking (Zhao et al. 2023), where mouse movements and key presses carry semantic meaning naturally interpretable by VLMs — fine-grained key events (down/up/hold) and relative mouse trajectories are preserved, unlike GUI-agent absolute-click teleportation [§2 Interface].
Three-stage training curriculum: (i) 1731 h human gameplay for action-primitive mastery, (ii) 200 h instruction-following data to ground control in language, (iii) 15 h reasoning data to enable hybrid thinking [§1].
Hybrid thinking lets the agent adaptively enter a reasoning mode and emit inner-monologue text before action chunks only when context demands it — combining ReAct-style every-step reasoning with hierarchical fast/slow split, claimed to avoid both hallucination and latency overhead [§2 Reasoning, §1].
Memory is “context as memory”: up to 20 recent observation/action steps kept as short-term memory, with reasoning steps retained as long-term memory — no specialized memory module, just an extended context window [§1].
Three-band emergent capability progression observed during pre-training: object interaction emerges first, then basic combat + GUI manipulation, then game-specific mechanisms and navigation, all from imitation alone with no explicit supervision [§1].
After instruction tuning, short-horizon control (10 s – several minutes) exceeds 80% success and generalizes to unseen objectives/scenarios [§1].
After reasoning fine-tuning, Lumine matches expert-human efficiency on the full five-hour Mondstadt main storyline despite reasoning training data covering only Act I — Acts II and III are reasoning-OOD but pre-training-in-distribution [§1].
Zero-shot generalization is reported in two directions: (a) cross-region within Genshin (Liyue, including reaching the Adeptus hidden in the mountains) and (b) cross-game without any fine-tuning to Wuthering Waves (100-minute mission) and Honkai: Star Rail (full ~5 h first chapter) [§1, Abstract].
End-to-end inference optimization yields a 25.3× overall latency reduction, sufficient to sustain the 30 Hz action emission rate needed for real-time play [§1].

Method

Lumine is a single Qwen2-VL-7B-Base finetuned to take a rolling window of past 720p RGB frames plus text history (including the agent’s own prior reasoning when present) and emit chunks of typed keyboard-and-mouse actions at 30 Hz, derived from observations sampled at 5 Hz. Actions are written as text tokens parsed deterministically by the harness — 96 keyboard keys with down/up/hold semantics plus relative mouse movement plus clicks — avoiding any custom action-token head and keeping reasoning and dialogue surface tokens contiguous with action tokens in the same decode.

The training recipe layers three SFT stages with strictly different objectives. Stage 1 (1731 h human gameplay) trains action-primitive mastery; Stage 2 (200 h instruction-following) grounds control in language by pairing short instructions with corresponding frame+action spans; Stage 3 (15 h reasoning data) teaches the agent when to switch into a “thinking” mode and emit internal monologue before producing the action chunk. The hybrid thinking flag is learned from data — the agent decides per-step whether to think or act directly, with the model preserving the last ~20 observation steps in context as short-term memory and keeping reasoning summaries as long-term memory. Finally an inference-optimization pass (action chunking + standard LLM-serving optimizations) brings end-to-end per-decision latency down by 25.3× to support real-time interaction.

Results

The headline qualitative result is that Lumine completes the entire five-hour Mondstadt main storyline of Genshin Impact at expert-human efficiency despite reasoning training data covering only Act I, with Acts II–III being reasoning-OOD [§1, Abstract]. Cross-region generalization within Genshin: zero-shot completion of the first one-hour mission in Liyue and successful navigation to the Adeptus hidden deep in the mountains, despite Liyue being held out from training [§1]. Cross-game generalization without any additional training: completion of a 100-minute mission in Wuthering Waves and the full ~5 h first chapter of Honkai: Star Rail (Herta Space Station) [Abstract, §1]. Short-horizon instruction-following success rate >80% on tasks of 10 s – several minutes after Stage 2, generalizing to unseen scenarios [§1]. Table 1 positions Lumine as 5 h task horizon vs <1 h for Cradle and 1 min for CombatVLA, with Open-World ✓, Reasoning ✓ (“Adaptive”), Real-Time ✓, and a K&M interface — the only filed row that combines all four. Quantitative per-task success rates and ablation curves are described in the paper but lie beyond the fetched 30 KB window — not reproduced here at filing time.

Why it’s interesting

Lumine is the closest open-recipe counterpart to SIMA 2: A Generalist Embodied Agent for Virtual Worlds — both are 7B-class VLA agents that play 3D commercial games via 720p screen pixels and structured-text keyboard+mouse actions, both refuse the API-route taken by AlphaStar/OpenAI Five, and both bet that emitting actions as text preserves the base VLM’s reasoning surface. The differences sharpen the design space: SIMA 2 trains across 7 licensed games + Genie 3 generative worlds with Gemini Flash-Lite + online RLVR + Gemini-synthesized “bridge” reasoning data and a self-improvement triple; Lumine trains on a single AAA game (Genshin Impact) with Qwen2-VL-7B + pure SFT in three stages + a hybrid-thinking flag, and only reports zero-shot transfer rather than RL on new games. Lumine also makes the task-horizon record on file — five-hour main-storyline completion versus SIMA 2’s reported short-to-medium-horizon spans — at the cost of a much narrower training distribution. The hybrid-thinking design (adaptive reasoning toggle) is structurally a third point between SIMA 2’s bridge-reasoning-everywhere SFT mixture and the every-step ReAct loop catalogued in Tool-Use Agents: reasoning is emitted only when context demands, addressing exactly the latency/hallucination critique that ReAct receives in the Tool-Use Agents §Reasoning discussion. Versus OSGym: Scalable OS Infra for Computer Use Agents, which scales the environment side of computer-use agents to a Kubernetes farm with verifiable rewards, Lumine takes the opposite bet — a single curated AAA environment with deep imitation rather than wide RL — and reports that human gameplay alone produces the emergent skill curriculum (interaction → combat/GUI → navigation/mechanisms). Finally, the cross-game zero-shot result (Genshin → Wuthering Waves → Honkai: Star Rail) is the most concrete filed data point that visuomotor skills learned from a single commercial-game distribution can transfer across distinct world dynamics, which is the central open question Sitzmann names in his bitter-lesson essay (The flavor of the bitter lesson for computer vision) — paired-action-observation data is the bottleneck, and gameplay is one of the few domains where it exists at scale.