WildWorld: A Large-Scale Dataset for Dynamic World Modeling with Actions and Explicit State toward Generative ARPG

TL;DR

WildWorld is a 108M-frame dataset for action-conditioned video world modeling, automatically collected from the AAA action RPG Monster Hunter: Wilds (Alaya Studio / Shanda AI Research Tokyo). Each frame carries 119 annotation columns — frame-wise action IDs from a 450+-action space, character skeletons, world states (positions, velocities, health, animation IDs), camera intrinsics + extrinsics, and depth maps — captured by instrumenting the game engine + rendering pipeline (OBS Studio + a custom Reshade shader splitting the display into RGB / depth sub-windows) under an automated NPC-driven gameplay system. The companion benchmark WildBench evaluates interactive world models on Action Following (Gemini 3 Flash judges generated vs. ground-truth segment alignment) and State Alignment (TAPNext-tracked skeletal keypoints vs. ground-truth screen-space trajectories), in addition to Video Quality and Camera Control. The paper positions explicit, semantically meaningful states as the primitive that current action-conditioned video datasets (SpatialVID, Sekai, OmniWorld, GF-Minecraft, PLAICraft, GameGen-X) miss, with the motivating example that actions like “shoot” affect hidden state (ammo count) that pixels alone can’t expose. Note: as flagged in the originating Slack thread, the dataset does not yet appear to be publicly released.

Key claims

Existing video world-modeling datasets entangle actions with pixel-level changes — they typically annotate basic movements/camera rotations whose effects are visually observable, leaving models unable to learn implicit state transitions (e.g. “shoot” depleting an ammo counter that gates whether a later “shoot” produces a projectile) [§1].
WildWorld contains 108M frames with 119 per-frame annotation columns, including 455 motion IDs across 24 character banks (5,960 unique (weapon, bank, motion) triplets), 527 monster motion IDs across 13 banks, character + monster skeletons, world state (position, rotation, velocity, animation IDs, health/stamina/mana-like resources), camera intrinsics + extrinsics, and depth [§3.4, §3.1].
The data acquisition platform separates actions (engine inputs), states (engine-maintained world state), and observations (renderer outputs) by hooking the corresponding game-engine stages, and disables HUD-stage shaders to yield clean HUD-free RGB frames [§3.1].
Automated gameplay is achieved by programmatically navigating in-game menus to sample random quest-NPC combinations and by relying on the game’s native rule-based companion AI plus target-lock camera; the authors argue trajectory diversity survives despite the scripted logic because monster behaviour is stochastic and multi-agent interaction is high-dimensional [§3.2].
Multi-source synchronization uses a custom Reshade shader that partitions the 2K display into four 720p sub-windows; a modified OBS Studio records RGB (lossy HEVC, target 16 Mbps / max 20 Mbps) and depth (lossless HEVC) as separate streams, with embedded timestamps for cross-source alignment [§3.2].
Filtering rejects samples by: duration <81 frames, adjacent-frame timestamp gap >1.5× target interval (~50 ms @30 FPS), >15 consecutive YUV-luma-extreme frames, sustained camera-character distance below threshold (third-person spring-arm contraction = occlusion), and first-frame inter-character skeletal-keypoint overlap >30% of either character’s projected area [§3.3].
Hierarchical captions are generated by Qwen3-VL-235B-A22B-Instruct (served via vLLM) over 1-FPS, 480p frames of single-action segments, with action + state ground-truth injected as prompt context to compensate for game-specific unfamiliarity; sample-level captions summarize segment captions via Gemini 3 Flash [§3.3].
WildBench evaluates along four axes: Video Quality (VBench MS / DD / AQ / IQ), Camera Control (ATE + RPE on ViPE-estimated camera trajectories vs. ground truth, with scalar-aligned translations), Action Following (Gemini 3 Flash judges segment-level consistency on three action groups — movement, fast displacement, attack — with category-specific prompts), and State Alignment (TAPNext tracks keypoints initialized from the GT first frame in generated I2V output, scored as average coordinate accuracy at 4/8/16/32 px thresholds against GT screen-projected skeletons) [§4.1].
The released benchmark curates 200 samples balanced across scenarios (100 cooperative player+NPC vs. monster; 100 one-on-one), difficulty levels, character/monster types, and event types including skill usage, knockdowns, deaths, and critical hits [§4.2].
Three baseline interactive world models are trained on WildWorld: CamCtrl (Wan2.2-Fun-5B-Control-Camera fine-tuned with GT per-frame camera poses instead of rule-converted Plücker embeddings from discrete inputs), SkelCtrl (Wan2.2-Fun-5B-Control fine-tuned with projected 3D-skeleton control videos), and StateCtrl (CamCtrl extended with a hierarchical state encoder — trainable embeddings for discrete states, MLP for continuous, with entity-level + global-level pooling) [§5.1].
Gameplay spans 5 stages × 4 weapon types × 4 player characters × 29 monster species under day/night × sunny/rain weather, with ~66% active-combat clips vs. ~34% mounted traversal; the top-150 character action IDs cover only 58.49% of samples, indicating a long-tailed action distribution that the paper notes but does not yet rebalance [§3.4].

Method

The contribution is a data-acquisition + annotation pipeline rather than a model. Data acquisition hooks three engine stages of Monster Hunter: Wilds: engine input (actions), engine world-state updates (positions, rotations, velocities, animation IDs, resource bars, skeletal poses for player + monsters), and the rendering pipeline (RGB + depth + camera intrinsics/extrinsics, with HUD-stage shaders disabled). All structured streams are serialized per engine tick as timestamped JSON; image streams co-record through a Reshade shader that tiles the 2K display into four 720p sub-windows fed to a modified OBS Studio for per-window encoding (RGB lossy HEVC at 16/20 Mbps; depth lossless HEVC). Automated gameplay navigates menus via the engine’s UI API and lets the game’s native behavior-tree AI fight monsters across randomly sampled quest+character+monster combinations, with the target-lock camera keeping the action framed.

Filtering applies the five rules in Key Claims (duration, temporal-continuity, luminance, camera-occlusion, character-occlusion). Hierarchical captioning segments samples on frame-wise action ID, captions each segment’s mid-frame (1 FPS, 480p) with Qwen3-VL-235B-A22B-Instruct + GT action/state in context, and summarizes per-sample with Gemini 3 Flash. The benchmark wires these into four scorers — VBench for VQ, ViPE + ATE/RPE for camera, Gemini 3 Flash + category-specific prompts for Action Following, and TAPNext-tracked GT-skeleton screen projections for State Alignment.

Results

The paper’s “results” are primarily diagnostic — it reports human-preference correlation for the proposed Action Following and State Alignment metrics, and trains the three baselines (CamCtrl, SkelCtrl, StateCtrl) on WildWorld to show that current interactive video models struggle with semantically rich action sequences and with maintaining long-horizon state consistency. The 30KB truncation point of the fetched body cuts off the headline tables, so numeric leaderboard claims are not extracted here; the qualitative conclusion the paper draws is that even when video and camera quality look fine, the action and state axes remain open — particularly for compositional action sequences and for events whose visual effect depends on hidden state (low-ammo, low-health, depleted-stamina). The dataset is sized to be the largest open action-conditioned world-modeling corpus (108M frames vs. the 147M-segment Action100M, but with structured per-frame state instead of free-text captions), and is presented as a primitive for both training and evaluation rather than as a target benchmark with a current SOTA leaderboard.

Why it’s interesting

WildWorld is the opposite end of the design space from Action100M: A Large-scale Video Action Dataset: both are large-scale open(-intended) video datasets with automated annotation pipelines, but Action100M annotates internet HowTo videos with free-text captions extracted by a Tree-of-Captions + LLM-aggregation pipeline, whereas WildWorld instruments a single AAA game to capture ground-truth structured state — skeletons, world coordinates, animation IDs, depth, camera poses — that no internet-scraping pipeline can recover. For the wiki, this pins down a fork in the World Foundation Models open-question “Does training interactive WFMs require paired action–observation data, which embodied-AI lacks?”: WildWorld is one concrete answer — synthesize the paired data inside a game engine where the state is observable by definition. The cost is a narrower domain (one ARPG, 29 monster species) and a question of action-space transfer to non-game environments that The flavor of the bitter lesson for computer vision flags as the hard part.

The paper is also the cleanest World Foundation Models counter-example to a pure end-to-end framing — MultiGen: Level-Design for Editable Multiplayer Worlds in Diffusion Game Engines reintroduces explicit external memory (minimap + pose set) for editability and multiplayer; WildWorld reintroduces explicit state annotations during pre-training to fix the action–pixel entanglement. Both argue that the Bitter-Lesson dissolution of intermediates is conditional on having a learning signal that can substitute for them, and both choose to keep the intermediate when that signal is missing. WildBench’s State Alignment metric — TAPNext-tracked skeletal keypoints vs. GT screen-space trajectories — is also a more direct probe of “did the world evolve the way it should” than RISE-Video’s LMM-judge over per-sample reasoning questions, and the two benchmarks should be read as complementary axes (action-conditioned state-tracking vs. text-conditioned implicit-rule reasoning) of the same WFM-evaluation problem.

The Slack note (“looks like they have not released the dataset yet”) is the operative caveat: until WildWorld lands on HF, it is a recipe paper — the value to a Luma reader is the engine-instrumentation + Reshade/OBS multi-stream recording approach and the WildBench metric definitions, not access to the corpus itself.