WorldModelGym: a decision-based fidelity benchmark for world models

TL;DR

WorldModelGym (WMGym) is Reka Labs’ benchmark that scores world models on decision-based fidelity rather than pixel realism or physical plausibility: given a short observation history and a menu of five candidate action sequences (four real + one random baseline), the world model simulates each and predicts cumulative reward; an environment-neutral argmax rule then picks the “best” sequence, and the model is scored by normalized regret against the real-environment return of its pick. A companion reward-prediction error metric flags models that pick well by luck rather than by accurate reward modeling. WMGym launches with 100+ tracks across Atari, Meta-World, DeepMind Control, and classic control, uses a Gymnasium-compatible step interface that treats world-model state as opaque (JEPA-style latents, tokens, or pixels all fit), and requires only branching from a frozen state — not full search-based rollouts — keeping compute-heavy models tractable.

Key claims

Decision-based fidelity is proposed as a targeted metric complementary to (not a replacement for) visual/physical plausibility benchmarks and rollout-accuracy benchmarks — the same authors’ PhysicalRealismBench occupies the plausibility axis, WMGym occupies the “is the simulator good enough for an agent to succeed” axis [§Introduction].
The evaluation protocol replaces expensive full-horizon rollouts with an open-loop multiple-choice probe: for each question the model ranks five pre-computed action sequences (one of which is a random baseline anchoring the worst-case score) by simulated cumulative predicted reward, and the argmax pick is scored against ground truth from the real environment [§The protocol].
Headline metric is normalized regret (best_return − true_return_of_picked) / (best_return − worst_return), with 0 = optimal plan and 1 = no better than the worst random choice [§The protocol].
A second metric, reward-prediction error (normalized diff between predicted and actual rewards), distinguishes models that decide well by luck (error cancellation) from models whose reward predictions are actually calibrated [§The protocol].
The Gymnasium step extension treats world-model state as an opaque object — JEPA-style latent vectors, tokens, or pixel video-diffusion states all fit the same evaluator, and the model never has to render frames (only update state + estimate reward) [§The WorldModel-Gym contract].
The protocol requires branching from a frozen state to evaluate multiple plans — this is the load-bearing implementation requirement that all submissions must satisfy [§The WorldModel-Gym contract].
Environment selection is deliberately restricted to tasks with fixed history windows, reproducible resets, controlled stochasticity, and horizons long enough for prediction errors to compound — “effective RL tasks do not necessarily translate into effective world-model benchmarks” [§Statistics].
Launch scale: 100+ tracks across four environment families — Atari, Meta-World manipulation, DeepMind Control, and classic control [§Statistics].
Each track is a declarative file (environment, canonical observation, scoring range), so the suite is extensible without touching the evaluator [§Statistics].
Named limitations: WMGym does not judge frame realism, and it evaluates only open-loop decisions — long-horizon closed-loop deployment behavior is out of scope [§Limitations].

Method

WMGym proceeds in three steps per probe. First, the model is initialized with a short observation history from a real environment. Second, it is handed a fixed menu of five action sequences (one random baseline plus four alternatives designed to look plausible to the model) and asked to simulate each — updating its internal state action-by-action and predicting per-step reward. Third, an environment-neutral argmax rule selects the sequence with the highest cumulative predicted reward, and the evaluator compares this pick to the ground-truth returns pre-computed by rolling all five sequences in the real environment. The autopilot analogy in the post makes the design intent explicit: WMGym is a “sudden high-stakes multiple-choice question” for the world model, and any performance difference across models under the fixed argmax rule reflects predictive accuracy rather than any special planning trick.

The example probe shown is a robot-arm push-button task where reward has two additive terms (reach + press-depth). The best sequence scores 620, three near-miss variants land at 210–270, and the random baseline scores 33 — the reward gradient is broad enough that a mildly-off reward model can still pick the best, but reward-prediction error will reveal whether the pick was principled. Submissions integrate via a lightweight adapter implementing the extended Gymnasium contract; standardized adapter checks and a unified format make submissions comparable. The leaderboard (live as of 2026-07-28) previews the evaluation policy in the ground-truth environment and its baseline score alongside every track’s model results.

Results

No model results are reported in the blog itself — the paper is a benchmark-and-leaderboard announcement, and the leaderboard at wm-gym.labs.reka.ai is the live results surface. The July 28 update states that any world model that can encode an observation history, take an action, and predict the resulting reward can be scored on the site. No baselines or reference numbers are quoted in the announcement.

Why it’s interesting

WMGym is a clean, deliberately-narrow probe of a question the wiki has been circling for months: which axis of “WFM-ness” actually matters for downstream use? The World Foundation Models concept explicitly names this as an open question (generation quality vs latent prediction quality), and WMGym’s answer is neither — it measures decision-usefulness at the reward level under a fixed argmax rule. This is a different axis than GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation (WMBench, which asks whether a WFM ranks policies consistently with real-robot rollouts), RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation (RoboWorld, which uses a WFM + VLM judge as a full neural simulator), World Value Models for Robotic Manipulation (WVM, which trains a WFM-derived value head), and DriftWorld: Fast World Modeling through Drifting (DriftWorld, which reports 0.99 rank correlation with real policy performance on Robomimic). WMGym’s twist is the open-loop multiple-choice reduction: no MPC, no policy search, no full rollout — just ranking a handful of pre-computed candidates. This complements Reka’s own PhysicalRealismBench-U (physical understanding, the perceptual axis) and slots the wiki’s benchmark ecosystem Video Generation Benchmarks into a cleaner three-axis frame: surface realism / physical understanding / decision usefulness. The Gymnasium contract extension also matters for RL Environment Platforms — WMGym is the first filed platform that treats learned world models as first-class Gymnasium Env implementations rather than as substrates that generate data for policy training.