Reka Inverse Dynamics Model for Interactive World Models

ListenFlow Over Pixels: How Reka Teaches Real Video to Speak WASD10:31

Show notes

Reka's Inverse Dynamics Model (RIDM) watches short video clips and predicts videogame-style motor commands — W/A/S/D, Shift, plus camera yaw and pitch — to pseudo-label real-world footage for training interactive world models. The load-bearing trick: project every frame into optical-flow space via a frozen RAFT-small *before* any trainable layer sees it, which lets a model trained entirely on gameplay transfer to real video it has never seen (86% vs 31% for a pixel baseline). We dig into the mechanism, the data resampling, the six benchmark configs, the failure modes — and the big open question the post leaves unanswered. **Sources** - Reka Inverse Dynamics Model for Interactive World Models (Reka AI Team, blog)

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

Reka ships RIDM, a compact inverse-dynamics model that watches short video clips and emits videogame-style low-level motor commands (W/A/S/D/Shift, yaw, pitch). The intended downstream use is pseudo-labeling real-world video so an interactive world model can be trained on action-labeled data without a human teleoperator loop. The core trick is to project every input into optical-flow space (via a frozen RAFT-small) before any trainable layer sees it, so the IDM — trained entirely on recorded gameplay with ground-truth key/mouse logs — transfers to real-world egocentric footage it has never seen. On real-world video the flow-based model gets 86% vs 31% for a pixel-based IMPALA baseline [Table 1].

Key claims

RIDM takes short video clips and predicts low-level motor commands (W/A/S/D/Shift keys plus yaw and pitch camera deltas) that describe the physical change between consecutive frames [§ The model].
The target downstream use is a three-stage pipeline: train the IDM on gameplay with engine-provided ground-truth actions, pseudo-label unlabeled real-world video with the IDM, then train a world-language-action model on the (video, pseudo-action) pairs [Fig. 1].
Projecting inputs into optical-flow space with a frozen RAFT-small (0.99M params) before any trainable layer is the mechanism that lets gameplay-trained weights transfer to real-world video — “the flow of a corridor and the flow of a street look alike, and a pixel of each does not” [§ The model, Fig. 4].
The flow-based model is 2.8M params total (1.8M trainable + 0.99M frozen RAFT); the pixel-based IMPALA baseline is 9.8M params fully trainable [§ The model].
Training data is 1280×720 @ 48 fps gameplay with 11 key commands + mouse + exact per-frame camera angles from the game engine, resampled into five motion pools (Forward 104k, Turn 104k, Sideways/back 53k, Tilt 1.5k, Sideways+turn 50k) to over-represent rare motions like strafes and tilts that would otherwise be swamped by forward motion [§ The data, Fig. 2].
On real-world video (78 clips, forward/turn-left/turn-right classification) the flow-based model gets 85.9% vs 30.8% for the pixel-based model; on the 47-clip turn-direction subset, 91.5% vs 51.1% [Table 1].
On GoPro footage (72 clips) the gap narrows: 75.0% vs 58.3% on the 3-way task; the pixel model actually wins turn-direction on 25 clips by one clip (76.0% vs 72.0%) [Table 1].
Known failure modes include false turns on static or dark frames, inability to disentangle sidestep-plus-opposite-yaw, false walk-key predictions on static cameras that are only zooming, and action-camera stabilization hiding part of the real rotation [§ Evaluation].
Weights and inference code are released under Apache 2.0 for Reka’s own components; the flow model additionally requires off-the-shelf RAFT-small weights from torchvision, which carry their own training-data license terms [§ Extras].

Method

The pipeline is Reka’s answer to the perennial IDM problem: a world model that follows user commands needs training frames paired with action labels, but captured real-world video has no action track. RIDM proposes to synthesize that action track by annotating real video with a model trained on games where actions are logged for free. Two architectural variants are compared. The pixel model is a direct port of VPT’s IMPALA trunk: resize each frame to 128×128, run three IMPALA residual blocks (each a stride-2 pool plus two GroupNorm-ReLU-conv3×3 residual blocks), pool to 4×4, flatten to a per-frame 2048-vector, apply bidirectional 8-head attention over a 150-frame window with stride 75, and head off W/A/S/D/Shift and yaw/pitch. The flow model inserts a hard information bottleneck: a frozen RAFT-small optical-flow network converts each pair of consecutive 128×128 frames into a 2-channel flow field, a 3-conv stride-16 stem reduces each flow map to 8×8×192, 257 tokens (256 patch + 1 CLS, with position / frame-index / frame-gap embeddings) go through a 4-layer, 4-head pre-norm transformer with GELU, and the CLS token predicts the actions over a 5-frame clip. All five training pools are forced-sampled rather than naturally sampled, because the “Tilt” pool is 70× rarer than “Forward” in the raw distribution.

Results

Flow model wins 5/6 benchmark configurations across three evaluation datasets (real-world video with camera-angle labels, GoPro footage, synthetic gameplay) [Table 1].
Largest gap is on real-world video — 86% vs 31% on the 3-way forward/left/right task and 91.5% vs 51.1% on the 2-way left-vs-right turn task, confirming the optical-flow projection is doing the sim-to-real transfer work [Table 1].
The one configuration where the pixel model wins is 25-clip GoPro turn direction (76.0% vs 72.0%, a one-clip margin) [Table 1].
No end-to-end result on the downstream interactive world model is reported — the post stops at the IDM-pseudo-label stage of Fig. 1.
No quantitative comparison against VPT (which uses the same IMPALA pixel trunk but trained on Minecraft) or against the DVA-style inverse-dynamics translator trained directly on robot data.

Why it’s interesting

RIDM carves out a distinctive position in the growing “inverse-dynamics model as the bridge between video and action” cluster. Direct Video-Action Models — Causal Video Models Are Data-Efficient Robot Policy Learners (DVA) also puts an IDM at the center of its stack, but trains the IDM on tens of hours of real robot data per embodiment and treats inverse dynamics as video→end-effector translation; RIDM instead trains on gameplay with free engine labels and targets video→user-command labels for world-model training. The optical-flow bottleneck is the load-bearing design choice — it is a direct bet that discarding photometric detail before any learned layer is the right way to cross the sim-to-real gap, counter to the usual recipe of adding photorealism to the training data. This complements the broader World Foundation Models push to make video world models interactively controllable (see Atlas: A World Model for Spatial Intelligence, Generative View Stitching) by solving the specific “where do action labels come from on real video” subproblem that is usually hand-waved. It also connects to the Synthetic Training Data axis in a non-obvious way: the synthetic actions here are the pseudo-labels, not synthetic pixels — gameplay provides free ground truth that bootstraps labeling of entirely real video.

The weakest part of the post is the eval ceiling. No downstream world model is reported, so we can’t yet tell whether 86% real-world action accuracy is high enough to produce a controllable WAM, or whether the ~14% error (dominated by sidestep-vs-opposite-yaw confusion and static-camera false positives) injects enough label noise to degrade the generator. The GoPro result also hints that camera stabilization — ubiquitous in modern consumer egocentric footage — eats into the optical-flow signal in ways the gameplay training distribution never saw.