Markov AI gaming dataset — 500+ hours of Valorant / Minecraft / GTA gameplay screen recordings + keystrokes/mouse

TL;DR

Dev Mandal (Markov AI) announces a 500+ hour open-source gaming dataset — screen recordings paired with keystroke and mouse-movement traces across Valorant, Minecraft, GTA, and other titles — framed as “the largest open-source dataset of gaming data for computer-use”. This is the second Markov AI action-labeled corpus after the ~12,300-hour computer-use-large professional-software drop from March 2026; both are trimmed screen-recording video, but the gaming set is the first Markov release to ship paired action labels (keystrokes + mouse) rather than pretraining-only video. Amit flags it as “useful for action learning” — the underlying HF dataset URL was in the tweet’s comment thread and not directly retrievable at filing.

Key claims

The dataset contains 500+ hours of gameplay screen recordings across Valorant, Minecraft, GTA “and more” [tweet body].
Each recording is paired with keystroke and mouse-movement traces (i.e. action-labeled at the OS-input level, not gamepad-overlay level) [tweet body].
Framed as the “largest open-source dataset of gaming data for computer-use” [tweet body].
Released by Markov AI (attributed to @markov__ai in the tweet); the actual download link is in the tweet’s reply thread and could not be retrieved at filing time [tweet body].

Method

Not detailed in the tweet. The framing (“computer-use” + “keystrokes/mouse movements”) suggests direct OS-input capture during gameplay sessions — the same recording mechanism as Markov AI’s earlier computer-use trajectory dataset, which shipped per-step screenshots + actions pairs. That would make this the gamepad-independent, keyboard-and-mouse-only counterpart to NitroGen: An Open Foundation Model for Generalist Gaming Agents‘s gamepad-overlay recovery pipeline: rather than parsing controller inputs off-screen from streamer overlays, Markov appears to capture keyboard/mouse events at the OS level directly. The precise capture protocol, data license, per-title breakdown, and whether the traces are timestamp-synchronized to individual video frames all wait on the dataset card itself.

Results

Not applicable — this is a data announcement, not a paper. No models trained on it, no evaluation numbers, no comparison to prior open gaming corpora.

Why it’s interesting

This slots directly into the three-way action-labeling debate the wiki has been triangulating:

NitroGen: An Open Foundation Model for Generalist Gaming Agents (NitroGen) solves labeling by parsing controller overlays that speedrunners visually render into their videos — 40,000 hours across 1,000+ games at the price of gamepad-only action space, F1≈0.84 joystick / ≈0.96 button accuracy. The Markov gaming dataset takes the other approach: capture inputs at the OS layer during recording (no post-hoc labeling), trading dataset size (500h vs 40,000h) for label fidelity (exact keystrokes + mouse events, no segmentation error) and a keyboard-and-mouse action space that maps directly onto Valorant, GTA, and Minecraft PC play.
Computer Use Large: 48k screen recordings (~12,300 hours) of professional software (Markov’s own software corpus) is the sibling: same team, same capture stack framing, but the earlier release was unlabeled screen recordings of Blender/Photoshop/AutoCAD/etc totaling ~12,300 hours and designed for VPT-style IDM-labeled pretraining. This new gaming set is smaller (500h vs 12,300h) but flips the design: it ships the labels rather than leaving them for downstream IDM. Together the two releases stake out Markov’s position — same “sourced screen video” substrate but shipped in both flavors (labels-later for scale, labels-now for gaming).
The First Fully General Computer Action Model (FDM-1) (Standard Intelligence FDM-1) trains a foundation model over 11M hours of proprietary keyboard-mouse-labeled screen recordings; this Markov gaming set is a public sliver in the same design corner (native OS-level input labels on screen video), specifically targeting the gaming subset where FDM-1’s white-collar corpus does not reach.

Practically, for World Foundation Models work, a 500h paired video+action corpus over three distinct games (Valorant = fast tactical FPS, Minecraft = open-world sandbox, GTA = large 3D open world) is small but distributionally rich — three genres whose input-conditional dynamics are very different. It’s directly usable as a fine-tuning substrate for policies pretrained on NitroGen and as an evaluation set for any world model that claims to predict game screens from actions. The “link in the comments” pattern also means the dataset should be refetched and re-filed once the HF page is locatable — this stub records the announcement and Amit’s framing (“useful for action learning”) but the concrete dataset schema is pending.