VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos

TL;DR

VideoAgentTrek is a video-driven pretraining pipeline that turns in-the-wild screen-recorded YouTube tutorials into structured (clip, action, thought) trajectories for computer-use agents, with no manual annotation. A channel-coherence discovery step mines ~10,000 hours of candidate footage from a handful of seed queries; a YOLO-based cursor preprocessor (ScreenFilter) keeps 7,377 hours of sustained-GUI segments; and a grounded VLM (Video2Action) recovers dense GUI events with tight temporal bounds, parameterized actions (click (x,y), drag path, scroll Δ, typed text), and step-level thoughts. A two-stage recipe — 26B tokens of video-mined continued pretraining followed by 8B tokens of clean SFT plus a ~1B-token GUI-grounding subset — lifts a base computer-use model from 4.5% → 14.13% on OSWorld-Verified (15.78% with test-time scaling) and from 38.5% → 69.3% step accuracy on AgentNetBench.

Key claims

Channel coherence is the discovery primitive: starting from a few seed keywords (e.g. “Excel tutorial”, “How to use Windows”), the pipeline validates a handful of seed videos per channel and promotes the entire channel as trusted if ≥80% pass, then expands via that channel’s related videos / tags / metadata, yielding 55,000 candidate videos and ~10,000 hours with minimal human oversight [§Data].
ScreenFilter is a lightweight cursor-centric preprocessor (YOLOv8x backbone) that keeps only segments where ≥80% of frames contain a cursor for at least 6s, with a 2s merge gap for temporal smoothing — applied to the 10,000-hour pool, it preserves 7,377 hours of verified GUI interaction [§Data].
Video2Action recovers detailed agent trajectories from raw screen videos by performing dense GUI event detection with tight temporal bounds, parameterizing each action from pixels (click (x,y), drag path, scroll Δ, typed text), and generating step-level thoughts that capture intent and expected state change [§Method].
Video2Action is trained on synchronized video–event logs paired with a grounded VLM, so the trajectory recovery model is itself supervised against on-screen ground-truth events rather than against pseudo-labels from another VLM [§Method].
Stage 1 (video pretraining) consumes 26B tokens from 39k videos → 1.52M ReAct steps, interleaves frames with stepwise text, and masks the loss to text only — i.e. the model conditions on pixels but is supervised only on action/thought text [§Method].
Stage 2 consolidates the policy with 8B tokens of curated human demonstrations (chat-style SFT), with a focused ~1B-token GUI-grounding subset that sharpens pointer–target alignment [§Method].
The decoupled grounding→policy recipe lifts OSWorld-Verified from a 4.5% base → 9.3% with Stage 2 only → 14.13% with Stage 1+2 (up to 15.78% with test-time scaling) [§Results].
On AgentNetBench, step accuracy rises from 38.5% (base) → 64.1% (Stage 2) → 69.3% (Stage 1+2), confirming that the video-mined Stage 1 contributes additively to a strong SFT recipe rather than being a substitute for it [§Results].

Method

The pipeline has three stages glued by a single representation: a (clip, action(type, params), thought) ReAct-style sequence. First, a channel-centric discovery loop bootstraps from seed keywords (e.g. “Excel tutorial”, “How to use Windows”), validates a few candidates per channel, and — if pass-rate ≥80% — promotes the entire channel as a trusted source, then expands recall via related videos, tags, and metadata. This recall-oriented design surfaces ~55,000 candidate videos (~10,000 hours). Second, ScreenFilter is a YOLOv8x cursor detector run frame-by-frame: clips where ≥80% of frames contain a cursor for ≥6s (2s merge gap) are kept, preserving 7,377 hours of sustained GUI interaction from the ~10,000-hour pool. Third, Video2Action — a grounded VLM trained on synchronized video–event log pairs — runs over the surviving clips, performing (i) dense GUI event detection with tight temporal bounds, (ii) parameter recovery from pixels (click (x,y), drag path, scroll Δ, typed text), and (iii) step-level thought generation. The output is a corpus of (clip, action, thought) training-ready sequences.

The downstream agent is trained with a two-stage schedule. Stage 1 is continued pretraining on the video-mined corpus — 26B tokens spanning 39k videos and 1.52M ReAct steps, interleaved with stepwise text, with the loss masked to text only so that the model is conditioned on raw pixels but supervised on action+thought tokens. Stage 2 is a chat-style SFT on 8B tokens of curated human demonstrations plus a ~1B-token GUI-grounding subset specifically tuned for pointer–target alignment.

Results

OSWorld-Verified Pass@1: base 4.5% → Stage 2 only 9.3% → Stage 1+2 14.13% → 15.78% with test-time scaling [§Results].
AgentNetBench step accuracy: base 38.5% → Stage 2 only 64.1% → Stage 1+2 69.3% [§Results].
Corpus stats: 55,000 candidate videos (~10,000 h) discovered → 7,377 h post-ScreenFilter → 39k videos / 1.52M ReAct steps / 26B tokens used in Stage 1.

Why it’s interesting

VideoAgentTrek is the unlabeled-video counterpart to OSGym: Scalable OS Infra for Computer Use Agents‘s parallel-OS-replica recipe for computer-use agents: both target the same OSWorld benchmark, but where OSGym scales agent rollouts on full OS replicas (1024-way Docker fleet, $0.23/replica/day), VideoAgentTrek scales pretraining trajectories from a passive source (YouTube) with no environment cost. It complements Computer Use Large: 48k screen recordings (~12,300 hours) of professional software (48k professional-software screen recordings, ~12,300 h) by automating the action labeling Markov AI relies on, and converges on essentially the same data scale as The First Fully General Computer Action Model (FDM-1) FDM-1 — but from a free, web-crawlable source rather than internal capture. For Tool-Use Agents it adds a visual tool-use pretraining recipe to a design space that has so far been dominated by text/API agents (MiroThinker, InfTool, Training-Free GRPO); for Synthetic Training Data it is the cleanest filed instance to date of “convert raw real video into structured action trajectories via a trained VLM” — a sibling to Action100M: A Large-scale Video Action Dataset‘s VLM+LLM hierarchical-captioning recipe, but with action-grounded labels rather than caption-style ones.