The First Fully General Computer Action Model (FDM-1)

TL;DR

Standard Intelligence’s FDM-1 is a foundation model for computer use, trained autoregressively on next-action prediction over video + key/mouse tokens — no chain-of-thought, no BPE text, no VLM transfer. The recipe stages an inverse dynamics model (IDM) trained on 40k hours of contractor-labeled screen recordings, then uses it to pseudo-label an 11M-hour internet-scale screen-recording corpus, then trains the forward dynamics model on that. A custom video encoder hits ~100× more token-efficient compression than OpenAI’s encoder (≈1.7 hours of 30 FPS video in 1M tokens), which is what makes minutes-to-hours of context tractable. The team also built RL eval infrastructure that drives >1M VM rollouts/hour across 80k forking Ubuntu VMs at 11 ms screen-capture-to-action round trip.

Key claims

Computer-use agents have been bottlenecked on (a) annotated data scale (<20 hours in the largest public dataset) and (b) context length (a few seconds for VLM-based approaches), making long-horizon tasks like CAD, finance, or coding livestreams unreachable [§intro].
An inverse dynamics model can label internet-scale screen recordings near-causally enough to substitute for contractor annotations: the IDM-labeled data outperforms contractor data on Target Accuracy, Symbolic Memory, and UI Manipulation, and underperforms (more slowly improving) on typing/verbal understanding [§Eval Infrastructure, Fig. evals].
Labeling video actions is fundamentally non-causal (e.g. a Cmd+C is only resolvable after the paste appears), and a masked-diffusion architecture beat CTC and causal cross-entropy at this task — overfitting more slowly and producing more accurate labels via top-k-confidence-first unmasking [§Inverse Dynamics, fn 6-7].
The custom video encoder, trained with a V-JEPA-style masked self-supervised prediction objective, converges ~100× faster than a ViT and hits state-of-the-art video-frame→token compression: 32k tokens ≈ 3.5 min, 200k ≈ 20 min, 1M ≈ 1h40m of 30 FPS video [§Video Encoder, Fig. context-table].
The forward dynamics model operates directly on video + action tokens with no language-model transfer, no CoT, no BPE, no tool use — keeping inference latency low and modeling action types VLMs cannot (scrolling, 3D modeling, gameplay) [§Forward Dynamics, fn 9].
Mouse-movement tokenization uses 49 exponentially-sized bins per X/Y component, normalized by screen dimension, so small frequent motions get fine bins and large rare motions coarse ones; an auxiliary next-click-position head is co-trained with movement tokens [§Forward Dynamics, Fig. exponential-binning].
The eval infrastructure runs >1M rollouts/hour on 80k forking Ubuntu VMs (1 vCPU, 8 GB) — a single H100 controls 42 VMs in parallel; VM forking from full memory snapshots is what makes test-time-compute reuse of a starting state cheap [§Eval Infrastructure].
End-to-end inference latency is 11 ms screen-capture → action, achieved via GPU/VM colocation, cumulative-seqlen packing, low-latency VNC, and custom Rust device-input bindings — important because the model was trained without lag in distribution [§Eval Infrastructure].
Computer-use pretraining transfers to embodied control: after fine-tuning on <1 hour of collected data, FDM-1 reaches 50% accuracy on a 3-class self-driving key-press task (no-op / left / right) navigating SF turns via a web interface, vs a video-encoder-only baseline with steeper scaling trends [§Now What?, Fig. self-driving-chart].
The thesis: with these two unlocks (multi-hour 30 FPS context + 11M-hour training corpus), computer action moves from a data-constrained regime to a compute-constrained one — the same regime shift that made GPT-3 possible from internet text [§Now What?].

Method

The pipeline has three stages. Stage 1 trains an IDM on 40k hours of contractor-labeled screen recordings; the IDM is a masked-diffusion model that conditions on all video frames simultaneously and predicts the action tokens between frames via iterative top-k-confidence unmasking — exploiting the fact that action labeling is non-causal (a Cmd+V needs to see future paste content to disambiguate). Stage 2 runs the IDM over the 11M-hour internet video corpus to produce action pseudo-labels. Stage 3 autoregressively trains the FDM on (video tokens, action tokens) sequences for next-action prediction, with mouse deltas tokenized into 49 exponentially-binned X/Y components and an auxiliary next-click-position head co-trained alongside.

The video encoder is trained separately via masked self-supervised prediction (V-JEPA-style) on screen recordings — chosen over fixed-compression patch tokenizers because screen-recording information density varies wildly (blank cursor pan vs dense scrolling text), and a fixed-ratio scheme has to trade text fidelity for compression. Probe tasks during encoder training: inverse dynamics, action prediction, frame reconstruction, random text transcription. The encoder achieves ~100× faster convergence than a ViT baseline and reaches the compression ratios above.

Results

Video encoder compression: 1M tokens ≈ 1h40m of 30 FPS video, ~50× better than the prior SOTA video encoder and ~100× better than OpenAI’s encoder.
IDM data vs contractor data on internal eval suite: IDM-labeled wins on Target Accuracy, Symbolic Memory, UI Manipulation; loses on typing/verbal understanding (attributed to IDM label noise on text). The team flags a future contractor + IDM mix.
Self-driving transfer: with <1 hour of fine-tuning data, FDM-1 reaches 50% key-press-class accuracy navigating SF blocks via a web driving interface; baseline (encoder only, no internet pretrain) sits well below with shallower scaling.
Eval infra: >1M rollouts/hour, 80k VMs, 42 VMs per H100, 11 ms round-trip latency.
No external benchmark numbers (e.g. OSWorld, Visual WebArena, Mind2Web) are reported in the post.

Why it’s interesting

This is the clearest articulation on file of the “VPT-for-computers” thesis — internet-scale unsupervised video as the substrate for computer-use agents, with IDM-labeling closing the action-supervision gap. It directly contrasts with the wiki’s current computer-use prior, which is dominated by VLM-finetuning + per-task RL recipes (e.g. the WebGym / VisGym line in VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents and the agentic-vision framing of Introducing Agentic Vision in Gemini 3 Flash). It also lands squarely in Sitzmann’s foundation-layer position from The flavor of the bitter lesson for computer vision — generative-rollout / dynamics-prediction objectives over raw pixels, no explicit intermediate representations, no language-model scaffolding — and pushes the same logic past video understanding into video action. The encoder-compression number (≈2 hours / 1M tokens) is the single most aggressive video tokenizer the wiki has seen and worth comparing against the VBVR / Action100M / CoPE-VideoLM stack (CoPE-VideoLM: Codec Primitives For Efficient Video Language Models).