NeuralOS: Towards Simulating Operating Systems via Neural Generative Models

TL;DR

NeuralOS is a neural framework that simulates a graphical operating system (Ubuntu XFCE) by directly predicting screen frames in response to mouse and keyboard inputs. The architecture splits the job in two: a recurrent neural network tracks “computer state” across frames, and a diffusion-based renderer decodes that hidden state into a desktop image. Training data is a large-scale corpus of Ubuntu XFCE recordings composed of both randomly generated interactions and realistic sessions produced by AI agents. The model is shown to render coherent GUI sequences, follow mouse motion accurately, and reliably trigger state transitions like application launches; fine-grained keyboard input remains the main failure mode. A notable side-result: training on synthesized Doom interactions teaches the model to render an application that was never installed in the host OS, suggesting that generative neural interfaces can be learned purely from synthetic demonstrations.

Key claims

An RNN computer-state head + diffusion renderer can predict GUI screen frames directly from (mouse, click, keyboard) inputs, producing realistic GUI sequences without hand-coded window managers [Abstract].
The dataset combines two collection regimes — random scripted interactions and AI-agent-produced realistic interactions — to cover both broad action coverage and plausible task structure [Abstract].
The model accurately captures mouse interactions and reliably predicts coarse state transitions such as application launches; fine-grained keyboard interaction is the explicit weak point [Abstract].
Synthesized training data lets NeuralOS simulate an application that was never installed on the host OS, demonstrated by a Doom application rendered from synthetic demonstrations alone [Abstract].
After two months of training on scripted synthetic interactions, NeuralOS was deployed and continuously fine-tuned on 746 real user demonstration sequences (~88K frame transitions over 14 days), yielding substantial improvements on user-frequent tasks like Firefox interaction and folder creation (reported in the follow-up Interactive Training paper that uses NeuralOS as its case study) [Interactive Training, arXiv:2510.02297 §Case Study].

Method

NeuralOS treats the OS as a partially observable dynamical system whose observation is the screen image. The pipeline is autoregressive over time: at each timestep the model receives the latest user input event (mouse position, click, keystroke) and the previous hidden state, advances a recurrent neural network to produce a new computer state vector, and conditions a diffusion-based image decoder on that state to render the next screen frame. Training data is Ubuntu XFCE screen recordings paired with the input events that produced them; the corpus mixes scripted random interactions (for input-space coverage) with sessions driven by AI agents (for realistic task structure). After the initial offline training, the system supports continuous on-line finetuning on user demonstrations, as demonstrated in the Interactive Training follow-up.

Results

Qualitative experiments on Ubuntu XFCE show realistic GUI sequence rendering, accurate cursor tracking, and reliable application-launch state transitions [Abstract].
Doom application simulation from synthetic demonstrations alone — a transfer test the host OS could not perform, validating the “purely synthetic UI demonstrations” learning path [Abstract].
Fine-grained keyboard interaction is reported as the explicit failure mode; no quantitative breakdown of input-class accuracy is given in the abstract [Abstract].
Real-user finetuning result reported in the follow-up: 14-day deployment + 746 demonstrations (88K transitions) → substantial improvement on Firefox interaction and folder-creation tasks via the Interactive Training framework [Interactive Training, arXiv:2510.02297 §Case Study].

Why it’s interesting

NeuralOS is an early, GUI-specific point in the interactive generative world model design space that Genie 3 / Project Genie productized — but where Genie targets game-like 3D worlds, NeuralOS targets a 2D desktop with discrete state (windows, menus, application launches) and discrete inputs (clicks, keys). The RNN-state + diffusion-renderer split is structurally similar to the recurrent-memory-vector + DiT-renderer pattern that MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation later formalized for long video, and the chunk-streaming-AR pattern in FlowAct-R1: Towards Interactive Humanoid Video Generation — NeuralOS predates both and uses the same factorization for a non-video domain.

It also sits at an interesting angle to the computer-use-agent stack: OSGym: Scalable OS Infra for Computer Use Agents, The First Fully General Computer Action Model (FDM-1) (FDM-1), and Computer Use Large: 48k screen recordings (~12,300 hours) of professional software all assume the real OS is the environment and train agents to act in it; NeuralOS inverts the relationship and learns the OS itself as a generative model, which in principle removes the parallel-VM fleet from the loop. The Doom-from-synthetic-demos result is the headline evidence that the generative-OS framing can host capabilities the host machine doesn’t have — the same trick generative game-world models use to host non-existent levels.