mjswan: real-time interactive MuJoCo + ONNX policy simulation in the browser

TL;DR

mjswan is a Python/JS framework for shipping MuJoCo simulations with neural-network policy control as static, browser-hosted demos. It glues together MuJoCo’s WebAssembly build, ONNX Runtime (web), and three.js so that physics, policy inference, and rendering all run client-side — no simulation server, no install. The repo also ships demos for Robot Descriptions, MuJoCo Playground, and MyoSuite, all hosted on GitHub Pages. Featured in the official MuJoCo README and adopted by the GentleHumanoid live demo. Not a research paper — a piece of infrastructure that makes interactive embodied demos cheap to ship and embed.

Key claims

Real-time MuJoCo physics + ONNX policy inference can be run entirely client-side via WebAssembly, removing the need for a simulation server [README §Features].
Interactive force application against the simulated body is a first-class affordance — users drag/push parts of the model at runtime [README §Features].
The framework is portable across desktop / mobile / VR (WebXR) and embeddable in a webpage or Google Colab notebook [README §Features].
Pre-built demos cover the Robot Descriptions zoo, MuJoCo Playground, MyoSuite, and a mjlab integration [README §Demos].
Distribution surfaces: PyPI (pip install mjswan), npm (npm install mjswan), and a CLI for running the bundled demos [README §Quick Start].

Method

mjswan is built on three external pieces: google-deepmind/mujoco’s wasm build for the physics engine, microsoft/onnxruntime for running exported RL policies in-browser, and three.js for rendering. A Python builder API (mjswan.Builder().add_project(...).add_scene(name=..., spec=mjspec).build()) takes a mujoco.MjSpec plus optional ONNX policy and bakes out a static site that can be hosted on GitHub Pages or any CDN. The runtime is a tight loop: step MuJoCo in WASM, query the ONNX policy, apply controls, render with three.js, and accept user force inputs back into the simulator. Acknowledged as inspired by the Facet project demo (Tsinghua) and built on zalo/mujoco_wasm.

Results

No quantitative results — it is a framework, not a method. Adoption signals: the MuJoCo README added a link on 2026-04-08, and the GentleHumanoid project adopted it for its live demo on 2026-05-06. Demo gallery hosted at ttktjmt.github.io/mjswan.

Why it’s interesting

For the team’s interactive-character work, mjswan is a candidate front-end for the coarse control side of a video-to-video puppetry pipeline. Jon’s framing on Slack: instead of feeding a video generator exact limb positions, let a user shove a rigged body around in a real-time browser sim, then run a video-to-video AI-human model over the top using only the fuzzy interaction information (forces, gross body state) — not the precise joint trajectories. mjswan does exactly the hard browser-side plumbing (WASM physics + force application + ONNX policy + cross-platform deploy) that such a demo needs. It is “probably heavy and unnecessary in-between” as a research dependency, but useful as a reference implementation of how to host an interactive embodied scene in pure JS.

The other carry-forward: mjswan is shaped like infrastructure that other researchers can fork to ship interactive demo pages of their controllers. That matters for how Luma might publish controllable-video demos — a static GitHub Pages site with a draggable rigged body is much friendlier to share than a Colab.