ACT in the browser — three.js SO-101 + leader-arm teleop + trained model running locally on the web

TL;DR

@pham_blnh built an end-to-end pipeline that brings the full lerobot ACT (Action Chunking Transformer) loop into a browser. The flow: design an SO-101 robot scene in three.js, connect it to a real leader arm for teleoperation, collect demonstrations, train an ACT model, then port the trained policy to run client-side. Demo at lerobot.binhph.am. Pitched as the first time ACT — the canonical lerobot imitation-learning recipe — has run locally on the web, and as a low-friction way to show “physical AI” without a physical robot in the room.

Key claims

• A full ACT teleop-train-deploy loop can be assembled on top of three.js (scene + visualization), a real leader arm (data collection), and a browser-hosted runtime (inference) [tweet body].
• The resulting browser demo runs locally — no server-side inference for the policy [tweet body, claim “runs locally on the web”].
• Framed as the first browser-local ACT demonstration of its kind, with the demo URL lerobot.binhph.am given as the artifact [tweet body].

Method

The tweet outlines a five-step pipeline rather than a paper:

1. Design an SO-101 (the lerobot reference arm) scene in three.js — i.e. the simulated proxy lives in WebGL, not a server-side MuJoCo/Isaac.
2. Connect that browser scene to a real leader arm hardware input, so a human can drive the simulated follower via leader-follower teleop.
3. Collect teleoperation demonstrations through that bridge.
4. Train an ACT model (the action-chunking-transformer imitation policy that ships with lerobot) on the collected data.
5. Port the trained ACT model to run in the browser and ship it as a static demo page.

No technical write-up is linked from the tweet; the artifact is the runnable demo at lerobot.binhph.am. The embedded video (~3 min) shows the working setup.

Results

No quantitative metrics — this is a working-system demo. Adoption signal at filing time: 34.4K views on the tweet. The functional result is that the trained ACT policy executes in-browser fast enough to demonstrate the SO-101 scene interactively.

Why it’s interesting

The closest filed sibling is mjswan: real-time interactive MuJoCo + ONNX policy simulation in the browser, which solves the upstream half of the same problem — shipping MuJoCo physics + an ONNX policy into the browser as a static site. This tweet completes the loop on the other end: instead of policies that were trained somewhere else, the training data itself is collected through the browser-leader-arm bridge, and the trained model is ported back into the same browser runtime. Together the two artifacts sketch a credible recipe for shareable, self-contained imitation-learning demos.

Within the VLA Models cluster, this is neither a new VLA recipe nor a counter-position to the existing levers (action pretraining, clean teleop, pointing/grounding, sensorimotor); it sits orthogonal as distribution infrastructure for the lerobot-style imitation-learning stack — the kind of thing that makes external researcher demos cheap to embed in a webpage or blog post, the way mjswan: real-time interactive MuJoCo + ONNX policy simulation in the browser already does for MuJoCo policies. For Luma’s character/scene work, the same browser-hosted-controller pattern is one route to publish controllable-scene demos without a Colab.

See also

• mjswan: real-time interactive MuJoCo + ONNX policy simulation in the browser — the upstream “physics + ONNX policy in a static browser site” framework; this tweet adds the data-collection + training half of the loop.
• VLA Models — broader cluster on VLA / imitation-learning recipes; this is an infra/demo entry rather than a new lever.
• RAIDEN: A Toolkit for Policy Learning with YAM Bimanual Robot Arms — TRI’s heavyweight teleop+data toolkit; sits at the opposite end of the friction-vs-fidelity axis from this lightweight browser pipeline.
• ABC: Scalable Behavior Cloning with Open Data, Training, and Evaluation — open ACT-style behavior-cloning training recipe at scale; this tweet is the laptop-scale “ship a demo” version of the same underlying class of model.