LeRobot Sim-to-Real Tutorial — 72% real-SO-100 success from LuckyEngine sim demos only

TL;DR

Lucky Robots ships an end-to-end tutorial for training a manipulation policy entirely on scripted simulator demos in LuckyEngine (a MuJoCo-based, Vulkan-rendered simulator with a Python gRPC SDK) and deploying it zero-teleop on a real SO-100 arm via LeRobot 3.0. The pipeline records 200 episodes with C# scene scripts (no human teleoperation anywhere), trains a two-camera ACT / DICE-IMLE policy at 96×96 / 30 Hz, sanity-checks it in a second simulator (Genesis) as a sim-to-sim generalization probe, and closes the loop on the physical robot. The case study reports a 17-percentage-point “transfer cliff” between LuckyEngine (84% in-sim) and Genesis (67% zero-shot) attributed to renderer differences, largely recovered by a DICE-v3 encoder pretrained on ~21k paired sim↔real frames with InfoNCE + VICReg + DANN losses, yielding 72% success on the real SO-100 with no real-world demonstrations in the training set.

Key claims

Scripted C# scene scripts inside LuckyEngine record LeRobot 3.0 datasets directly to disk without any teleoperation — the Observer API streams action, observation.state, and observation.images.<cam> (h264 mp4 chunks) into the standard data/, videos/, meta/ layout [§1].
A vanilla ACT policy trained on 200 LE-recorded SO-100 episodes reaches 84% in-sim success at checkpoint 044800, up from 52% at 020160, on a lego-pick-and-place task with a 6-DoF joint-state policy and ~66M-param 1D U-Net generator [§6, Vanilla IMLE table].
Zero-shot transfer to Genesis (different physics solver and renderer, same MJCF and camera intrinsics) drops the same checkpoint from 84% to 67% — a 17 pp “transfer cliff” attributed to renderer differences (colour cast, micro-shading, edge sharpness) rather than physics [§6].
A frozen DICE-v3 image encoder — trained on ~21k paired LE↔Real frames producing L2-normalized (B, 128, 6, 6) grids with dense InfoNCE + VICReg + DANN losses — recovers most of the transfer cliff: 76% in Genesis and 72% on the real SO-100 at 25-trial evaluations [§6, DICE-IMLE tables].
Total per-step latency budget on the real robot is 33 ms (30 Hz): camera async_read + encoder read ≈5–8 ms, preprocess ≈2–3 ms, policy.predict with EMA + AMP 9–17 ms, action mapper + safety scan <1 ms, send_position ≈2–4 ms [§5, latency table].
Deployment-side gotchas that break policies silently: unmirrored camera flip (cv2.flip(..., 1) if LE camera was mirrored), radians↔degrees action unit mismatch, uncalibrated joint zero, and clipping-instead-of-aborting when actions drift off-distribution [§5].
Genesis is prescribed as a cheap sim-to-sim generalization probe before touching hardware, with axis-conversion hz_to_gs(p) = (p[0], -p[2], p[1]) bridging LE’s Y-up and Genesis’s Z-up conventions [§4].

Method

The recipe is a five-stage pipeline. (1) Record in sim. A C# script inside LuckyEngine drives an SO-100 through waypoints via DriveSO100ToWaypoints() inside an Observer.StartRecording() / EndCurrentEpisode(ok) loop; the engine writes LeRobot-3.0-compatible parquet chunks and mp4-encoded camera streams in parallel. 200 episodes with two 96×96 cameras at 30 Hz produced the case-study dataset. (2) Train ACT. lerobot-train runs 100k steps on a consumer GPU (RTX 3090/4070/4090, 2–6 hours) with policy.chunk_size=100 and default CVAE kl_weight=10. (3) In-domain eval. The luckyrobots Python SDK exposes a synchronous session.step() returning ObservationResponse with state and camera frames together; the policy is looped for MAX_STEPS per trial across N_TRIALS. (4) Sim-to-sim probe. The same checkpoint is run in Genesis (a different physics solver and renderer) using the same MJCF and matched camera intrinsics after Y-up → Z-up conversion — a leading indicator of real-world transfer. (5) Real deployment. LeRobot’s SOFollower at use_degrees=True with two RealSense cameras at 640×480/30 fps, resized to 96×96, driven by an ActionMapper that converts between real-robot units and dataset units with dataset stats + calibration; a safety scan aborts (rather than clips) out-of-range actions.

The transfer story hinges on DICE-v3, a frozen vision encoder pretrained on ~21k LE↔Real image pairs producing 128-dim L2-normalized feature maps at 6×6 spatial resolution. It’s trained with a mixture of dense InfoNCE (per-spatial-location contrastive), VICReg (variance/invariance/covariance regularization), and DANN (domain-adversarial) losses, and then swapped into the ACT visual backbone in place of the vanilla ResNet18 + SpatialSoftmax.

Results

Vanilla IMLE in LE (25 trials × 3 checkpoints): success 52% → 60% → 84% at steps 020160 / 022400 / 044800; lift 72/92/96; grasp 68/92/96 [§6].
Transfer cliff: checkpoint 044800 drops from 84% (LE) to 67% (Genesis zero-shot), a 17 pp gap attributed to the renderer, not the physics [§6].
DICE-IMLE in Genesis sim2sim (25 trials): 76% success at step 006280 (lift 80, grasp 84); 64% at step 009430 (lift 68, grasp 72) — the earlier checkpoint transfers better [§6, DICE-IMLE Genesis table].
DICE-IMLE on real SO-100 (25 trials each): 60% at step 18000, 72% at step 19500 — the headline number for a policy that saw zero teleoperation data anywhere in its training [§6, DICE-IMLE Real SO-100 table].

Why it’s interesting

The tutorial reads like an operational counterpart to RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies and Lightwheel’s Introducing SimReadyGen — Agentic Simulation Generation for Physical AI — an actual on-hardware validation of the “record entirely in a simulator, deploy on the real robot” thesis, with a quantified renderer-gap number (17 pp) that most sim-only training claims never surface. That transfer-cliff finding lines up with SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation‘s emphasis on scene-level realism as the bottleneck, and the DICE encoder recovery pattern mirrors REGRIND: A Minimalist Retargeting-Guided RL Recipe for Dexterous Manipulation‘s finding that a modest bridging step (there, retargeting-guided RL; here, paired-frame contrastive pretraining) is enough to close most of the gap without ever collecting real demos. Contrasts with Evo-RL: Open Real-World Offline RL on SO-101 and AgileX PiPER which needs real-world offline RL rollouts on the same SO-101 platform — this pipeline gets to comparable real-robot success rates without any real teleop, at the cost of a paired-frame calibration dataset.