RAIDEN: A Toolkit for Policy Learning with YAM Bimanual Robot Arms

TL;DR

RAIDEN is Toyota Research Institute’s end-to-end data-collection toolkit for YAM robot arms, covering the full pipeline from hardware bring-up (camera calibration, teleop) through dataset packaging (multi-camera recording, format conversion, visualization). It targets the part of the bimanual-manipulation stack that papers like ABC and Spirit v1.5 build on top of but that has not had a single MIT-licensed, opinionated release: heterogeneous-camera mixing (ZED + RealSense in the same session), automated hand-eye + scene extrinsic calibration via ChArUco boards, leader-follower or SpaceMouse teleop, multi-depth backends (RealSense IR / ZED SDK / TRI Stereo / Fast Foundation Stereo), and conversion to a flat per-frame format ready for policy training. The toolkit ships a single CLI (rd ...) with subcommands for each pipeline stage, plus a terminal UI (rd console) for reviewing demos and correcting success/failure labels. Roadmapped but not yet shipped: built-in policy training/inference and a LeRobot-format converter.

Key claims

RAIDEN supports flexible bimanual or single-arm teleoperation via either active leader-follower arms or SpaceMouse end-effector control [README §Key features].
Inverse kinematics is manipulability-aware, built on PyRoki and J-Parse for smooth singularity-aware control rather than naive numerical IK [README §Key features].
Four depth backends are supported in a single session: RealSense IR structured light, ZED SDK stereo, TRI Stereo, and Fast Foundation Stereo (NVlabs) — the last giving manipulation-tuned high-quality depth [README §Key features].
Heterogeneous-camera mixing is first-class: ZED and Intel RealSense cameras can be combined freely across scene and wrist roles in a single recording session [README §Key features].
Automated extrinsic calibration uses ChArUco-board hand-eye estimation for wrist cameras and a separate static-extrinsic procedure for scene cameras [README §Key features].
The metadata workflow is decoupled from collection: a terminal UI (rd console) lets operators review demos, correct success/failure labels, and manage tasks and teachers post-hoc [README §Key features, §Commands].
Output is a flat policy-ready file format with synchronized frames, per-frame extrinsics, and interpolated joint poses, plus a separate rd shardify step to export WebDataset shards [README §Key features, §Commands].
A 3D-printable compliant fin-ray gripper design is included as part of the hardware story, not just the software [README §Key features, “Fin-ray gripper support”].
Policy training/inference, a LeRobot-format converter, and initial-scene-condition management are explicitly roadmapped but not yet shipped [README §Roadmap].

Method

RAIDEN is a single Python package exposing one CLI (rd) whose subcommands trace the manipulation-data lifecycle in order. rd list_devices enumerates connected cameras, arms, and SpaceMouse; rd record_calibration_poses + rd calibrate produce hand-eye extrinsics for wrist-mounted cameras and static extrinsics for scene cameras using a ChArUco board; rd teleop runs uncovered teleoperation for rehearsal; rd record collects timestamped demos with synchronized RGB-D from arbitrary camera mixes plus arm state; rd console opens a terminal UI for review and labeling; rd convert produces the policy-ready flat format with synchronized frames and per-frame extrinsics; rd shardify packages converted episodes as WebDataset shards; rd visualize plays a converted recording back through Rerun (the data-layer tool also tracked in Rerun — The Data Layer for Physical AI); and rd serve launches a policy inference server (the policy-side hooks are stubbed pending the roadmapped training integration).

The depth-backend story is the part that distinguishes RAIDEN from naive teleop rigs. Each camera role (scene vs wrist) can be paired with one of four depth providers — RealSense’s onboard IR depth, ZED SDK stereo, TRI’s own Stereo network, or NVlabs Fast Foundation Stereo — and the operator picks the right one per camera per session. For singularity-aware bimanual control, IK runs through PyRoki with J-Parse manipulability scoring rather than a damped-least-squares baseline, which matters for the wrist-twist and shared-workspace configurations bimanual teleop runs into.

Results

This is a hardware/software toolkit release, not a paper, so headline numbers are not in scope. What the release commits to qualitatively: a single CLI covering calibration → teleop → record → convert → visualize, four depth backends, heterogeneous-camera mixing, ChArUco-based automated extrinsics, leader-follower + SpaceMouse teleop, fin-ray gripper CAD, and WebDataset shard output. Reported policy-training integration, LeRobot-format export, and reproducible initial-scene-condition management are not yet in the release. The disclaimer is unusually direct (“research software, as-is, no warranty; the authors accept no liability for damage to property, equipment, or persons”) — useful context when evaluating it for production use.

Why it’s interesting

This is the first MIT-licensed, opinionated TRI-published end-to-end data-collection toolkit on the wiki, and it fills a gap the existing VLA papers have all dealt with privately. ABC: Scalable Behavior Cloning with Open Data, Training, and Evaluation (ABC) built its own passive-leader DAgger rig and abcdl dataloader specifically because no such common substrate existed; RAIDEN is essentially the upstream version of what ABC released as part of its own stack, and the roadmapped LeRobot exporter would slot it cleanly into the same data-pipeline lineage as Unitree open-sources UnifoLM-WBT-Dataset — humanoid whole-body teleoperation dataset (Unitree UnifoLM-WBT-Dataset). The four-depth-backend story — and especially the Fast Foundation Stereo integration — also pulls in the perception-foundation-model thread: it gives toolkit users a way to swap in modern foundation-model depth for the same teleop session that a 2022-era stereo rig would use RealSense IR for. Less directly, it sits adjacent to RoboCasa365 — Large-Scale Simulation of Everyday Tasks for Generalist Robots (RoboCasa365) on the “release a curated substrate for manipulation policy training” axis — RoboCasa365 ships the sim platform + demo set, RAIDEN ships the real-world data-collection plumbing on the other side of the sim-to-real boundary. For Luma’s robotics team this is a candidate hardware-bring-up shortcut; for the wiki it’s a data-point on the slowly emerging open-source robotics-tooling stack.