Aria Digital Twin to MCAP — Load ADT Episodes in FiftyOne

TL;DR

An end-to-end recipe from Voxel51’s Harpreet Sahota (~300 lines of Python) for converting one Meta Aria Digital Twin (ADT) episode — shipped as a .vrs recording plus eight ground-truth zips behind Meta’s non-commercial license — into a single synchronized .mcap file that FiftyOne’s multimodal viewer opens natively, no exporter or custom loader. The pipeline authors twelve Foxglove-schema channels (RGB, depth, segmentation, synthetic renders, 2D and 3D object boxes, human skeletons, eye gaze, semi-dense point cloud, 6-DoF device trajectory, IMU, plus a static-boxes channel) against one shared boot-nanosecond clock, and documents the specific silent gotchas that break sync (Y-up-vs-Z-up frame convention, per-modality write order desyncing playback, redrawing all 354 objects every frame vs logging the 304 static ones once). Useful as a template for turning any non-MCAP robotics dataset (VRS, protobuf, rosbags, PNG+JSON folders) into a format that drops into MCAP-native multimodal viewers.

Key claims

One Aria Digital Twin episode (Apartment_release_multiskeleton_party_seq101_M1292, 145 s) is a 12.4 GB download across seven ADT components, with raw depth alone accounting for 7.8 GB [Step 1, size table].
The MCAP output is one 2.5 GB file with 181,444 messages across 12 channels: 4,349 RGB frames + matching depth/segmentation/synthetic, 2,832 annotation and skeleton messages, 150 point-cloud republishes, 7,198 pose messages, 145,371 IMU samples, and one static-box message covering 304 objects [Step 2, end-to-end counts].
ADT’s world frame is Y-up while Foxglove/ROS 3D viewers assume Z-up — a silent axis-swap gotcha handled by three geometry helpers reused across every 3D and image stream [Step 2, “Where does the axis and rotation math live?”].
304 of 354 tracked objects never move in this episode; logging static objects once with a persistent lifetime and only redrawing the 50 dynamic ones reduces the /scene/3d_boxes channel from ~600 MB to 77 MB and keeps the 3D tile from lagging behind the video [Step 2, “The 3D boxes”].
Aria cameras are physically mounted rotated, so image streams need a 90° clockwise rotation and 2D box coordinates must be remapped consistently ((x, y) → (height - 1 - y, x)); skipping either misaligns boxes from objects [Step 2, “Why rotate the image, and rotate the boxes differently?”].
Writing each modality in its own loop produces contiguous per-channel blocks spanning the full episode; without a post-hoc reorder_mcap pass that re-sorts every message by log_time, a reader stepping through time has to jump between physically distant regions of the file per channel per frame — the usual root cause of stuttering multimodal playback [Step 2, reorder rationale].
Point clouds and dynamic transforms need periodic republishing (~1 Hz for the cloud, ~50 Hz for TF) because dynamic transforms only resolve within a ~50 ms boundary clamp of the playhead; downsampling too aggressively makes the device marker flicker or vanish [Step 2, transform notes].
The Aria Digital Twin license (personal, non-commercial research only, no redistribution or public display of the dataset “or any materials or works derived from it”) blocks sharing the authored MCAP files, but does not block sharing the pipeline code [FAQ, license section].
Message counts and a successful reorder_mcap are structural checks, not correctness checks — the post recommends at least one physical-plausibility assertion (e.g. semi-dense point cloud spans ~17–21 m per axis, apartment-scale) before trusting the output [Step 2, “Is a clean run enough?”].

Method

The pipeline is one Python script broken into three steps against foxglove-sdk 0.26.0, projectaria-tools 2.2.0, mcap 1.3.1, and fiftyone 1.21.0. Step 1 downloads seven ADT components using Meta’s aria_dataset_downloader (from a licensed CDN JSON manifest) or a plain-Python equivalent, and symlinks the results into the video.vrs + mps/slam/ layout that AriaDigitalTwinDataProvider expects. Step 2 authors the MCAP in five passes inside one foxglove.open_mcap(...) block: (a) static 3D boxes logged once with persistent lifetime, (b) semi-dense point cloud republished ~1 Hz so it doesn’t disappear on scrub, (c) 6-DoF device pose downsampled to ~50 Hz to stay inside the dynamic-transform boundary clamp, (d) a per-frame loop keyed to the RGB camera’s timestamps that emits RGB + eye gaze + 2D/3D dynamic boxes + human skeleton + depth + segmentation + synthetic (all rotated 90° CW and axis-corrected), and (e) IMU straight from the raw VRS provider. A final reorder_mcap pass rewrites the file so each disk chunk holds a narrow time slice across all channels, which is what actually keeps playback synchronized. Step 3 creates a FiftyOne dataset by treating one .mcap file as one multimodal sample — fo.Sample(filepath=...) is enough because FiftyOne reads Foxglove-schema MCAPs natively and the viewer auto-tiles Image / 3D / Plot channels sharing one playback clock.

Results

Not a benchmark — this is a tooling walkthrough. Concrete numbers: 145 s ADT episode → 2.5 GB .mcap, 181,444 messages, 12 synchronized channels, playing back at 30 Hz RGB in FiftyOne’s multimodal viewer. Static-vs-dynamic box split takes /scene/3d_boxes from ~600 MB to 77 MB. The recipe generalizes to any non-MCAP source (VRS, protobuf, rosbags, “a folder of PNGs and a JSON”), which is the tweet’s framing: robotics datasets ship in whichever format the collecting lab preferred, and almost none of them drop into a multimodal viewer without a fight.

Why it’s interesting

Sits at the substrate-tooling layer of Human-to-Robot Retargeting — the concept page tracks retargeting recipes that source supervision from egocentric-video corpora, and Aria Digital Twin is one of the higher-fidelity real+GT-3D-hand-object corpora in that space. HOT3D (HOT3D — Egocentric Multi-View Dataset for 3D Hand-Object Interaction Understanding) already flagged Aria + Quest 3 multi-view egocentric with mocap 3D ground truth as a natural benchmark for hand-object retargeting; ReViV (ReViV: Reconstructing the Viewer and the View in 4D from Monocular Egocentric Video) reports SOTA on Aria Digital Twin specifically as one of its benchmark suite. This post is the middleware that lets ADT drop into a scrubbable multimodal viewer without writing a per-project loader — complementing dataset catalogs like datasets.bot — Curated catalog of robotics and embodied-AI training datasets (which tells you what datasets exist) with a template for what to do once you’ve downloaded one. The gotcha inventory (axis conventions, static-vs-dynamic split, per-modality vs interleaved write order, transform boundary clamp) is the kind of tacit knowledge that usually lives in one engineer’s head and is precisely why “shipping a dataset in MCAP-native form” isn’t yet the industry default.