Ego-OSCAR: Egocentric Open source Stereo CAptuRe System

TL;DR

Ego-OSCAR is a fully open-source head-mounted stereo-inertial capture device (hardware-synchronized global-shutter stereo camera + 6-axis IMU + embedded SBC + real-time microcontroller) with a complete bill of materials under USD 200 in commercially-available parts plus 3D prints, and a companion 550-hour egocentric stereo+IMU dataset across 100+ environments collected by a distributed contributor network of 25+ operators. It is pitched as the cheapest defensible substrate for crowdsourced egocentric capture — deliberately not competing with the per-unit fidelity of Project Aria — with hardware CAD, capture pipeline, watchdog firmware, per-session stereo calibration, and 209,315 free-form action segments + per-frame hand detections all released. The paper’s argument is that egocentric data is a bottleneck for VLA and world-action models and the device itself has received little attention: monocular rolling-shutter consumer cameras with unsynchronized IMU dominate open corpora, and closed platforms like Project Aria cannot be freely distributed to large contributor networks.

Key claims

A COTS-only, no-custom-PCB stereo-inertial capture rig can be built under USD 200 per unit and reach sub-0.03 px per-camera reprojection error after per-session calibration, with 0.4 px mean epipolar error across 13 deployed devices after rectification [§Sensor fidelity].
Global-shutter hardware-synchronized stereo (as opposed to monocular wide-FOV with mirrors, e.g. UMI-style) enables standard stereo calibration + rectification pipelines and successful SGBM + RAFT-Stereo disparity across the full 126° field of view [§Data utility].
Treating pose estimation as an offline batch problem rather than running on-device SLAM lets the hardware optimize strictly for data ingestion (Rockchip RK3588 with hardware MJPEG-decode + H.264-encode on a wearable power budget) — the RK3588 sits at the inflection point where hardware media acceleration meets wearable power [§Embedded SBC].
A real-time microcontroller (Xiao ESP32-S3) that (a) taps the camera start-of-exposure signal to bridge the camera and IMU clock domains, (b) drives an RGB status LED, and (c) implements a 1 Hz heartbeat watchdog is what turned early field deployments from silently-empty sessions into an eventual 96% end-to-end usable-session rate [§Real-time microcontroller, §Deployment scale].
After per-session offset correction the residual visual-inertial lag is 700 µs (Kalibr Cam-IMU test), with ICM-20948 accelerometer noise density 3.64 × 10⁻² m/s²/√Hz and gyroscope bias instability 9.68 × 10⁻⁴ rad/s — consumer-grade, acceptable for gravity alignment / motion classification / coarse VI fusion but explicitly not sufficient for long-horizon inertial integration; the I²C-bus design makes a BMI088/ISM330 swap possible without firmware changes [§IMU noise, §Consumer-grade IMU].
VINS-Fusion stereo-inertial converges on 12 of 20 held-out sequences (vs 15/20 for an active-stereo Intel RealSense) — reported explicitly as a convergence rate, not a trajectory-accuracy result, because no motion-capture or surveyed ground truth is available; the paper declines to ship a camera-trajectory annotation layer for this reason [§Data utility].
The Ego-OSCAR-550h release ships 1,462 sessions with the IMU stream present in 1,271 (86.9%); the rest are released without an inertial stream rather than with an unverified one, and sessions that fail calibration or lack a usable synchronized trace are excluded rather than shipped with caveats [§Quality control].
The dataset ships two corpus-wide annotation layers (not raw sensor streams): 209,315 free-form action segments (median 94 per session) covering 460 action verbs, 32,630 object phrases, 57,104 verb–object combinations with a long-tailed distribution (top 20 expressions = 1.5% of instances, no single expression > 0.31%), plus per-frame 3D hand reconstructions [§The Ego-OSCAR-550h dataset].

Method

The rig is a ~280 g head-worn all-in-one: a Dexcin USB dual-Omnivision global-shutter stereo camera module (1280×720/30 FPS per camera, 126° FOV, 42 mm baseline, single UVC endpoint, start-of-exposure trigger exposed on a STRB pin), a Radxa ROCK 5C SBC (Rockchip RK3588, hardware MJPEG decode + H.264 encode), a TDK ICM-20948 6-axis IMU sampled at 120 Hz over I²C, and a Seeed Xiao ESP32-S3 microcontroller. The MCU serves three roles: it interrupts on the camera’s SoE pin to timestamp every frame exposure against its monotonic clock while simultaneously reading the IMU, forwarding a merged (SoE timestamps + IMU samples) stream to the SBC over UART — an offline pass then aligns SoE timestamps to video frame numbers to produce a synchronized trace; it drives an RGB LED strip from a 1 Hz heartbeat sent by the SBC; and it implements a two-second watchdog that puts the device into a visible error state if the SBC hangs.

Calibration is per-session (not per-device-model) — every session ships a calibration.json with a pinhole + radial-tangential (k1, k2, p1, p2, k3) fit per camera plus a 3×3 rotation, translation, and the ~42 mm baseline, from an 8×6 chessboard with 30 mm squares captured at native 1280×720. Three quality filters gate the release: watchdog-driven capture-time abort of silently-empty sessions; per-batch validation of decodability, expected clip count/duration, and companion IMU/calibration artifacts; and selection-time screening for hand visibility with rejection (rather than caveat-shipping) of sessions that fail calibration or lack a usable synchronized trace. The device is powered by a hot-swappable 10,000 mAh USB-PD bank giving 5–6 hours of sustained capture.

Results

Deployment scale: 96% end-to-end usable-session rate across 25+ operators, 1,462 released sessions totaling ~550 h, 100+ environments (residential kitchens/living rooms/bedrooms + small commercial), with three field-identified failure modes (thermal shutdown above 35 °C in sessions >90 min, SD card I/O errors, cable strain at the camera-to-SBC connector) engineered out [§Deployment scale].
Sensor fidelity: sub-0.03 px per-camera reprojection error; 0.4 px mean per-pixel epipolar error after rectification across 13 devices; 700 µs residual visual-inertial lag after per-session offset correction [§Sensor fidelity].
VI odometry: VINS-Fusion converges on 12/20 held-out 1–3 min sequences (vs 15/20 for an Intel RealSense with active stereo) — explicitly framed as an operating-envelope characterization, not a trajectory-accuracy result [§Data utility].
Annotations: 209,315 action segments, 460 verbs, 32,630 object phrases, 57,104 verb–object combinations; top 20 task expressions = 1.5% of instances; median 94 segments per session; per-frame 3D hand reconstructions ship alongside the video + IMU [§The Ego-OSCAR-550h dataset].
Activity mix (labeled hours): cooking + food prep 187 h; generic manipulation + transitions 106 h; dishwashing + cleanup 90 h; textile/craft 54 h; laundry 45 h; organizing 39 h; cleaning 29 h [§The Ego-OSCAR-550h dataset table].

Why it’s interesting

Ego-OSCAR is the first filed capture-device-as-artifact release in the Luma wiki’s egocentric-data cluster: prior open drops in the same lineage — EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World (Aria-Gen-1 across a consortium), RekaDaily-10k: Collecting 10,000+ Hours of Egocentric Household Manipulation Data (RekaDaily-10k, 10,312 h of unscripted household video), Egocentric-1M — largest egocentric video dataset (Build AI / Eddy Xu announcement) (Build AI’s Egocentric-1M), Ego1 — egocentric capture headset for Physical AI (General Intelligence Labs) (GI Labs Ego1 headset), HOT3D — Egocentric Multi-View Dataset for 3D Hand-Object Interaction Understanding (HOT3D on Aria + Quest 3) — either standardized on a closed high-fidelity platform or a proprietary in-house rig with dataset numbers alone. FPV Labs’s contribution flips the dependency: publish the rig so a distributed contributor network can build the corpus, with the paper’s opening argument being that the field’s data bottleneck now sits at the device more than the model. It is the closest sibling to HiFi-UMI: Learning Deployable Manipulation Policies from High-Fidelity UMI Data Alone HiFi-UMI, which is also a head-mounted offline stereo-inertial rig with sub-ms cross-sensor sync, though HiFi-UMI targets action-fidelity for UMI-style hand-held bimanual collection while Ego-OSCAR targets cheapest-defensible-substrate for pure egocentric video + IMU. The design choice of not running SLAM on-device — treat pose as an offline batch problem, spend the wearable power budget entirely on ingestion — is a distinct architectural bet worth watching against Project Aria’s on-device pose estimation.