ReViV: Reconstructing the Viewer and the View in 4D from Monocular Egocentric Video

ListenOne Video, Six Things: ReViV Reads the Viewer and the View8:43

Show notes

ReViV is a single feed-forward model that reconstructs both the *viewer* — full-body pose, hand motion, gaze — and the *view* — camera trajectory and depth — in 4D from one monocular egocentric RGB video. Instead of chaining specialist perception heads and feeding in pre-computed camera poses, it learns the joint distribution over six coupled modalities with a Masked Generative Egocentric Transformer. The authors report state-of-the-art results across HoloAssist, HOT3D, ARCTIC, Aria Digital Twin, and TACO, with fast inference and no heavy task-specific priors. We unpack the mechanism and what the "predict camera jointly" inversion means for anyone working with egocentric video. **Sources** - ReViV: Reconstructing the Viewer and the View in 4D from Monocular Egocentric Video

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

ReViV is a unified feed-forward model that reconstructs both the viewer (full-body motion, hand motion, gaze direction) and the view (camera trajectory, depth) in 4D from a single monocular egocentric RGB video. It formulates the task as learning the full joint distribution over RGB + camera + gaze + body + hand + depth, and implements it with a single Masked Generative Egocentric Transformer that produces temporally consistent outputs across all six modalities in one forward pass with fast inference. Reports SOTA on HoloAssist, HOT3D, ARCTIC, Aria Digital Twin, and TACO for holistic ego-body/hand/gaze reconstruction and camera tracking, with competitive egocentric depth estimation — without relying on pre-computed camera trajectories or heavy task-specific priors that prior systems require. Code and models released.

Key claims

Egocentric 4D reconstruction can be posed as learning the full joint probability distribution over six coupled modalities (RGB, camera, gaze, body, hand, depth) rather than a chain of separately-trained perception heads [§Abstract].
A single feed-forward Masked Generative Egocentric Transformer suffices to produce all six outputs simultaneously and temporally consistently, removing the two-stage “reconstruct camera first, then everything else” dependency that prior egocentric pipelines require [§Abstract].
Removing the auxiliary pre-computed camera trajectory input (a common assumption in prior egocentric body/hand reconstruction) does not degrade quality when scene and human motion are modeled jointly [§Abstract].
Reports SOTA across holistic ego-body reconstruction, hand reconstruction, gaze reconstruction, and camera tracking, with highly competitive egocentric depth estimation on HoloAssist, HOT3D, ARCTIC, Aria Digital Twin, and TACO [§Abstract].
Achieves the above with fast inference speed relative to the reconstruction pipelines it replaces [§Abstract].

Method

ReViV takes a single monocular egocentric RGB video and outputs a temporally-consistent 4D reconstruction consisting of the camera trajectory, per-frame depth, and the viewer’s full-body pose, hand pose, and gaze direction. Rather than chaining separate specialist models, the paper formulates the problem as learning the joint distribution p(RGB, camera, gaze, body, hand, depth) and trains a Masked Generative Egocentric Transformer that consumes the RGB video tokens together with masked slots for the other five modalities and jointly predicts them in a single feed-forward pass. The masked-generative formulation is what unifies the pipeline: rather than requiring pre-computed camera poses as input (as HumanEgo-style pipelines do via Aria MPS SLAM) or running per-frame regressors per modality, the transformer generates the full stack from RGB alone at inference. Evaluation spans five benchmarks — HoloAssist (holistic activity), HOT3D and ARCTIC (hand-object), Aria Digital Twin (Aria devices), and TACO (categorical action) — providing coverage across egocentric hardware and interaction regimes.

Results

SOTA on holistic ego-body reconstruction, hand reconstruction, gaze reconstruction, and camera tracking across HoloAssist / HOT3D / ARCTIC / Aria Digital Twin / TACO; competitive on egocentric depth [§Abstract].
Fast inference vs prior multi-stage egocentric reconstruction pipelines [§Abstract].
Does not rely on pre-computed camera trajectories or heavy task-specific priors, unlike prior baselines [§Abstract].

Why it’s interesting

ReViV is the strongest joint-generative-transformer treatment of egocentric perception the wiki has seen. It unifies the perception layer that HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos and LAMP: Localization Aware Multi-camera People Tracking in Metric 3D World both build on — where the quality of the SLAM + 21-keypoint hand pipeline is treated as a load-bearing bounded input — into a single masked-generative feed-forward model that removes the pre-computed-camera dependency entirely. That inversion (predict camera jointly with body/hand/gaze/depth instead of consuming it as input) is a concrete test of whether the multi-modal joint-generative recipe from Video Generation Models are General-Purpose Vision Learners (GenCeption unifies keypoints + camera + depth + normals in one text-to-video backbone) generalizes to first-person perception. It also gives the pose cluster its first end-to-end system that emits gaze alongside body and hand — a modality neither MoCapAnything nor MAMMA nor SAM 3D Body handle. For consumers of egocentric perception in the downstream robot-learning stack (2606.20521-humanescale-egocentric-human-video-can regime and adjacent), the practical question is whether the “predict camera + body + hand jointly” recipe survives at web-scale egocentric video, where SLAM signal quality varies wildly.