EgoTL: Egocentric Think-Aloud Chains for Long-Horizon Tasks

TL;DR

EgoTL introduces a “say-before-act” capture protocol and dataset for egocentric long-horizon household tasks: collectors speak step-by-step goals and reasoning out loud with word-level timestamps, then the pipeline calibrates physical properties via metric-scale spatial estimators, a memory-bank walkthrough for scene context, and clip-level navigation/manipulation tags. The result is EgoTL-Bench, which evaluates VLMs and world models on six tasks across three layers (memory-conditioned planning, scene-aware action reasoning + next-action prediction, perceptual-metric understanding) over minute-long sequences across >100 daily household tasks. Frontier VLMs and world-model rollouts fall short on long-horizon spatial grounding, and the authors show that finetuning on EgoTL’s human CoT aligned with metric labels measurably improves planning, step-wise reasoning, instruction following, and spatial grounding.

Key claims

Existing egocentric supervision relies heavily on VLM auto-labeling, which is noisy on action labels, CoT, and spatial annotations, and the errors compound over long-horizon spatial instruction following [Abstract].
A “say-before-act” think-aloud protocol with word-level timestamps yields step-by-step human CoT aligned to the egocentric stream, complemented by metric-scale spatial estimators, a memory-bank walkthrough for scene context, and clip-level tags for navigation and manipulation [Abstract].
EgoTL-Bench decomposes egocentric spatial understanding into six tasks across three layers — memory-conditioned planning, scene-aware action reasoning, next-action prediction, action recognition, direction recognition, and distance estimation — with MCQ accuracy and mean relative accuracy (MRA) for distance estimation [EgoTL-Bench Overview].
World models are evaluated on 60-second CoT-conditioned rollouts with CLIP-Score and VBench; current models produce locally photorealistic frames but drift in object size/position, occlusions, and room connectivity, violating basic spatial constraints over long horizons [Long-Horizon Video Generation].
Finetuning foundation models with EgoTL human CoT aligned to metric labels on the training split improves long-horizon planning and reasoning, step-wise reasoning, instruction following, and spatial grounding [Abstract].

Method

EgoTL is a data-collection-plus-benchmark pipeline. The capture side uses a say-before-act protocol where the wearer narrates the abstract task, the step-by-step plan, and ongoing reasoning while performing it, producing word-level-timestamped CoT aligned with the egocentric video. Physical properties (distances, object positions) are calibrated with metric-scale spatial estimators, a memory-bank walkthrough provides scene context for memory-conditioned planning, and the stream is segmented into clip-level tags for navigation primitives (walking, turning, sitting/standing) and detailed manipulation actions.

EgoTL-Bench then exposes six evaluation tasks: (1) memory-conditioned route planning given a memory-bank walkthrough and a high-level goal; (2) scene-aware action reasoning in cluttered environments (e.g., move an obstacle before opening a door); (3) next-action prediction from current frame plus abstract task; (4) action recognition for the ongoing manipulation; (5) direction recognition for egocentric motion primitives; (6) distance estimation in meters with MRA. World models are additionally benchmarked on 60-second CoT-conditioned video rollouts using CLIP-Score and VBench. The same human-CoT-with-metric-labels split is used to finetune foundation models, demonstrating downstream gains.

Results

Multiple-choice VLM tasks are reported with accuracy; distance estimation uses mean relative accuracy (MRA); world-model rollouts use CLIP-Score and VBench [EgoTL-Bench Overview].
Two chance-level baselines are reported alongside open-source and proprietary VLMs plus an EgoTL-finetuned model; current frontier foundation models “still lag in robust long-horizon spatial grounding” while EgoTL finetuning improves planning, step-wise reasoning, and rollout consistency [EgoTL-Bench Overview].
The website teaser shows GPT failing on five of six showcased tasks (memory route planning, action reasoning under occlusion, next-action prediction, action recognition, direction recognition) and underestimating walking distance (0.95 m vs. 3.98 m ground truth) [Compare your spatial intelligence abilities with latest GPT].
Qualitatively, world-model rollouts drift on long horizons — object size/position changes, inconsistent occlusions, camera jumping between rooms — even when individual frames remain photorealistic, signaling a missing explicit egocentric spatial backbone [Long-Horizon Video Generation].

Why it’s interesting

This is the third egocentric dataset filed in two weeks alongside EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World and Egocentric-1M — largest egocentric video dataset (Build AI / Eddy Xu announcement) / Build AI's early egocentric release — 400k action labels, 2.5k clips, 2× open-source dataset size (Eddy Xu tweet, Oct 22 2025) — but where EgoVerse and Egocentric-1M optimize for raw scale of policy-grade annotations, EgoTL is the first to bake spoken human CoT with word-level timestamps directly into capture, which is exactly the supervision missing from VLM auto-labeling pipelines. It also doubles as a hard egocentric benchmark: the world-model rollout failures it documents (object drift, room teleportation, occlusion inconsistency) overlap with the spatial-supersensing failures catalogued in Cambrian-S: Towards Spatial Supersensing in Video and the long-context spatial-memory gaps targeted by RELIC: Interactive Video World Models with Long-Horizon Memory, so the benchmark side gives the field a shared yardstick for the world-model camp’s persistent claim that minute-scale generation is “almost there.”