Skip to content
Bud's Wiki

Jiahua Dong — 30,000-hour egocentric world model learns hands cleanly, not objects; neither 100× data nor 4× model closes the gap

TL;DR

Section titled “TL;DR”

A 7-tweet research thread from Jiahua Dong reporting on a world model trained on 30,000 hours of first-person manipulation video from Mecka AI (1.15M clips, 32k tasks, 14k contributors). Headline finding: the model learns hand motion cleanly but object dynamics stay much worse — and the gap does not close under 100× data (300 → 30,000 h) or 4× model (Cosmos 3 Nano 16B → Super 64B); closing it at the measured rate would take ~4 orders of magnitude more compute. A modified training signal that adds extra noise in moving regions raises the object ceiling but object fidelity still trails the agent’s. Pre-training on 30k hours of human video beats the released Cosmos 3 across all metrics on downstream humanoid transfer, with the extra training-signal changes stacking further gains; against a simulator, the world model usually gets the outcome right. Conclusion: contacts, forces, and hidden geometry are never visible in pixels, exactly what simulators represent well — combining the two may be what’s needed. Code, checkpoints, and OOD benchmark coming soon.

Key claims

Section titled “Key claims”
  • World model trained on 30,000 h of first-person manipulation video learns where hands go precisely, but what happens to the things they touch remains much worse [Tweet 1/7].
  • Scaling data 300 h → 30,000 h (100×) does not narrow the hand–object fidelity gap [Tweet 1/7, 2/7].
  • Scaling model from Cosmos 3 Nano (16B) to Super (64B) improves object interaction only barely; closing the remaining gap at the observed rate would take four orders of magnitude more compute [Tweet 3/7].
  • Giving the model the 3D hand pose at input causes hands to saturate with a fraction of the data otherwise required [Tweet 2/7].
  • Changing the training signal — adding extra noise where things move so the model cannot copy pixels and must predict their motion — raises the object-fidelity ceiling, but object fidelity still stays far below the agent’s [Tweet 4/7].
  • Pre-training on 30k h of human video beats the released Cosmos 3 across all metrics on downstream humanoid transfer; the proposed training-signal changes stack further improvements [Tweet 6/7].
  • Against a simulator, the world model usually gets the outcome correct (qualitative demo) [Tweet 6/7].
  • Author’s structural explanation: “A video model learns what it can see. But how an object responds depends on what it can’t: mass, friction, stiffness, and forces hidden under the hand” [Tweet 5/7]. Failure modes shown — spoons stay on the plate, rigid boards stretch [Tweet 5/7].
  • Takeaway stated in-thread: egocentric video scaling helps and transfers to robots, but contacts, forces, and hidden geometry are precisely what simulators represent well — combining the two may be what’s needed [Tweet 7/7].

Method

Section titled “Method”

A video world model is trained on Mecka AI’s egocentric manipulation corpus — 30,000 hours of first-person video, 1.15M clips, 32k tasks, 14k contributors [Tweet 2/7]. The study runs two scaling axes against each other: data (300 h → 30,000 h) and model (Cosmos 3 Nano 16B → Cosmos 3 Super 64B). A separate ablation feeds 3D hand pose as input and measures how fast hand prediction saturates. A third intervention changes the training signal by adding extra noise in moving regions so the model cannot shortcut via pixel-copying. Downstream, the pretrained model is evaluated as a world prior for humanoid transfer against the released Cosmos 3 across all metrics, and against a simulator as outcome reference. No paper, checkpoints, or OOD benchmark linked at filing time — the thread announces them as forthcoming.

Results

Section titled “Results”
  • Data scaling (fixed architecture): hand prediction improves on both axes with more data, object prediction improves but the hand-object gap does not close across 300 → 30,000 h [Tweet 2/7].
  • Model scaling (fixed data): Nano 16B → Super 64B closes a small fraction of the object gap; extrapolating the measured rate would take ≈10⁴× more compute [Tweet 3/7].
  • Hand-pose input ablation: hands saturate “with a fraction” of the data otherwise required [Tweet 2/7].
  • Modified training signal: raises the object-fidelity ceiling but does not close the gap to hand fidelity [Tweet 4/7].
  • Downstream humanoid transfer: 30k-h pretraining beats released Cosmos 3 across all reported metrics; proposed training-signal changes stack further gains [Tweet 6/7].
  • Qualitative failure modes called out: spoons stay on the plate when they should fall; rigid boards stretch rather than stay rigid [Tweet 5/7].

(No numeric tables disclosed in the thread.)

Why it’s interesting

Section titled “Why it’s interesting”

Directly updates two debates on the wiki simultaneously. First, it is the sharpest filed single-paper evidence that pixel-rollout WFMs trained on egocentric video alone hit a structural ceiling on the thing the WFM cluster most wants — contact dynamics — and that the ceiling is not a scale-headroom problem: 100× data and 4× params both bend the curve only slightly. This lands against the Sitzmann “generative-rollout should dissolve explicit 3D / simulators” position in World Foundation Models and pushes the cluster’s open “action-conditional WFM chicken-and-egg” question toward a hybrid answer the author names explicitly (video pretraining + simulator for the invisible channels). Second, the downstream result that 30k h of human video beats the released Cosmos 3 on humanoid transfer is the same direction as HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining HumanScale and Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models Dyna-2’s cross-embodiment scaling on YAM — egocentric video is a scalable pretraining substrate for robots — but adds the first filed public datapoint pitting a scaled human-video WM against a released frontier WFM (Cosmos 3) as the baseline rather than a VLA or an internal control. The 3D-hand-pose-input saturation echoes the EgoScale / Dyna-2 finding that the channel chosen matters, not just the hour count. Underlying substrate (Mecka) overlaps with the EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World EgoVerse consortium that already lists Mecka as a contributor.

See also

Section titled “See also”