OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modeling

TL;DR

OmniWorld is an ICLR 2026 dataset for 4D world modeling that combines a newly collected synthetic OmniWorld-Game subset (96K sequences, 18.5M frames, 1280×720, from game engines) with eleven curated public corpora spanning simulator, robot, human-egocentric, and internet-video domains. Each clip is (re-)annotated for a uniform stack of modalities — metric depth, camera pose, text caption, optical flow, foreground masks — using specialized per-modality pipelines. OmniWorld-Game alone surpasses prior synthetic 4D datasets (Sintel, TartanAir, PointOdyssey, Dynamic Replica, SeKai-Game) in both modal coverage and frame scale. The paper also establishes a benchmark exposing SOTA limitations and shows that fine-tuning on OmniWorld yields gains on both 4D reconstruction and video generation.

Key claims

4D world modeling is fundamentally data-bound: existing datasets lack dynamic complexity, multi-domain diversity, and the joint spatial-temporal annotations needed for 4D geometric reconstruction, future prediction, and camera-controlled video generation [§1].
OmniWorld-Game is a synthetic subset of 96K clips at 1280×720, 24 fps, totaling 18.5M frames, with fully (re-)annotated depth, camera, text, optical flow, and foreground masks [Tab. 1].
The full OmniWorld corpus integrates eleven public datasets across four domains: simulator (OmniWorld-Game), robot (AgiBot 20K, DROID 35K, RH20T 109K), human (RH20T-Human, HOI4D, Epic-Kitchens, Ego-Exo4D, HoloAssist, Assembly101, EgoDex), and internet (CityWalk 7K) — with previously-missing modalities filled in by the authors’ annotation pipelines [Tab. 1].
OmniWorld-Game beats prior public synthetic 4D datasets on the joint criterion of modality coverage and frame scale: 18.5M frames with all six tracked annotations, versus e.g. SeKai-Game (4.3M frames, missing depth/optical-flow/masks) and PointOdyssey (216K frames, missing text and masks) [Tab. 2].
Fine-tuning existing SOTA methods on OmniWorld yields measurable gains on 4D geometric reconstruction and camera-controlled video generation tasks, validating the dataset as both training fuel and an evaluation benchmark [Abstract, §5].
A companion benchmark — OmniWorld-Game Benchmark — evaluates 3D geometric prediction and camera-control video generation against the curated 4D annotations [Abstract, project page].

Method

The acquisition pipeline collects raw RGB video from the eleven source domains, applies a video-slicing/filtering stage to retain high-quality dynamic sequences, and then routes each clip through a stack of specialized annotation models: metric depth via a depth estimator, camera poses via a pose-recovery pipeline, foreground masks via a segmenter, optical flow via a flow estimator, and dense text captions via an MLLM. For each source dataset, only the missing modalities are (re-)annotated — modalities that already ship with ground truth (e.g. camera poses in DROID, RH20T, AgiBot) are preserved. OmniWorld-Game itself is rendered from game engines and ships with all six modalities natively. The dataset is released on HuggingFace (InternRobotics/OmniWorld, ~30.7 TB) under the ICLR 2026 acceptance, and the codebase + benchmark are at github.com/yangzhou24/OmniWorld.

Results

The headline result is the joint scale/modality table (Tab. 1): 605K sequences across domains, 273M+ frames, with the simulator subset alone contributing 18.5M frames at full annotation. Against prior synthetic 4D datasets (Tab. 2), OmniWorld-Game is the only entry with the full modality stack (depth + camera + text + optical flow + masks) at frame counts >4M. Fine-tuning experiments report gains on both 4D reconstruction and video generation tasks (quantitative numbers in the paper §5; specific deltas not reproduced here). The HuggingFace dataset card reports 204,902 downloads and 980,300 rows since release.

Why it’s interesting

OmniWorld lands exactly in the supervision-infrastructure gap that the 4D Scene Generation cluster has been flagging: every prior filed paper there (VerseCrafter, NeoVerse, SpaceTimePilot) builds its own ad-hoc auto-annotation stack (MegaSAM+Grounded-SAM2, online degradation simulators, CamTime Blender renders), and DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modeling (DynamicVerse) is the previous attempt to make this step a reusable artifact. OmniWorld pushes that further: it bundles a cross-domain corpus (simulator + robot + ego + internet) with a uniform six-modality stack and a paired benchmark, which DynamicVerse does not. For World Foundation Models, it provides the kind of paired (geometry, dynamics, language) supervision that latent-predictive WFMs (V-JEPA family) and generative-rollout WFMs (Genie 3, LingBot-World) both need but rarely share a corpus over. The auto-annotation recipe also slots into Synthetic Training Data alongside Action100M: A Large-scale Video Action Dataset as another industrial-scale automated-labeling pipeline applied to real video.