Sekai: A Video Dataset towards World Exploration

TL;DR

Sekai (せかい, “world”) is a 5,000+ hour egocentric video dataset built specifically for training interactive world-exploration models. It consists of two parts: Sekai-Real (~5,000 hours of walking and drone-view YouTube footage from 101 countries and 750+ cities, audio preserved on walking clips) and Sekai-Game (60 hours of recorded Lushfoil Photography Sim gameplay with ground-truth annotations from a Lua/UE5 capture toolchain). Every clip carries six annotation channels — location, scene type, weather, crowd density, captions (~200 tokens), and camera trajectories — produced by a curation pipeline that combines TransNetV2 shot detection, COVER quality filtering, Qwen2.5-VL-72B captioning, and a MegaSaM SLAM variant swapped with Video Depth Anything. The authors also release a 400-hour curated “best-of-the-best” subset (Sekai-Real-HQ) and use it to train YUME (ゆめ, “dream”), an image-conditioned interactive video model controlled by keyboard and mouse. NeurIPS 2025.

Key claims

Sekai-Real contains 5,000+ hours of YouTube walking and drone clips from 101 countries and 750+ cities, 720p/30 fps, with audio retained on walking videos [§1, §3].
Sekai-Game contains 60 hours of Lushfoil Photography Sim (UE5) capture with ground-truth camera poses, location, weather, and time-of-day, obtained via an RE-UE4SS Lua toolchain that hooks Unreal’s object system at runtime [§3.4].
All walking clips are ≥60 s; average duration is 2 minutes with a maximum of 39 minutes — longer than OpenVid-1M’s <25 s average and MiraData’s 72.1 s average [§2.2, §1].
Caption length averages over 176 words per clip and over 200 tokens — significantly above OpenVid-1M’s ~130 tokens — produced by a two-stage Qwen2.5-VL-72B pipeline that first labels four orthogonal categories (scene/weather/time/crowd) then generates time-ordered descriptions [§3.3, §4].
Camera trajectories are annotated for ~600 hours after sampling using MegaSaM with two swaps: Depth Anything → Video Depth Anything (better temporal consistency) and UniDepth → DepthPro for the metric-depth stage; the authors also add multi-GPU parallel inference to the official MegaSaM implementation [§3.3].
The curation pipeline produces 6,620 h of YouTube clips from 8,623 h of raw input, with five filter stages: GPU-accelerated TransNetV2 (5× faster), luminance filtering (removes 300 h), COVER technical-quality filtering (drops bottom 10%), VideoSubFinder subtitle filtering (drops ~5%), and SfM-based camera-trajectory sanity checks [§3.2].
A 400-hour Sekai-Real-HQ subset is built via four-stage diversity sampling — quality (COVER), content (InternVideo2 + mini-batch K-Means per country), location (inverse-frequency over 750 cities), category (joint inverse-probability over weather × scene × time × crowd), and camera trajectory (direction-vector × jitter bins) — to mitigate the long-tail bias [§3.5, Fig. 5].
YUME is trained on a subset of Sekai-Real-HQ as an interactive image-to-video world-exploration model with keyboard+mouse control [§5].

Method

Sekai’s contribution is the curation pipeline and the dataset itself, not a generative model. The pipeline has four stages — collection, pre-processing, annotation, sampling — applied separately to YouTube (Sekai-Real) and game footage (Sekai-Game), with the game pipeline differing primarily in that ground-truth annotations are recorded at capture time rather than inferred post-hoc.

For Sekai-Real, video URLs are hand-collected from popular walking/drone YouTubers and extended via keyword search (walk, drone, HDR, 4K). After downloading 1080p+ raw video, the pipeline runs shot detection (TransNetV2 at threshold 0.4, refactored for PyNVideoCodec + CVCUDA GPU acceleration), splits shots into one-minute clips, re-encodes to 720p/30 fps H.265 at 4 Mbps, mutes the audio for drone clips but preserves it (AAC, 48 kHz, FFmpeg-trimmed) on walking clips, then filters by luminance, COVER quality, subtitle detection, and SfM trajectory plausibility. Annotations are bolted on via: GPT-4o extracting per-chapter location from YouTube title+description with ISO 3166 codes (matched to clips via interval tree, ~8% discarded for ambiguous matches); Qwen2.5-VL-72B serving the two-stage caption pipeline over vLLM behind an Nginx load balancer; and a modified MegaSaM stack for camera trajectories.

For Sekai-Game, the authors use Lushfoil Photography Sim as a UE5-based “realistic synthetic” source. RE-UE4SS hooks the Unreal object system at runtime to capture ground-truth camera poses synchronized with OBS Studio’s video recording, standardizing camera configuration and hiding the GUI. Captured poses are then calibrated to compensate for capture delays and interpolated to per-frame alignment.

The final dataset hierarchy is: Sekai-Real (5,000+ h, hand-annotated subset has camera trajectories) → Sekai-Real-HQ (400 h, four-stage diversity-sampled “best of the best”) → YUME training subset. Both walking and drone views are kept throughout.

Results

Scale: 5,000+ hours real + 60 hours game; 101 countries / 750+ cities; top 8 countries (Japan, US, UK, …) account for ~60% of duration; long-tail distribution explicitly motivates the HQ sub-sampling [§4, Fig. 4].
Sekai-Real-HQ has higher mean quality and lower variance than Sekai-Real on COVER scores, and a more uniform country distribution [§4, Fig. 5a–b].
Caption tokens: Sekai-Real averages over 200 tokens/clip vs. OpenVid-1M ~130 [§4, Fig. 5c].
Camera-trajectory jitter distribution is “smoother” on Sekai-Real-HQ than on a pre-Trajectory-Diversity 500-hour subset [§4, Fig. 5d].
YUME (the demonstrator model) receives an image and supports unrestricted exploration via keyboard/mouse — the paper shows examples (Fig. 6) but does not report quantitative VBench / FVD / WorldScore numbers against ReCamMaster, TrajectoryCrafter, or Voyager. The authors explicitly note “insufficient training” and “insufficient camera-trajectory annotation” as limitations [§7].
No model-vs-model leaderboard or quantitative head-to-head comparison against Ego4D / EPIC-KITCHENS / Panda-70M / OpenVid-1M / MiraData is included beyond duration and average-caption-length comparisons in §2.2.

Why it’s interesting

Sekai is a purpose-built dataset for the world-exploration framing — egocentric, long-duration, worldwide, with the four annotation channels that camera-controlled and audio-aware video generators actually need. That makes it a candidate pretraining corpus for the “frozen Wan + adapter” cluster catalogued in Camera-Controlled Video Diffusion (NeoVerse, VerseCrafter, SpaceTimePilot all manufacture synthetic supervision because real captures can’t replay a dynamic scene at different camera trajectories — Sekai’s pose annotations sidestep that bottleneck on the real-data side, at least for the ~600-hour SfM-annotated subset). It also complements Egocentric-1M — largest egocentric video dataset (Build AI / Eddy Xu announcement) (Egocentric-1M) on the dataset-axis question: Egocentric-1M is internet-scale factory-floor first-person video with no pose / location / weather annotations; Sekai is two orders of magnitude smaller but carries the structured annotations needed for explicit world-exploration training. The audio-preserved walking clips also make Sekai a candidate pretraining corpus for the joint-audio-video tracks in Joint audio-video generation — most other open video datasets either lack audio entirely or muted it during curation.