SpatialVID: A Large-Scale Video Dataset with Spatial Annotations

TL;DR

SpatialVID is a 7,089-hour corpus of in-the-wild dynamic YouTube videos cut into 2.71M clips, each annotated with per-frame camera pose, depth maps, dynamic-region masks, structured captions, and a derived sequence of cinematographic motion instructions (dolly in, pan left, truck right, …). The raw collection — 33,443 manually screened videos totaling 21,789 hours — was funneled through a hierarchical filter (PySceneDetect cuts, CLIP+MLP aesthetics, luminance, OCR text, motion intensity) before geometric annotation via an enhanced MegaSaM pipeline (Depth-Anything-v2 + UniDepth-v2 + SAM2-refined dynamic masks). Authors also release SpatialVID-HQ, a balanced 1,146-hour / 0.37M-clip subset. CVPR 2026.

Key claims

The collection mixes real-world dynamic content (people, vehicles, animals) with rich camera parallax — 2.71M clips / 127.6M frames vs the next-largest comparable real dynamic dataset (CamVid-30K at 30K clips, DynPose100K at 100K) [Table 1, §1].
Manual screening at the raw video level (not per-clip) is the load-bearing curation step: excluded categories include dominant-foreground-motion footage, static viewpoints, zoom-only transformations, severe blur/illumination, wide-angle distortion, and intrusive overlays [§3.1.1].
MegaSaM is empirically the best accuracy/time trade-off for in-the-wild camera-pose annotation: more accurate than DROID-SLAM/COLMAP/Fast3R, faster than MonST3R at similar accuracy, and more robust than VGGT in low-feature-point regimes [§3.3, Fig. 6].
Swapping MegaSaM’s internal depth model for UniDepth-v2 + Depth-Anything-v2 improves pose-annotation reliability [§3.3].
An acceleration-based physical-plausibility check, plus three new motion-trajectory metrics — MoveDist (path length), RotAngle (cumulative rotation), TrajTurns (turn count) — are used to filter physically implausible trajectories and to balance the HQ subset [§3.3].
Motion-probability maps from MegaSaM are too coarse for dynamic-object masking; the pipeline refines them with adaptive thresholding + contour detection + 4-anchor-point SAM2 prompts, then computes a dynamic-ratio metric per clip [§3.3].
Camera trajectories are mapped to a controlled cinematographic vocabulary via thresholded relative pose changes between adjacent frames, yielding human-readable motion instructions (dolly in, pan left, truck right, …) [§3.4, Fig. 7].
Even with aggressive filtering, existing high-quality video corpora (Panda70M, MiraData) yielded only ~10% pass-through on the SpatialVID pipeline — motivating fresh collection from YouTube via motion-keyword queries (walk, tour, drone) [§3.1.1].

Method

A three-stage pipeline operates on raw YouTube video. Stage 1 — Collection & preprocessing: 33,443 manually screened videos (21,789 h) are cut by a modified PySceneDetect (interval-based multi-frame comparison) into 7M+ clips of 3–15s, transcoded to a uniform 1920×1080 H.265 baseline. Stage 2 — Filtering: each clip is scored on aesthetic quality (CLIP+MLP, threshold 4.0), luminance (range [20, 140]), OCR text ratio (threshold 30%), and motion intensity, scored on first/middle/last frames. Stage 3 — Annotation: an enhanced MegaSaM pipeline produces per-frame camera intrinsics, extrinsics, and depth maps. UniDepth-v2 + Depth-Anything-v2 replace the default monocular depth backbone. SAM2 refines motion-probability maps into clean dynamic masks via 4 perimeter-anchor prompts per contour. The acceleration-based check + MoveDist/RotAngle/TrajTurns triple is used as a trajectory sanity gate and as a balanced-sampling axis for SpatialVID-HQ. Motion instructions are derived from thresholded relative-pose deltas after temporal smoothing. Structured captions integrate scene description + camera-motion summary + hierarchical semantic labels (weather, lighting, time of day).

Results

The full SpatialVID release is 2.71M clips / 127.60M frames at 720P+; SpatialVID-HQ is 0.37M clips / 20.63M frames. Table 1 positions it against 15 prior datasets: it is the only entry in the “real + dynamic + camera + depth + structured captions” cell, and at ~2.71M clips it is ~10× DynPose100K, ~27× Stereo4D-style fisheye datasets, and ~90× Princeton365 in clip count. Annotation pipeline ablations on internal benchmarks justify MegaSaM over DROID-SLAM, COLMAP, Fast3R, MonST3R, and VGGT [§3.3, Fig. 6]. No downstream-training numbers (e.g. a camera-controlled video generator finetuned on SpatialVID) are reported in this version — the contribution is the dataset.

Why it’s interesting

SpatialVID is supervision infrastructure for the Camera-Controlled Video Diffusion cluster — every paper in that concept (SpaceTimePilot, NeoVerse, VerseCrafter, OmniTransfer, Kling-MotionControl) explicitly cites the camera-pose-annotated data bottleneck and builds an ad-hoc auto-annotation stack to work around it. SpatialVID promotes that step to a reusable, much larger artifact than the prior Sept-2025 baselines: ~2.71M dynamic clips with both camera pose and depth and structured captions and motion instructions. It is the real-data analogue of SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time‘s synthetic CamTime grid, and it complements DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modeling (DynamicVerse, also auto-annotated metric-depth + camera + moving-object + captions, but smaller and project-page-only). Compared to Sekai: A Video Dataset towards World Exploration (Sekai, 5,000 h, world-exploration framing) SpatialVID is ~1.4× larger, dynamic-scene-focused rather than first-person walking, and the only one of the three that ships cinematographic motion-instruction labels directly usable as Hunyuan-GameCraft-style training signal (Hunyuan-GameCraft: High-dynamic Interactive Game Video Generation with Hybrid History Condition).