Object Concepts Emerge from Motion

TL;DR

A biologically-motivated self-supervised recipe for object-centric visual representations: use off-the-shelf optical flow (VideoFlow) plus a trivial BFS clustering on flow-similarity to derive pseudo instance masks from raw uncalibrated video, then train an image encoder with a margin-based contrastive loss that pulls together features inside a pseudo-instance and pushes apart features across instances (ignoring background-vs-background pairs). Pretrained on ~48M frames from OpenDV-YouTube + nuPlan (2,700 h of driving video), Swin-T/S/B/L and ResNet-50 backbones beat ImageNet-22K supervised, Semantic-SAM, SimMIM, MoCo v3, and DINOv2-scale-matched baselines on monocular depth (KITTI), 3D detection (nuScenes BEVFormer V2), and 3D occupancy. Features are complementary to DINOv2 — concatenating them gives the best depth numbers, indicating motion-derived “semantic instance” features capture a level of abstraction DINO/CLIP/MAE miss.

Key claims

Existing vision foundation models (DINO, CLIP, MAE) cluster features by semantic category but not by semantic instance; PCA visualizations show the proposed model produces distinct, coherent per-object embeddings that the baselines do not [Fig. 1, §1].
Under rigid-scene + pinhole-camera assumptions, optical-flow gradients are a function of depth gradient + camera motion + intrinsics, so flow discontinuities are reliable proxies for object boundaries — including for stationary foreground (trees, signs) under ego-motion, not only moving objects [§3.1, Eqs. 1–3].
Pseudo-instance generation requires only optical flow + forward-backward consistency check + BFS clustering with two thresholds (Δflow=1.0, min-pixels=200); no camera calibration, no human labels, scaling to 2,700 hours / 48M frames [§3.2].
The contrastive loss treats the largest pseudo-cluster as background and ignores background–background pixel pairs; this is the mechanism that handles the under-segmentation inherent in flow-based labels [§3.3, Eq. 4–5].
Monocular depth (KITTI Eigen split, Swin-T): RMSE 2.016 / δ<1.25 0.975, beating Semantic-SAM (2.169/0.971) and ImageNet-22K (2.182/0.969) at the same backbone — Swin-T matches Swin-L with ImageNet-22K pretraining [Table 1].
Monocular depth (KITTI online leaderboard, Swin-L + DCDepth decoder): SILog 9.54, outperforming Semantic-SAM-pretrained DCDepth (9.60), NDDepth (9.62), VA-DepthNet (9.63), and NeW CRFs (10.39) on the primary metric [Table 2].
Features are complementary to DINO: concat(Ours, DINO) gives lowest SILog 6.796 and competitive RMSE 2.014, beating either alone — interpretation is that DINO encodes semantic category while motion-supervised features encode instance structure [Table 6, §4.3].
3D detection (nuScenes, BEVFormer V2): Swin-L attains NDS 55.80 / mAP 47.29, beating ImageNet-22K-Swin-L (54.59/45.91) and SimMIM-Swin-L (54.98/46.52); ResNet-50 also beats COCO + ImageNet-1K + MoCo v3 at the same backbone [Table 3].
At 800×450 resolution, Swin-L attains NDS 52.03 / mAP 41.79 — comparable to DINOv2 ViT-L (51.91 / 42.05) with significantly fewer parameters and lower FLOPs [Table 4, §4.4].
Generalization to unseen indoor scenes despite outdoor-only driving pretraining — argued (qualitatively) to indicate the features encode object composition and structure rather than memorized appearance [§1, §4.4 conclusion].

Method

Pretraining data is the union of OpenDV-YouTube (1,747 h, front-view driving across 244 cities) and nuPlan (1,200 h, 8-camera driving in 4 cities), sampled at 0.3-s intervals. VideoFlow (a 5-frame-in / 3-flow-out off-the-shelf model) estimates optical flow over each 1-second window. A BFS pixel-clustering algorithm with thresholds (Δflow=1.0, min-cluster-size=200) and a forward-backward consistency check yields pseudo-instance masks; only frames with ≥2 clusters (i.e. ≥1 foreground instance) are kept, producing 48M labeled images.

The network is a backbone (Swin-T/S/B/L or ResNet-50) → FPN → semantic-segmentation-style fused output head, producing a stride-4 feature map of channel-64 features that are then 2× upsampled and L2-normalized. The training loss is a margin-based contrastive loss between sampled pixel pairs: same-instance pairs are pulled to high similarity, different-instance pairs are pushed below a margin m=0.5, and background-vs-background pairs are masked out (since the BFS labels chronically under-segment background into a single mega-cluster). 200 pixels are sampled per image, batch size 2048, 200 epochs of cosine-decay AdamW + 10 warmup at LR 1e-3, plus a 20-epoch fine-tune with two random crops per image and a cross-crop loss term. Downstream tasks plug in only the pretrained backbone weights.

Results

KITTI monocular depth (Eigen split): Swin-L attains RMSE 1.929, SILog 6.558, δ<1.25 0.981 — best in Table 1, beating SimMIM Swin-L (1.941 / 6.542 / 0.979) on RMSE/accuracy and roughly matching on SILog [Table 1]. Combined with DINO features (concat after the backbones) the SILog drops to 6.796 with RMSE 2.014 [Table 6]. KITTI online leaderboard: SILog 9.54 with Swin-L + DCDepth, top on the primary metric [Table 2]. nuScenes 3D detection at full resolution: Swin-L NDS 55.80 / mAP 47.29 (best across backbones tried); ResNet-50 NDS 52.55 / mAP 43.22, beating COCO-supervised + MoCo-v3 + ImageNet-1K at the same backbone [Table 3]. At 800×450: Swin-L NDS 52.03, comparable to DINOv2-ViT-L (51.91) at fewer params [Table 4]. The model also generalizes from outdoor pretraining to unseen indoor scenes (qualitative similarity maps, §4.2).

Why it’s interesting

This paper is a clean empirical pressure-test of an argument that’s currently being debated inside the wiki’s World Foundation Models cluster: The flavor of the bitter lesson for computer vision argues that motion / generative-rollout objectives should replace hand-crafted CV intermediates as the perception pre-training substrate; this paper takes the strongest non-generative interpretation of that thesis — “use motion as a free supervisory signal for object-level grouping” — and shows it beats ImageNet-22K, SimMIM, Semantic-SAM, and (parameter-matched) DINOv2 on three downstream driving tasks. The complementarity-with-DINO finding is the most interesting result: it’s evidence that the “semantic category” axis (CLIP/DINO) and the “semantic instance” axis (this work) are genuinely orthogonal levels of abstraction, and a Luma-relevant signal that perception backbones for embodied / world-model stacks may want both. The contrast with V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning is the natural other comparison: V-JEPA 2.1 also uses video, also targets dense features, and also competes with DINOv3 — but its supervisory signal is masked-latent prediction across time, whereas this paper extracts a discrete pseudo-instance label from optical flow and trains contrastively on it. Both lines land at “video-derived self-supervision matches or beats image-only SSL on dense tasks”, from opposite ends of the dense-versus-discrete spectrum. The biggest open question this paper does not address is whether the BFS-on-flow recipe scales to non-driving video (crowds, indoor handheld, social) where rigid-scene + dominant-ego-motion assumptions break — a gap that masks-from-video methods (V-JEPA’s mask-denoise, Action100M’s HowTo100M-derived motion segments via Action100M: A Large-scale Video Action Dataset) avoid by not relying on flow at all.