Back to the Features: DINO as a Foundation for Video World Models

TL;DR

DINO-world is a generalist video world model from FAIR (LeCun, Bojanowski et al.) that predicts future frames in the latent space of a frozen DINOv2 ViT-B/14 encoder rather than in pixel space. A causal cross-attention transformer predictor (up to ViT-g) is trained on >60M uncurated web videos with a next-frame smooth-L1 objective and variable-FPS sampling, then optionally post-trained with action blocks for planning. The thesis: decoupling visual representation learning (frozen DINOv2) from temporal dynamics modeling sidesteps the cost of pixel-space generation while inheriting strong semantic features. Reports SOTA segmentation/depth forecasting (VSPW mid-term 47.0 mIoU vs 40.7 for COSMOS-12B and 40.2 for COSMOS-4B), competitive intuitive-physics performance vs V-JEPA at smaller scale, and superior planning over training-from-scratch baselines on three simulated RL environments.

Key claims

Latent-space world modeling on a frozen DINOv2 encoder beats both pixel-space generative WFMs (COSMOS 4B/12B) and joint-trained latent WFMs (V-JEPA) on dense forecasting — DINO-world ViT-B 47.0 VSPW mid-term mIoU vs 40.7 COSMOS-12B / 4.6 V-JEPA ViT-H [§4.1, Table 1].
Decoupling representation from dynamics is the core architectural commitment: the frame encoder is frozen DINOv2 ViT-B/14 with registers; only the cross-attention predictor is trained, which the authors argue is a separation-of-concerns that V-JEPA’s joint-training scheme sacrifices [§3.1, §2.2].
The predictor is a cross-attention decoder (not a self-attention transformer over patch tokens): a learned query for each target spatio-temporal coordinate cross-attends to all past patch tokens, with attention masked block-triangularly to enforce causality across frames [§3.2, Eq. 2-3].
3-axis split-RoPE (temporal seconds, horizontal grid position, vertical grid position) on separate head-dimension thirds lets the model handle variable input resolutions (relative spatial coords) and variable FPS (absolute time in seconds) at train and inference time [§3.2 Positional encoding].
Training samples time-deltas uniformly from a predefined range rather than taking contiguous frames, so the predictor sees an unbiased distribution of prediction horizons rather than collapsing onto short intervals [§3.2 Variable FPS].
Predicting at every token position (smooth-L1 against DINOv2 features) is cheaper-per-loss-term than masked-reconstruction objectives like V-JEPA / DINO-Foresight, which only compute loss on a small fraction of mask tokens [§3.2 Training objective].
Intuitive-physics: DINO-world ViT-B matches V-JEPA ViT-H on IntPhys / GRASP / InfLevel surprise-score benchmarks despite using a smaller frame encoder; COSMOS scores near-perfect on simpler IntPhys but lags on GRASP/InfLevel [§4.1 Intuitive physics, Table 2].
Action-conditioned post-training adds an action block after each cross-attention block that updates the query token with the corresponding action; can be initialized as identity, trained on a small action-labeled dataset, and optionally with the base predictor frozen — avoiding interleaving action tokens with patch tokens [§3.3].
Pre-training on uncurated web videos at scale gives a measurable advantage over training the same predictor from scratch on action-labeled trajectories for downstream planning on simulated RL environments [§4.3].
Training resource is dramatically lower than pixel-space WFMs: 1B parameters total vs 22M GPU-hours / 12B parameters for COSMOS, with the authors framing this as the practical payoff of dropping the pixel decoder [§3.1 Frame encoder].

Method

DINO-world separates the world model into two stages. Frame encoder: each video frame is independently encoded by a frozen DINOv2 ViT-B/14 with registers — the same checkpoint used by DINO-Foresight — producing a feature tensor at the last layer. Patch tokens at spatio-temporal coordinates (t, h, w) are the state the predictor learns to predict. Predictor: a stack of cross-attention pre-norm transformer blocks (up to ViT-g scale: 40 blocks, 1408 dim, 16 heads). For each future coordinate (t, h, w), a learned query embedding is initialized and cross-attends to all past patch tokens (key-value pairs), followed by an MLP. A final linear projection emits the predicted patch token in DINOv2 latent space. RoPE in three axes: head dimension is split in thirds for (t, h, w); spatial uses relative grid positions (resolution-invariant), temporal uses absolute seconds (FPS-invariant). Loss: smooth-L1 against the DINOv2 target features, computed at every query position, with block-triangular masking so queries at frame t only attend to context up to t. Training data: 60M+ uncurated web video clips, 5–60s, mixed FPS; sample time-deltas uniformly from a fixed range rather than contiguous frames so the model sees a uniform distribution of horizons.

For action conditioning, after each cross-attention block an additional action block updates the query token using the action a_t. These blocks can be initialized as identity, trained on observation-action trajectories from RL environments, and the base predictor can be kept frozen to avoid catastrophic forgetting of the unconditional world model.

For evaluation on dense forecasting (VSPW, Cityscapes, KITTI), a “present-time” linear segmentation/depth head is trained once on raw DINOv2 features; at inference, the predictor rolls forward k frames and the same linear head is applied to the predicted features. This isolates the quality of the predicted features themselves from any decoder learning.

Results

Dense forecasting (Table 1): VSPW mIoU — short-term: DINO-world 51.6, COSMOS-4B 46.6, COSMOS-12B 46.6, DINO-Foresight 44.7, V-JEPA ViT-H 4.9, Copy-Last 47.9; mid-term: DINO-world 47.0, COSMOS-12B 40.7, COSMOS-4B 40.2, DINO-Foresight 37.7, V-JEPA ViT-H 4.6, Copy-Last 42.1. Cityscapes mid-term mIoU: DINO-Foresight 57.2 (domain-trained on driving), DINO-world 55.1, COSMOS-12B 45.9. KITTI depth RMSE: DINO-world short-term 3.214, DINO-Foresight short-term 3.562, COSMOS-12B 4.157 — lower is better.

Intuitive physics (Table 2): DINO-world ViT-B comparable to V-JEPA ViT-H across IntPhys / GRASP / InfLevel. COSMOS scores ~perfect on IntPhys but is weaker on the other two. DINO-Foresight underperforms on synthetic-domain physics benchmarks (its training domain is real driving). All large-scale WFMs reach adequate but bounded physics performance.

Planning: action-conditioned fine-tuning of the pretrained DINO-world predictor on three simulated RL environments outperforms training the same predictor from scratch on the same action data — the value of large-scale unconditional video pre-training is preserved through the action-conditioning step.

Headline V-JEPA contrast: V-JEPA’s predictor was designed as a head to train the encoder, not for accurate forecasting; on dense-forecasting and physics benchmarks V-JEPA ViT-H scores 4.6 VSPW mid-term mIoU and 7.491 KITTI mid-term RMSE — i.e. far worse than even Copy-Last [Table 1]. DINO-world’s separation of encoder (frozen DINOv2) from predictor (a real forecasting model) directly addresses this.

Why it’s interesting

This is the cleanest empirical statement on file of the DINO-as-foundation thesis for world models — the natural counterpart to the V-JEPA line tracked in World Foundation Models and V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning. Where V-JEPA 2/2.1 jointly trains a video-aware encoder and a predictor, DINO-world freezes the strongest still-image SSL encoder and concentrates all dynamics learning in a separate cross-attention predictor — and on dense forecasting at least, decoupled-and-frozen wins. This complements V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning by attacking the prediction quality dimension (where DINO-world dominates) while V-JEPA 2.1 attacks the representation density dimension (where it now matches DINOv2 / DINOv3); the two papers together suggest the right WFM stack is “frozen vision-foundation encoder + dedicated cross-attention predictor”, with the encoder line still up for grabs. The pixel-space comparison vs COSMOS 4B/12B is the sharpest cost-of-pixels datapoint in the cluster: COSMOS uses 22M GPU-hours and emits pixels that, when re-encoded by DINOv2 for the dense-forecasting probe, lose to a 1B latent-only predictor. The action-conditioned post-training story (add identity-initialized action blocks after each cross-attention block, optionally freeze the base) is also notable: it’s a much cleaner separation than DINO-WM’s action-token interleaving, and recalls the structural choice in V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning‘s V-JEPA 2-AC variant. For Luma the practical upshot is that DINOv2 + a forecasting head trained on uncurated web video may be a competitive recipe for video understanding and planning tasks at far lower training cost than pixel-space generation. Also worth noting: the author list (LeCun, Bojanowski, Massa, Khalidov) overlaps heavily with both the DINOv2 paper and the V-JEPA line — this is FAIR explicitly stress-testing the frozen alternative to V-JEPA’s joint scheme.