V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning

TL;DR

V-JEPA 2.1 (FAIR/Meta + Universidad de Zaragoza) extends the V-JEPA mask-denoising objective so the prediction loss covers all tokens — both masked and visible context — instead of only masked positions. The original V-JEPA loss left context tokens unsupervised, which let them act as register-like global aggregators and made the encoder’s feature maps spatially incoherent on dense tasks. Adding a distance-weighted context loss, hierarchical deep self-supervision at multiple intermediate layers, modality-specific image/video tokenizers, a 163M-image+video corpus, and scaling to ViT-G/2B produces representations that are simultaneously dense, temporally consistent, and globally semantic — SOTA on Ego4D STA (7.71 mAP), EPIC-KITCHENS-100 anticipation (40.8 R@5), NYUv2 depth linear probe (0.307 RMSE), SSv2 (77.7%), with a +20% real-robot grasping success rate over VJEPA-2 AC.

Key claims

The original V-JEPA loss applies only to masked tokens, leaving visible context tokens unsupervised; the encoder learns to use those tokens as global aggregators (register-like) rather than as spatially-grounded local features, which is why V-JEPA 2 dense features are weak (22.2 mIoU on ADE20K, 0.682 RMSE on NYUv2) despite strong global recognition [§2.2, Table 2.3].
Extending the predictive loss to context tokens with a distance-weighted scheme — patch-specific weight 1/√d where d is block-distance to the nearest masked patch — adds local-continuity pressure that recovers dense feature quality (ADE20K 22.2 → 33.8 mIoU, NYU 0.682 → 0.473 RMSE) without destroying global recognition once paired with deep self-supervision [§2.3.1, Table 2.3].
Deep Self-Supervision — applying the prediction + context losses at four encoder layers (3 intermediate + output), fused via a small MLP — recovers global recognition lost to the bare context loss (SSv2 62.5 → 72.1, IN1K 72.6 → 80.8) and improves dense tasks further (ADE20K 33.8 → 38.6, NYU 0.473 → 0.463) [§2.3.2, Table 2.3, Fig. 5].
A multi-modal tokenizer with a 2D conv for images and 3D conv for videos (plus modality-learnable tokens at encoder and predictor input) replaces V-JEPA 2’s hack of treating each image as a duplicated 16-frame static clip — eliminates that compute waste and lifts ADE20K from 40.8 → 41.4 mIoU [§2.3.4, Table 2.3].
Vision-Mix-163M (LVD-142M images + curated 19M YT-1B + SSv2/Kinetics/HowTo100M re-weighted toward motion) replaces VJEPA-2’s 1M-image ImageNet subset and lifts every metric over the previous data mix; SSv2 sampling weight tripled (0.056 → 0.170), YT-1B nearly 4× (0.188 → 0.720) [§2.3.3, Table 2 dataset breakdown].
Scaling from ViT-L 300M to ViT-G 2B + a higher-resolution cool-down phase (videos to 64 frames at 256² and images to 448²) is responsible for the final jump: SSv2 72.6 → 77.7, IN1K 81.6 → 85.5, ADE20K 41.4 → 47.9, NYU 0.415 → 0.307 [§2.3.5, Table 2.3].
V-JEPA 2.1 ViT-G sets new SOTA on Ego4D Short-Term Object-interaction Anticipation (7.71 mAP All) — beating DINOv3 ViT-7B (5.68) and V-JEPA 2 ViT-g (6.02) on the same protocol [Table 3.1].
Frozen V-JEPA 2.1 features drive a +20% real-robot grasping success rate vs VJEPA-2 AC on Franka arms in new environments zero-shot, and SOTA navigation on Tartan Drive (5.687 ATE) at 10× the planning speed of the prior best [§1, §3.3, §3.4].
Dense-prediction state of the art with a linear probe (frozen encoder): NYUv2 RMSE 0.307, ADE20K 47.9 mIoU, Pascal VOC 85.0 mIoU, YouTube-VOS 72.7 J-Mean [§1, §3.5].
Distilled smaller variants (ViT-B 80M, ViT-L 300M) are released alongside the 1B/2B teachers; distillation uses a frozen teacher (no EMA), single-layer loss, 12-block predictor, otherwise identical hyperparameters [§2.3.6, §3.10].

Method

V-JEPA 2.1 keeps the JEPA scaffold — an x-encoder processes a masked clip, an EMA target encoder processes the unmasked clip, and a predictor maps from context+mask-token positions to the target representations — but changes four things. (1) Loss: predict targets at every token position, with an L1 loss on masked tokens (weight 1) and a distance-weighted L1 loss on context tokens (weight 1/√d where d is the spatio-temporal block distance to the nearest masked token, plus a 50–100-epoch warmup of the global context-loss coefficient). (2) Depth: the encoder concatenates features from three intermediate blocks and the output layer along channels, an MLP fuses them, and the predictor emits four parallel outputs — both losses are applied at each level. (3) Tokenizer: 2D 16×16 conv for images, 3D 2×16×16 conv for videos, modality-learnable token added to both encoder and predictor inputs; this lets image and video samples share the encoder without temporal duplication. (4) Data + scale: Vision-Mix-163M with re-weighted video sources, separate per-modality workers (128 video clips × 16 frames, 2,304 images per global batch), ViT-G 2B trained 135k iters at fixed LR + 12k iters cool-down with raised resolution. A frozen-teacher distillation pipeline then produces ViT-L (300M) and ViT-B (80M) students.

Results

On the headline anticipation benchmarks, V-JEPA 2.1 ViT-G hits 7.71 mAP All on Ego4D STA (vs DINOv3 ViT-7B at 5.68, V-JEPA 2 ViT-g at 6.02, STAformer at 5.67) — and beats DINOv3 ViT-7B despite using ~3.5× fewer parameters [Table 3.1]. EPIC-KITCHENS-100 action anticipation reaches 40.8 R@5 (paper Abstract). Linear-probe dense tasks: NYUv2 RMSE 0.307 (vs 0.682 for V-JEPA 2), ADE20K 47.9 mIoU (vs 22.2), Pascal VOC 85.0 mIoU, YouTube-VOS 72.7 J-Mean [§1, §3.5]. Global tasks under attentive probe: Something-Something-v2 77.7% (vs 72.8 for V-JEPA 2), ImageNet 85.5%, PerceptionTest VQA 83.1% [§1, Table 2.3]. Embodied: +20% grasping success rate over VJEPA-2 AC on real Franka arms, zero-shot in new environments; Tartan Drive ATE 5.687 with 10× planning speed of prior SOTA [§1, §3.3, §3.4]. The ablation in Table 2.3 cleanly attributes each gain — context loss is what fixes dense features (and the cost is recovered by deep supervision), Vision-Mix scaling adds another point or two everywhere, ViT-G scaling is the biggest single jump.

Why it’s interesting

This is the most direct empirical statement to date of an idea the wiki has been circling from another direction: in masked-prediction transformers, tokens that are not supervised on their own position drift into being global aggregators, which destroys their local representation quality. The same family of effects shows up in attention-sink papers (The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks, A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training) where specific tokens absorb attention mass to act as no-op offload sites; V-JEPA 2.1 fixes the analogous problem at the self-supervised objective level rather than at attention. The work also strengthens the latent-predictive WFM thesis the wiki has been tracking via Action100M: A Large-scale Video Action Dataset (Action100M as the V-JEPA training corpus) and Inference-time Physics Alignment of Video Generative Models with Latent World Models (VJEPA-2 as a frozen physics-plausibility reward): V-JEPA 2.1 inherits all those downstream uses while extending the family to dense tasks (depth, segmentation, tracking) that DINOv2/v3 had owned. This puts the JEPA line into direct competition with the DINO line on the same benchmarks, with comparable-or-better numbers at smaller parameter counts. For Luma, the most interesting practical bit is the frozen-encoder embodied result (+20% grasping, 10× faster planning) — a sign that the same backbone could plausibly serve as both world-model representation and downstream policy input, vs maintaining separate encoders. The contrast with The flavor of the bitter lesson for computer vision is also worth noting: Sitzmann argues for generative-rollout WFMs as the right pre-training objective, while V-JEPA 2.1 is the strongest current evidence that latent-predictive WFMs can match dense-feature SOTA without ever generating a pixel.