Skip to content
Bud's Wiki

V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning

TL;DR

Section titled “TL;DR”

V-JEPA 2 is the FAIR/Meta paper that scales the Joint-Embedding Predictive Architecture from the original V-JEPA to a 1B-parameter ViT-g encoder pre-trained on VideoMix22M (22M clips, >1M hours of video plus 1M ImageNet images) with a two-phase warmup-constant-cooldown schedule that trains at 256² × 16-frames then cools down at 384² × 64-frames. Frozen encoder features under an attentive probe hit 77.3 top-1 on Something-Something v2 (motion) and 39.7 R@5 on Epic-Kitchens-100 action anticipation, and aligning the encoder to Llama-3.1-8B yields SOTA on Perception Test, TempCompass, TemporalBench and TOMATO at the 8B class. The paper then post-trains a 300M-parameter action-conditioned predictor (V-JEPA 2-AC) on just 62 hours of unlabeled DROID robot video and deploys it zero-shot on Franka arms for image-goal pick-and-place via latent-space model-predictive control — no lab-specific data, no task-specific reward.

Key claims

Section titled “Key claims”
  • Data + model + iterations + resolution each contribute independently to a cumulative +4.0-point average understanding gain over the ViT-L/2M-clip V-JEPA baseline (+1.0 data, +1.5 model, +0.8 iterations, +1.7 higher-resolution cool-down) [§2.2, Fig. 3].
  • 3D-RoPE replaces sincos absolute position embeddings; this is called out as necessary to stabilize training of the largest (1B) encoder [§2.1].
  • A progressive-resolution training schedule — warmup and constant phase at 256² × 16 frames, learning-rate decay phase at 384² × 64 frames — cuts total compute by 8× vs training at full resolution throughout, while retaining the benefit of longer/higher-resolution inputs [§2.4, Fig. 5].
  • Increasing pretraining clip length from 16 to 64 frames improves average understanding by +1.7 points even when evaluation is fixed at 16 frames; longer than 64 frames yielded no further gain on these tasks [§2.4].
  • YT-Temporal-1B (1.4M video-hours, uncurated) is curated into “Curated-YT-1B” via retrieval against a target distribution of Kinetics/SSv2/COIN/EpicKitchen — a ViT-L trained on Curated-YT-1B alone is competitive with the full VM22M mix at that scale, and larger models benefit more from adding VM22M [§2.3].
  • V-JEPA 2 sets SOTA on Something-Something v2 motion recognition (77.3 top-1, attentive probe) and Epic-Kitchens-100 action anticipation (39.7 R@5), a +44% relative improvement over the previous best on Epic-Kitchens [Abstract, §5, §6].
  • Aligning the frozen V-JEPA 2 encoder with Llama-3.1-8B via a light connector yields SOTA in the 8B class on MVP, PerceptionTest (84.0), TempCompass (76.9), TemporalBench and TOMATO — contradicting prior wisdom that vision encoders need language supervision to align well with LLMs [§7].
  • V-JEPA 2-AC is a 300M-parameter 24-layer transformer trained on 62 hours of DROID with a teacher-forcing latent-prediction loss plus a two-step rollout loss for autoregressive stability [§3.1].
  • V-JEPA 2-AC deployed zero-shot on Franka arms in two Meta labs (never seen during training) solves prehensile Grasp and Pick-and-Place tasks with novel objects via image-goal MPC — no task-specific reward, no lab-specific fine-tuning [§4].
  • V-JEPA 2-AC end-effector state is a 7D vector (3D position, 3-axis orientation, 1D gripper open/close) and actions are deltas between consecutive states; the predictor uses block-causal attention over interleaved (a_k, s_k, z_k) tokens with a shared feature-map embedding [§3.1].

Method

Section titled “Method”

Stage 1 (mask-denoising pre-training) trains an encoder + predictor pair with the standard V-JEPA loss on VideoMix22M plus 1M ImageNet images (duplicated temporally into 16-frame static clips). Four scaling ingredients are stacked: (a) 10× data, (b) ViT-L → ViT-g (300M → 1B), (c) 90k → 252k iterations under a warmup-constant-cooldown schedule, (d) progressive resolution — warmup and constant phase at 256² × 16-frame clips, LR-decay cool-down at 384² × 64-frame clips. 3D-RoPE (splitting the channel dim into temporal/H/W thirds) replaces sincos position embeddings so that variable-resolution inputs stay consistent. Attentive probing — output tokens pass through a small stack of learnable transformer blocks, then query tokens cross-attend and pool — is used as the primary frozen-feature evaluation instead of a linear head or average pool. Stage 2 (V-JEPA 2-AC) freezes the 1B encoder and trains a 300M block-causal transformer predictor on 62 hours of DROID data. Each 4-second clip is sampled at 4 fps into 16 frames, encoded per-frame into feature maps of shape 16×16×1408, and interleaved with actions a_k (7D end-effector delta) and states s_k (7D end-effector pose) into a sequence (a_1, s_1, z_1), (a_2, s_2, z_2), …. The predictor is trained with teacher forcing plus a rollout loss over one recurrent step to reduce autoregressive drift. At inference, given a current state and a goal image, the model performs latent-space MPC — sample candidate action sequences, roll them out through the frozen encoder + predictor, and pick the sequence whose final predicted latent is closest to the goal latent.

Results

Section titled “Results”

Motion + appearance understanding on the 6-task frozen attentive-probe suite (SSv2, Diving-48, Jester, Kinetics, COIN, ImageNet): baseline ViT-L on VM2M averages 84.2%; VM22M lifts to 85.2%; ViT-g adds another 1.5; longer training adds 0.8; and 384²/64-frame cool-down brings the average to 88.2% [Fig. 3]. Something-Something v2: 77.3 top-1 (SOTA on motion) [§5]. Epic-Kitchens-100 action anticipation: 39.7 R@5, a +44% relative improvement over the previous best [Abstract, §6]. Language-aligned VQA at 8B scale: PerceptionTest 84.0, TempCompass 76.9, plus SOTA on MVP / TemporalBench / TOMATO [§7]. Robot deployment: V-JEPA 2-AC on Franka arms in two Meta labs solves Grasp and Pick-and-Place zero-shot on unseen objects, with no additional robot-specific data (specific success-rate numbers are given in the accompanying blog release, Introducing the V-JEPA 2 world model and new benchmarks for physical reasoning, at 65–80%). Progressive-resolution scaling cuts training cost by 8× vs uniform 384²/64-frame training [Fig. 5].

Why it’s interesting

Section titled “Why it’s interesting”

The wiki has V-JEPA 2 as its foundational latent-predictive World Foundation Models entry via the Introducing the V-JEPA 2 world model and new benchmarks for physical reasoning launch post; this arxiv paper is the technical report behind it, and filling it in is important because V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning (V-JEPA 2.1) diagnoses exactly the failure mode of the objective used here — that visible context tokens are left unsupervised and drift into register-like global aggregators, hurting dense features — and the delta between the two papers is the cleanest “one-liner” fix in recent SSL literature. Two design points are worth flagging for Luma. First, the progressive-resolution cool-down (256²/16 → 384²/64 at the end) is a general recipe for cheaply training video models at longer/higher resolution — the paper reports 8× compute savings and shows it stacks additively with data/model/iteration scaling; this is directly relevant to Hyperparameter scaling laws and to any video-generation training run that would otherwise pay full quadratic cost at target resolution. Second, V-JEPA 2-AC’s 62-hour post-training recipe — freeze a large action-agnostic backbone, learn only a small action-conditioned predictor, run latent-space MPC — is a cheap alternative to full VLA Models pipelines and contrasts sharply with the diffusion-rollout VLA line (ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?, Data Pyramid for Embodied Manipulation). The paper is also the first to publicly claim that a purely-visual SSL encoder (no language supervision) can beat language-supervised video encoders as an LLM adapter — this contradicts wisdom in the Unified Multimodal Models cluster and is worth watching.

See also

Section titled “See also”