LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics

TL;DR

LeVJEPA is the first video encoder trained under LeJEPA’s collapse-free objective — a single encoder optimized with a global-vs-local invariance MSE regularized by SIGReg, with no target encoder, no stop-gradient, no predictor, and no masked reconstruction. Because the architecture is fully symmetric, the encoder is compatible with aggressive uniform random token dropping (95% of patch tokens discarded) and with block-causal attention, both at no accuracy cost — in fact, token dropping raises ImageNet-1K accuracy from 33.9% at ρ=0 to 47.6% at ρ=0.95. At matched epochs on identical data, LeVJEPA matches or exceeds V-JEPA 2 across ViT-S/B/L at 5.6×–20.8× less total pretraining compute; at matched total FLOPs it exceeds the strongest video baseline by +7.6 points on ImageNet-1K while nearly doubling a compute-matched DINOv2’s accuracy on Something-Something-v2 (30.4 vs 16.9). Semantically-organized patch tokens emerge despite [cls]-only supervision.

Key claims

The trainable architecture reduces to one encoder plus one projector, and the objective to a single hyperparameter λ (invariance MSE + λ·SIGReg); no predictor, no target/EMA encoder, no stop-gradient, no masked prediction [§Objective and architecture].
Uniform random token dropping is not a computational approximation — it monotonically improves accuracy: ImageNet-1K top-1 rises from 33.9% at ρ=0 to 47.6% at ρ=0.95, a positive correlation between drop rate and downstream quality [§Token dropping].
Spatial arrangement of the retained tokens matters as much as the count: a tube variant that keeps identical spatial locations across frames attains 39.6% vs 50.7% for uniform random dropping — reversing the ordering established in masked video modeling [§Token dropping].
Conventional temporal patch aggregation at the input is unnecessary: per-frame tokenization at a matched token budget attains 50.7% vs 47.4% on ImageNet-1K and 30.4% vs 28.8% on Something-Something-v2 [§Token dropping].
Because no asymmetry between branches is required, the encoder admits block-causal attention (patch tokens attend bidirectionally within their frame, causally across frames) at no measurable accuracy cost — 51.2 vs 50.7 IN1K top-1 for block-causal vs bidirectional — enabling KV-caching for video encoders [§Block-causal attention].
At matched epochs on identical data, LeVJEPA matches or exceeds V-JEPA 2 across ViT-S/B/L at 5.6× to 20.8× less pretraining compute (20.8× at ViT-S, 5.6× at ViT-L) [Abstract, §Empirical comparison].
At matched total FLOPs on a 20% K710 subsample, LeVJEPA ViT-B beats V-JEPA 2 by +9.4 on ImageNet-1K (61.0 vs 51.6) and beats VideoMAEv2 by +7.6 (61.0 vs 53.4), while trailing VideoMAEv2 by 3.2 on Something-Something-v2 (40.4 vs 43.6) [§Empirical comparison, Table 3].
Semantically organized patch representations emerge without any patch-level loss — only the [cls] token is supervised, but query-based cosine similarity maps stay confined to individual objects rather than diffusing across the frame [§Patch-level representations].
Compared against a compute-matched DINOv2 trained on frames of the same videos, LeVJEPA trails by 3.1 points on ImageNet-1K (50.7 vs 53.8) but nearly doubles Something-Something-v2 accuracy (30.4 vs 16.9) — video-pretrained encoders finally outperform image-pretrained ones on motion at equal FLOPs [§Comparison with image pretraining].
Compute-bar drops: a ViT-Tiny trained for 12 hours on a single consumer GPU on 8 Walking Tours videos (~620k frames) improves from 8.9% to 25.2% ImageNet-1K top-1 under frozen evaluation [§Computational requirements and data scaling].
Full-scale run: a ViT-L/16 pretrained for 100 epochs on the union of K710 + SSv2 + Walking Tours + PE Video attains 67.5% on ImageNet-1K and 55.0% on Something-Something-v2 under frozen attentive probing — without objective modification or hyperparameter retuning [§Computational requirements and data scaling].

Method

Each video contributes a 16-frame clip from which V+1 views are constructed: one global view at full resolution and V local views under spatial cropping + photometric augmentation, all sharing the same temporal window. Every view is processed by the same encoder E_θ; a learnable [cls] token is the clip-level readout, mapped by a projector to embedding z_v ∈ ℝ^K. The training loss is L = L_inv + λ·L_SIGReg, where L_inv is MSE between each local embedding and the global one and L_SIGReg (from LeJEPA / LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels) constrains the embedding distribution to an isotropic Gaussian via Cramér–Wold projection to 1D Epps–Pulley tests. The trade-off weight λ is the objective’s only hyperparameter. After patch embedding, a fraction ρ (default 0.95) of the patch tokens of each view is discarded uniformly at random — the retained tokens are the encoder’s sole observation of the clip. The paper adopts block-causal attention (bidirectional within a frame, causal across frames), so each frame representation is a function of the current and preceding frames alone — a property the objective’s symmetry permits and that KV-caching exploits at inference. There is no predictor, no target encoder, no EMA, no stop-gradient, no masked-region reconstruction, and no patch-level auxiliary loss.

Results

At matched epochs on identical data, LeVJEPA matches or exceeds V-JEPA 2 across ViT-S/B/L at 5.6× (ViT-L) to 20.8× (ViT-S) less total pretraining compute; at ViT-B the two methods are within one point at 4.8 vs 36.4 ExaFLOPs [§Empirical comparison]. Under a fixed total FLOP budget on a 20% K710 subsample at ViT-B, LeVJEPA reaches 61.0 IN1K / 40.4 SSv2 / 44.6 K400 vs V-JEPA 2’s 51.6 / 42.5 / 40.7 and VideoMAEv2’s 53.4 / 43.6 / 37.4 — a +7.6 IN1K margin over the strongest baseline while remaining competitive on motion [Table 3]. Against a compute-matched DINOv2 on frames of the same videos: 50.7 vs 53.8 on IN1K (image-pretrained still ahead on appearance), 30.4 vs 16.9 on SSv2 (video-pretrained nearly doubles image-pretrained on motion) [§Comparison with image pretraining]. Block-causal attention lifts IN1K from 50.7 to 51.2 while enabling KV-cached inference. Token-drop rate correlates positively with accuracy from ρ=0 (33.9%) to ρ=0.95 (47.6%). Number-of-local-views scaling: 47.6% at V=4 → 50.2% at V=10 — results not saturated. Scaled ViT-L/16 (100 epochs, K710+SSv2+WT+PE Video) hits 67.5 IN1K / 55.0 SSv2 frozen attentive-probe. Compute floor: a ViT-Tiny on 8 Walking Tours videos (~620k frames) trained for 12h on one consumer GPU goes 8.9% → 25.2% IN1K.

Why it’s interesting

LeVJEPA is the sharpest single-paper cash-out yet of the JEPA-family design ethos this wiki has been tracking — it takes the two-loss end-to-end recipe of LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels (LeWorldModel: MSE + SIGReg) and stress-tests it at video-encoder scale, arriving at the same compute-Pareto position (5.6×–20.8× less compute vs V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning) that LeWorldModel reported vs PLDM at planning scale (48× faster). Structurally this matters because V-JEPA 2’s successor V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning fixes dense-feature quality by extending the loss to all tokens, while LeVJEPA gets semantically-organized patch representations without patch-level supervision at all — the same phenomenon The Obsessed Encoder — Latent-Space Misallocation in JEPA-Style Self-Supervised Models flagged as pathological in DINOv3/LeJEPA/LeWorldModel (SIGReg reports healthy statistics while capacity is spent on the wrong signal) here appears to work in the right direction, though the paper’s evaluations are all downstream accuracies rather than the low-entropy-goal probes Enigma used. The 95%-token-drop finding directly complements Diffusion training efficiency‘s ongoing thread on spatial token reduction (TREAD, adaptive-patch ViTs) — same lever, different domain — and the block-causal-attention-at-no-cost result is a concrete inference-serving win the frozen-V-JEPA-as-reward-model / world-model-as-planner lines have not previously had access to. Given the x-jepa — Explorations into JEPA approaches (lucidrains) repo already ships a portable minimal implementation of the LeJEPA+LeWorldModel core, this is likely reproducible on internal data.