LeVJEPA: Collapse-Free Video Pretraining at 20x Less Compute

Video is the most information-dense signal we can pretrain on — and the most expensive. LeVJEPA, from the LeJEPA line (LeCun, Balestriero, et al.), rips out the entire machinery that made video pretraining stable, replaces it with one provably collapse-free regularizer, and ends up faster and better: 5.6–20.8× less compute to match V-JEPA 2, and a 7.6-point win on ImageNet-1K at matched FLOPs. It's the strongest argument yet that video, not images, should be the default substrate for visual pretraining.

The Problem: Stability Has a Tax

Self-supervised video encoders can't just be trained with an invariance loss — without constraints, all embeddings collapse to a constant. The field's answer has been architectural heuristics: an exponential-moving-average target encoder, a stop-gradient, a capacity-limited predictor, or reconstructing masked patches in pixel space. They work; they also cost. V-JEPA 2 pays for stability in extra branches, extra hyperparameters, and compute. LeJEPA showed on images that collapse can be excluded provably instead of engineered around. The open question was whether that carried to video. It does.

The Method: One Encoder, One Regularizer, No Tricks

LeVJEPA is a single shared encoder plus a projector. No target encoder, no predictor, no stop-gradient, no masked reconstruction. An invariance (MSE) loss pulls each local [cls] embedding toward the global view's embedding, with gradients flowing through both branches. The anti-collapse work is done by SIGReg: project the batch of embeddings onto random directions and penalize deviation from a standard Gaussian via a normality test — the embedding distribution is constrained to an isotropic Gaussian, which provably rules out collapse.

That simplicity unlocks two properties. First, pretraining cost is governed by how many tokens the encoder observes, so 95% of patch tokens are dropped uniformly at random — less compute, and per the paper, better downstream accuracy. Second, with no branch asymmetry required, the encoder can use block-causal attention at no measurable cost: temporal ordering becomes a property of the encoder itself, not a training constraint. The whole objective reduces to a single hyperparameter.

flowchart LR
    A[Video clip] --> B[Drop 95% of patch tokens]
    B --> C[Shared block-causal encoder]
    C --> D[Global view embedding]
    C --> E[Local view embeddings]
    D --> F[MSE invariance pull]
    E --> F
    C --> G[SIGReg normality test]
    G --> H[Isotropic Gaussian · no collapse, provably]
    F --> H

The Numbers

• Epoch-matched on identical data: matches or beats V-JEPA 2 across ViT-S/B/L at 5.6–20.8× less total pretraining compute.
• FLOP-matched (20% K710 subsample, frozen encoders): LeVJEPA scores 61.0 on ImageNet-1K vs VideoMAEv2's 53.4 and V-JEPA 2's 51.6; 44.6 on Kinetics-400 (best of the three); 40.4 on Something-Something-v2 — competitive, though slightly behind on the motion-centric set (43.6 / 42.5).
• Matched FLOPs: +7.6 points over the strongest video baseline on ImageNet-1K.
• vs compute-matched DINOv2 trained on frames of the same videos: approaches it on appearance-centric tasks while nearly doubling motion-centric accuracy.

Limitations — Read These First

• Eval is recognition-centric: attentive/linear probing on ImageNet-1K, K400, SSv2. No generative video, no embodied/action-conditioned evaluation — this measures representation quality, not world-model capability.
• Motion-heavy gap: SSv2 slightly trails the baselines even at matched FLOPs; the win is concentrated on appearance-centric benchmarks.
• Scale envelope: ViT-S/B/L on a K710 subsample. The 5.6–20.8× claim holds at these sizes; giant-model behavior is untested here.
• 95% token dropping is aggressive — it works on this protocol, but robustness across data regimes and decoder needs is open.
• Single-author-family bias: SIGReg results are compared chiefly against V-JEPA 2 / DINOv2 (JEPA family); independent replication would sharpen the picture.

Why Someone Building Things Should Care

If video pretraining costs an order of magnitude less than image pretraining at equal quality, the default substrate flips — video becomes the training data of choice for general-purpose vision encoders, which is what the paper's closing claim argues. Block-causal encoders are also natively streaming-friendly: every frame's representation depends only on the past, which matters for online and embodied use. And the transferable principle — make cost proportional to tokens seen, not to architectural complexity — is one any pretraining pipeline can steal. For teams doing world models or robotics perception, this is the objective to try next: one hyperparameter, no EMA schedule to tune, no predictor games. The JEPA line keeps making "less machinery, more proof" look like the right bet.