TL;DR
- V-JEPA 2 scales masked-only latent L1 prediction with larger video data, models and training, adding 3D-RoPE and a trained progressive-resolution schedule.
- Masked video pretraining uses bidirectional clip context. Robotics uses a separate V-JEPA 2-AC predictor trained on 62 hours of video paired with recorded actions and end-effector states, then searches for plans in latent space.
- Frozen classification, anticipation and robot deployment are separate evaluations. Keep backbone, resolution, clip length and probe settings attached to scores; the runnable reference and feature inspection do not reproduce the robot planner.
- The V-JEPA 2.1 follow-up adds weighted visible-position prediction and multi-level supervision for dense features. Its checkpoints and results are distinct from the original V-JEPA 2 outputs inspected here.
Paper Overview
V-JEPA showed that predicting masked video regions in latent space β rather than reconstructing pixels β produces superior video representations. V-JEPA 2 asks the natural follow-up: what happens when you scale this idea to its limits, and what new capabilities emerge?
The answer is striking. V-JEPA 2 scales the joint-embedding predictive architecture to a ViT-g encoder (over 1 billion parameters), trains on VideoMix22M (a curated dataset spanning more than 1 million hours of internet video), and retains masked-only L1 representation prediction while scaling model, data, training duration, and input resolution. In the architecture, 3D Rotary Position Embeddings replace fixed sinusoidal encodings. The higher-resolution ViT-g384 reaches 77.3% frozen top-1 on Something-Something v2 using 384px input and 64-frame clips; the common-protocol ViT-g reaches 75.3% with 256px input and 16-frame clips (PDF v1, Table 4). The separately evaluated action-anticipation model reaches 39.7 recall@5 on Epic-Kitchens-100 (Table 5).
But V-JEPA 2βs most remarkable contribution goes beyond classification. The authors extend V-JEPA 2 into V-JEPA 2-AC, an action-conditioned world model that can plan robot actions in latent space. Trained on just 62 hours of robot video with recorded actions and end-effector states from the Droid dataset, V-JEPA 2-AC is deployed zero-shot on physical Franka robotic arms in two different labs β achieving mean pick-and-place success across the two labs of 80% for cups and 65% for boxes by planning in 16 seconds what pixel-generation approaches like Cosmos take 4 minutes to compute. This demonstrates a path from self-supervised video understanding to embodied intelligence, all without task-specific labels, reward signals, or data from the target robots.
Why Video Understanding Needs Temporal Reasoning
Most image recognition benchmarks can be solved by analyzing appearance alone β a single frame of a dog is enough to classify βdog.β Video understanding is fundamentally harder because many actions can only be distinguished through temporal reasoning. Consider βpushing something leftβ versus βpushing something rightβ β a single frame shows a hand near an object, but the direction of motion is invisible without observing change across time.
SSv2 emphasizes motion-dependent distinctions, but an image-pretrained encoder can also process a sequence frame by frame. In PDF v1, Table 4 and Section 5, DINOv2 with registers reaches 50.7% SSv2 and 83.6% K400 by concatenating frame-wise features before a trained attentive probe. Original V-JEPA ViT-H reaches 74.3%/84.5%, and V-JEPA 2 ViT-g reaches 75.3%/86.6% under the same 256px evaluation protocol. These are system-level comparisons with different pretraining data and model sizes. Classification or anticipation accuracy does not by itself demonstrate temporal causality.
The V-JEPA 2 Pipeline
V-JEPA 2 follows the joint-embedding predictive framework established by I-JEPA and V-JEPA: an encoder processes visible context, a predictor maps this context to predictions for masked regions, and a momentum-updated target encoder provides the ground-truth embeddings. The training objective operates entirely in latent space β no pixel reconstruction, no decoder, no auxiliary losses.
The input video is divided into spatiotemporal tubelets of size 2 \times 16 \times 16 (2 frames Γ 16Γ16 pixels). Multi-block masking removes 85β95% of these tubelets, creating large contiguous gaps that force the model to reason about motion and semantics rather than interpolating from nearby visible patches. The encoder (ViT-g, 1B+ parameters) processes the visible tubelets using 3D Rotary Position Embeddings, and the predictor (ViT-S) maps the encoded context plus positional mask tokens to predicted embeddings. The target encoder β an exponential moving average of the main encoder with stop-gradient β processes the full video to produce target embeddings. The loss is L1 between predicted and target representations:
where \hat h = P_\phi(\Delta_y,E_\theta(x)) gives predictions, h = E_{\bar\theta}(y) gives full-video targets, M indexes masked positions, E_\theta is the encoder, P_\phi is the predictor, E_{\bar\theta} is the EMA target encoder, x represents visible patches, y represents the complete video, and \text{sg} denotes stop-gradient.
What Mask Denoising Means
In Section 2.1 and Figure 2 of the paper, mask denoising means predicting representations of missing video tubelets from visible context. The online encoder receives only retained tubelets. The predictor then combines their features with learned mask queries at the missing positions. The target encoder sees the complete video, and its stop-gradient features supply the regression targets.
This follows the original V-JEPA objective: both use masked-only L1 loss. It is not a switch from L2 to L1, and the encoder does not receive Gaussian-corrupted versions of the missing tubelets. The main architectural change here is 3D-RoPE, alongside scaling of data, model size, training duration, and input resolution. Because masked pretraining attends bidirectionally to the retained clip, it also differs from the action-conditioned, causal future prediction used later for robotics.
Progressive Resolution Training
Training a ViT-g on 64-frame video clips at 384Γ384 resolution from the start would require approximately 60 GPU-years on A100s. V-JEPA 2 avoids this through a progressive training strategy that starts at low resolution and scales up, achieving an 8.4Γ reduction in compute.
The training schedule has three stages. First, a warmup of 12,000 iterations at 16 frames and 256Γ256 resolution. Second, a constant phase of 228,000 iterations at the same low resolution β the bulk of training. Third, a cooldown of 12,000 iterations where both spatial resolution (256β384) and temporal length (16β64 frames) increase while the learning rate linearly decays. The model learns coarse spatiotemporal patterns during the long low-resolution phase and sharpens them during the brief high-resolution cooldown.
3D Rotary Position Embeddings (3D-RoPE) assign temporal, height, and width coordinates to the attention queries and keys. Their feature dimensions are split into three segments, with separate rotations for each axis. When the token grid grows during cooldown, those coordinates extend to the new grid. The model still receives additional training at the increased resolution; neither positional coordinates nor the reported schedule guarantee arbitrary-resolution generalization.
From Understanding to Planning: V-JEPA 2-AC
V-JEPA 2-AC extends the frozen V-JEPA 2 encoder into an action-conditioned world model β a system that can predict what will happen in a scene given a sequence of robot actions. The key insight is that planning in latent space is both faster and more effective than generating pixel-level video of the future.
The architecture is a ~300M parameter transformer with 24 layers, block-causal attention, and 7D action vectors (3D position, 3D orientation, gripper state). Given the current scene encoded by the frozen V-JEPA 2 encoder and a proposed action, the predictor outputs the predicted next state β also in latent space. By chaining these predictions, the model can βimagineβ multi-step futures without generating a single pixel.
Planning uses the Cross-Entropy Method (CEM) to search for action sequences whose predicted future states are closest (by L1 distance) to an encoded goal image. This search runs entirely in the compact latent space, completing in about 16 seconds. By contrast, video-generation approaches like Cosmos must render full-resolution pixel predictions for each candidate plan, taking over 4 minutes per query β a 15Γ slowdown.
The most remarkable aspect is the training data requirement: just 62 hours of robot video with recorded actions and end-effector states from the Droid dataset. No task labels. No reward signals. No success indicators. The action-conditioned predictor learns purely from observing how actions cause state changes, and this minimal training is sufficient for zero-shot deployment on physical Franka arms in two different labs, achieving mean pick-and-place success across the two labs of 80% for cups and 65% for boxes.
Implementation
The distinction between masked video pretraining and action-conditioned planning becomes concrete in the training loop. The eight steps below follow tubelets through masking, 3D positions, prediction, and a momentum-teacher update. After this numerical reference, a separate frozen-model inspection shows actual original V-JEPA 2 ViT-L features. Neither is a reproduction of the robot planner.
A runnable, reduced training core
This offline PyTorch program starts from random weights and checks the mechanics with synthetic video-shaped tensors. It does not reproduce VideoMix22M pretraining, progressive resolution, task probes, or V-JEPA 2-AC planning. The actual pretrained inspection follows below.
Download PythonPinned requirements
python -m pip install -r requirements-vjepa2.txt
python vjepa2_training.py --smoke
1. Project video tubelets in a fixed order
Original paper
The released encoder uses a 3D convolution with non-overlapping 2 Γ 16 Γ 16 tubelets. The reference keeps two-frame tubelets and reduces each spatial patch to 8 Γ 8.
Math and tensor shapes
The convolution projects each tubelet to 48 features. Flattening time, height, and width in this order makes every token id a recoverable 3D coordinate. The full ViT-L export below uses 1024 features and the original 16-pixel patch size.
[2, 3, 8, 32, 32] β [2, 64, 48]; grid 4 Γ 4 Γ 4
Python implementation
class Tubelets(nn.Module):
"""Projects non-overlapping video tubelets in time-height-width order."""
def __init__(self, width=48, tubelet=2, patch=8):
super().__init__()
if min(width, tubelet, patch) <= 0:
raise ValueError('Tubelet dimensions and width must be positive.')
self.size = (tubelet, patch, patch)
self.proj = nn.Conv3d(3, width, self.size, stride=self.size)
def forward(self, video):
if video.ndim != 5 or video.shape[1] != 3:
raise ValueError('Expected [batch, 3, time, height, width].')
if any(n <= 0 or n % p for n, p in zip(video.shape[2:], self.size)):
raise ValueError('Video dimensions must be positive tubelet multiples.')
grid = tuple(n // p for n, p in zip(video.shape[2:], self.size))
return self.proj(video).flatten(2).transpose(1, 2), grid2. Unite spatial blocks across the clip
Original paper
The author masker forms unions of blocks, derives visible and target indices, and shares their spatial extent through time. The reference demonstrates this geometry with two rectangles per mask.
Math and tensor shapes
Each pair is a disjoint partition, with increasing original ids. This small profile uses two masks shared across the batch, rather than the full recipeβs per-sample multiblock scales and roughly 90% masking. It is bidirectional masked pretraining, not a mask that hides only the future.
Two (context ids, target ids) pairs; smoke seed keeps 32 of 64 tokens
Python implementation
def make_masks(grid, generator, count=2):
"""Unions spatial rectangles across ALL times; returns ordered complements."""
time, height, width = grid
if min(time, height, width) < 2 or count < 1:
raise ValueError('Mask grid axes must be >= 2; count must be positive.')
pairs = []
for _ in range(count):
spatial = torch.zeros(height, width, dtype=torch.bool)
for _ in range(2):
h, w = max(1, height // 2), max(1, width // 2)
top = int(torch.randint(height - h + 1, (), generator=generator))
left = int(torch.randint(width - w + 1, (), generator=generator))
spatial[top:top + h, left:left + w] = True
target = spatial.expand(time, -1, -1).flatten()
# nonzero preserves chronological, row-major tubelet ids.
pairs.append(((~target).nonzero().flatten(), target.nonzero().flatten()))
return pairs3. Put time, row, and column into attention
Original paper
Query and key channels are split into three even-width axis segments. Any leftover channels pass through unchanged; positions come from original grid ids.
Math and tensor shapes
This function implements standard pairwise sine/cosine rotations and is tested against their norm and relative-position properties. The pinned author source documents a frequency-repeat quirk in its original checkpoints. The genuine export preserves that released behavior, so this teaching function is deliberately not a checkpoint-compatible attention replacement.
Q, K [2, 4, N, 12]; 4 channels per axis
Python implementation
def rope3d(values, ids, grid):
"""Rotates Q/K pairs separately for time, row and column positions."""
axis_width = 2 * ((values.shape[-1] // 3) // 2)
if axis_width < 2:
raise ValueError('Attention head width must be at least six.')
_, height, width = grid
positions = (ids // (height * width), (ids // width) % height, ids % width)
frequencies = 10000.0 ** (-torch.arange(
0, axis_width, 2, device=values.device, dtype=values.dtype) / axis_width)
parts = []
for axis, position in enumerate(positions):
angle = position[:, None] * frequencies[None, :]
pairs = values[..., axis * axis_width:(axis + 1) * axis_width]
even, odd = pairs[..., 0::2], pairs[..., 1::2]
rotated = torch.stack((even * angle.cos() - odd * angle.sin(),
even * angle.sin() + odd * angle.cos()), dim=-1)
parts.append(rotated.flatten(-2))
return torch.cat(parts + [values[..., 3 * axis_width:]], dim=-1)4. Encode only visible context
Original paper
Tubelets are embedded, selected by visible indices, and then processed by the encoder. Masked content never enters encoder self-attention.
Math and tensor shapes
The same two pre-norm transformer blocks process every mask view. Keeping original ids is essential: compacting the sequence must not renumber a tokenβs position. A test changes all masked pixels and verifies that the visible encoder output is unchanged.
[2, 64, 48] β [2, 32, 48] for either smoke mask
Python implementation
class Encoder(nn.Module):
"""Encodes retained tubelets while preserving their original positions."""
def __init__(self, width=48):
super().__init__()
self.tubelets = Tubelets(width)
self.blocks = nn.ModuleList([Block(width) for _ in range(2)])
self.norm = nn.LayerNorm(width)
def forward(self, video, ids=None):
tokens, grid = self.tubelets(video)
if ids is None:
ids = torch.arange(tokens.shape[1], device=video.device)
tokens = tokens[:, ids] # Drop masked tubelets before self-attention.
for block in self.blocks:
tokens = block(tokens, ids, grid)
return self.norm(tokens)5. Build full-video targets without gradients
Original paper
The target encoder processes the complete clip under no_grad. The released training loop additionally applies per-token layer normalization to its outputs.
Math and tensor shapes
The teacher starts as a copy of the student. Its weights move only through EMA; disabling autograd for this branch is separate from putting it in evaluation mode. The teacher can see missing content because its outputs are supervision, not predictor inputs.
Full targets [2, 64, 48]; teacher parameters require no gradients
Python implementation
def make_teacher(encoder):
"""Starts the momentum target at the student weights, without gradients."""
return copy.deepcopy(encoder).requires_grad_(False).eval()
@torch.no_grad()
def teacher_targets(teacher, video):
"""Encodes the full video, then normalizes each target feature vector."""
targets = teacher(video)
return F.layer_norm(targets, (targets.shape[-1],)).detach()6. Append learned queries at missing positions
Original paper
The predictor projects visible features, appends learned mask tokens, applies attention, and returns only predictions for target positions.
Math and tensor shapes
This reduced predictor uses one block and one learned query vector, with position supplied by RoPE. The release uses a deeper predictor and a bank of mask tokens. Target order must match teacher gathering; reversing target ids must reverse predictions, which the behavior test checks.
[2, 32, 48] β 64 predictor tokens of width 24 β [2, 32, 48]
Python implementation
class Predictor(nn.Module):
"""Combines encoded context with learned queries at missing positions."""
def __init__(self, encoder_width=48, width=24):
super().__init__()
self.input = nn.Linear(encoder_width, width)
self.query = nn.Parameter(torch.zeros(1, 1, width))
self.block = Block(width, heads=2)
self.norm = nn.LayerNorm(width)
self.output = nn.Linear(width, encoder_width)
def forward(self, context, context_ids, target_ids, grid):
queries = self.query.expand(context.shape[0], len(target_ids), -1)
tokens = torch.cat((self.input(context), queries), dim=1)
ids = torch.cat((context_ids, target_ids))
# RoPE uses original ids even though context and targets are concatenated.
tokens = self.block(tokens, ids, grid)
return self.output(self.norm(tokens[:, len(context_ids):]))7. Regress masked features with L1
Original paper
Only masked positions contribute to the loss. With loss exponent one, the released loop averages absolute feature error within each view, then across mask views.
Math and tensor shapes
Mean here includes batch, masked tokens, and feature channels. Each mask has equal weight even if its token count differs. The target is detached, so gradients reach the online encoder, predictor, and learned queries only. Both original V-JEPA and V-JEPA 2 use L1.
Two scalar mask losses β one scalar; no negative pairs or pixel decoder
Python implementation
def masked_loss(encoder, predictor, teacher, video, masks):
"""Averages masked-only L1 across masks, never reconstructing visible tokens."""
if not masks:
raise ValueError('At least one mask is required.')
targets = teacher_targets(teacher, video)
grid = tuple(n // p for n, p in zip(video.shape[2:], encoder.tubelets.size))
terms = []
for context_ids, target_ids in masks:
all_ids = torch.cat((context_ids, target_ids))
expected = torch.arange(targets.shape[1], device=all_ids.device)
if not len(context_ids) or not len(target_ids) or not torch.equal(
all_ids.sort().values, expected):
raise ValueError('Context and target ids must partition the video.')
context = encoder(video, context_ids)
prediction = predictor(context, context_ids, target_ids, grid)
terms.append(F.l1_loss(prediction, targets[:, target_ids]))
return torch.stack(terms).mean()8. Update online weights, then the teacher
Original paper
The training loop steps the optimizer before updating the target from the new encoder weights. Section 2.4 reports a fixed EMA coefficient in V-JEPA 2.
Math and tensor shapes
The runnable smoke test performs three updates with AdamW and a fixed seed. After training, the encoder can process the full token grid for a downstream probe. Robot planning additionally needs the separately trained V-JEPA 2-AC predictor, actions, end-effector states, and a planning loop; none is implied by this encoder output.
Reduced profile m = 0.99; teacher excluded from optimizer
Python implementation
@torch.no_grad()
def update_teacher(encoder, teacher, momentum=0.99):
"""Updates the target AFTER the online optimizer step, with fixed momentum."""
if not 0 <= momentum <= 1:
raise ValueError('EMA momentum must lie in [0, 1].')
for online, target in zip(encoder.parameters(), teacher.parameters()):
target.mul_(momentum).add_(online, alpha=1 - momentum)
def train_step(encoder, predictor, teacher, optimizer, video, masks):
optimizer.zero_grad(set_to_none=True)
loss = masked_loss(encoder, predictor, teacher, video, masks)
loss.backward()
optimizer.step()
update_teacher(encoder, teacher)
return float(loss.detach())Inspect genuine frozen ViT-L features
The original V-JEPA 2 ViT-L target encoder loaded all 292 tensors strictly from the official checkpoint. Each photograph below is repeated for 16 identical frames. These are appearance-only inputs, not motion videos, action predictions, or a benchmark. The authorβs native encoder produces 2048 tubelet features of width 1024; their mean gives one vector per clip.
Pooled feature L2 norm: 53.194
First 4 channels: 0.142, -0.858, 0.013, -2.176
Pooled feature L2 norm: 50.436
First 4 channels: 0.024, -0.183, -0.414, 0.085
Cosine similarity between the two mean-pooled clip features: 0.5526. This is a feature-space measurement for these two inputs, not a class probability. The artifact contains the full two Γ 1024 float32 vectors.
Preprocessing resizes the shorter edge to 256 using PIL bilinear interpolation, center-crops 256 Γ 256 pixels, normalizes with ImageNet mean and standard deviation, and repeats the frame. Source hashes, crop coordinates, checkpoint hash, exact command, and runtime are recorded in the manifest. The native loader replaces only stochastic depth and removes an unused import; it preserves the released RoPE frequency-repeat quirk, unlike the standard pairwise rotations in the teaching core.
Manifest and numeric outputFeature vectorsNative export scriptSeparate action-conditioned method (Β§3)
How V-JEPA 2 Compares
Video Self-Supervised Method Comparison
PDF v1, Table 4 reports frozen-encoder classification, not fine-tuned results. The common protocol trains a four-layer attentive probe; image encoders concatenate frame-wise features. 256px; SSv2 16Γ2Γ3, K400 16Γ8Γ3 (frames Γ temporal Γ spatial crops). Literature rows use their reported frozen protocols and potentially different heads. The g384 row increases spatial resolution and SSv2 clip length. Section 5 gives the distinctions.
| Method | Architecture | Evaluation | SSv2 (%) | K400 (%) | Protocol group |
|---|---|---|---|---|---|
| VideoMAEv2 | ViT-g (1B) | Frozen; literature probe/inputs | 56.1 | 82.8 | Literature |
| InternVideo2-1B | ViT (1B) | Frozen; literature probe/inputs | 67.3 | 87.9 | Literature |
| InternVideo2-6B | ViT (6B) | Frozen; literature probe/inputs | 67.7 | 88.8 | Literature |
| VideoPrism | ViT-g (1B) | Frozen; literature probe/inputs | 68.5 | 87.6 | Literature |
| DINOv2 + registers | ViT-g (1.1B) | Frozen; frame-wise concatenation + 4-layer attentive probe | 50.7 | 83.6 | Common protocol: image |
| PEcore G | ViT-G (1.9B) | Frozen; frame-wise concatenation + 4-layer attentive probe | 55.4 | 88.5 | Common protocol: image |
| SigLIP2 | ViT (1.2B) | Frozen; frame-wise concatenation + 4-layer attentive probe | 49.9 | 87.3 | Common protocol: image |
| V-JEPA | ViT-H (600M) | Frozen; 4-layer attentive probe | 74.3 | 84.5 | Common protocol: video |
| InternVideo2 stage 2 | ViT (1B) | Frozen; interpolated video positions + 4-layer attentive probe | 69.7 | 89.4 | Common protocol: video |
| V-JEPA 2 ViT-L | ViT-L (300M) | Frozen; 4-layer attentive probe | 73.7 | 85.1 | Common protocol: video |
| V-JEPA 2 ViT-H | ViT-H (600M) | Frozen; 4-layer attentive probe | 74.0 | 85.3 | Common protocol: video |
| V-JEPA 2 ViT-g | ViT-g (1B) | Frozen; 4-layer attentive probe | 75.3 | 86.6 | Common protocol: video |
| V-JEPA 2 ViT-g384 | ViT-g (1B) | Frozen; 384px; SSv2 64Γ2Γ3, K400 16Γ8Γ3; same probe | 77.3 | 87.3 | Higher-resolution protocol |
| I-JEPA | Image encoder | Not evaluated in Table 4 | Unmeasured | Unmeasured | Context only |
| MAE | Image encoder | Conceptual pixel-reconstruction baseline; unmeasured here | Unmeasured | Unmeasured | Context only |
VideoMAEv2
Literature
InternVideo2-1B
Literature
InternVideo2-6B
Literature
VideoPrism
Literature
DINOv2 + registers
Common protocol: image
PEcore G
Common protocol: image
SigLIP2
Common protocol: image
V-JEPA
Common protocol: video
InternVideo2 stage 2
Common protocol: video
V-JEPA 2 ViT-L
Common protocol: video
V-JEPA 2 ViT-H
Common protocol: video
V-JEPA 2 ViT-g
Common protocol: video
V-JEPA 2 ViT-g384
Higher-resolution protocol
I-JEPA
Context only
MAE
Context only
V-JEPA 2's insight
Under the stated 256px protocol, ViT-g exceeds original V-JEPA ViT-H by 1 percentage points on SSv2 and 2.1 on K400. These system comparisons also change model scale and training data; they do not isolate a causal effect of the objective.
Trade-offs
Compare groups separately. Higher g384 scores also use 384px input and longer SSv2 clips. Literature results use different evaluation heads/inputs. MAE and I-JEPA are context rows with no measured Table 4 score. Classification accuracy alone does not establish temporal causality or robot control.
Key Results
The frozen classification values below come from PDF v1, Table 4. The 256px rows share the four-layer attentive-probe protocol, with SSv2 16Γ2Γ3 and K400 16Γ8Γ3 inputs (frames Γ temporal Γ spatial crops). The g384 row uses 384px input on both tasks and 64Γ2Γ3 SSv2 inputs. Encoders remain frozen; these are not fine-tuned scores.
| Encoder | SSv2 top-1 | K400 top-1 | Evaluation |
|---|---|---|---|
| V-JEPA ViT-H | 74.3% | 84.5% | Common 256px protocol |
| V-JEPA 2 ViT-g | 75.3% | 86.6% | Common 256px protocol |
| V-JEPA 2 ViT-g384 | 77.3% | 87.3% | Higher resolution and longer SSv2 clips |
| DINOv2 ViT-g with registers | 50.7% | 83.6% | Frame-wise features, common 256px protocol |
For the stated pair, common-protocol ViT-g minus original V-JEPA ViT-H is +1.0 SSv2 percentage points and +2.1 K400 percentage points. Data and architecture also differ, so these deltas do not isolate the objective. Literature rows in the comparison above use their own frozen probes/inputs and should be read separately. The 85.2 and 88.2 figures in Table 4 are six-task averages, not K400 scores.
Robotics is a separate evaluation: Table 2 reports mean pick-and-place success across the two labs of 80% for cups and 65% for boxes for V-JEPA 2-AC. These are two deployment labs, not a DROID benchmark; DROID supplies the post-training interaction data. Table 3 compares per-action planning time and success with Cosmos. The separate anticipation and video-QA evaluations appear in Tables 5 and 6, rather than in this common-protocol classification comparison.
Why V-JEPA 2 Matters
V-JEPA 2 demonstrates that self-supervised video models can bridge the gap from perception to action. Prior work in the JEPA family β I-JEPA, V-JEPA β focused on learning representations. V-JEPA 2 shows that the same latent space that understands video can also predict the consequences of actions and plan goal-directed behavior. Its robotics results rely on a separate action-conditioned predictor trained with interaction data; video pretraining alone does not identify robot actions or establish a general model of physics and causality.
The engineering implications are equally significant. V-JEPA 2-AC achieves useful robotic manipulation from 62 hours of robot video paired with actions and end-effector states β no reward engineering, no demonstration labeling, no simulation-to-real transfer. This dramatically lowers the barrier to deploying robots in new environments: collect suitable robot trajectories with actions and state measurements, post-train the predictor, and evaluate planning in the target setting. The progressive training strategy makes the compute cost tractable (7 GPU-years instead of 60), and the 3D-RoPE architecture supports the change in token coordinates during the trained resolution schedule. Together, these advances make V-JEPA 2 not just a stronger video encoder but a foundation for embodied AI systems that learn from watching and act by imagining.
V-JEPA 2.1: From Global Recognition to Dense Features
The 2026 follow-up, V-JEPA 2.1, asks whether a strong video representation also preserves useful information at each spatial location. Recognizing an action and locating an object boundary require different information.
Predict Visible Positions Too
The released training loop computes separate predictions and losses for masked and context positions. The EMA teacher still processes the complete input. V-JEPA 2.1 adds a weighted context term to the masked L1 objective:
Here C indexes visible positions; \hat h_i and h_i are predictor and teacher features. Context weights warm up during training and emphasize positions nearer masked regions. See Β§2.3.1.
Which positions get a direct prediction loss?
The encoder still sees visible context. V-JEPA 2.1 adds teacher-feature matching at those visible positions.
V-JEPA 2
Masked positions only
Cβ
Visible
No direct term
Mβ
Masked
L1 match
Cβ
Visible
No direct term
Mβ
Masked
L1 match
V-JEPA 2.1
Masked + weighted context
Cβ
Visible
Weighted L1
Mβ
Masked
L1 match
Cβ
Visible
Weighted L1
Mβ
Masked
L1 match
Preserve Global Understanding with Deep Supervision
The encoder exposes four representation levels. Their normalized features are concatenated, fused for the predictor, and matched against corresponding teacher levels. Local information can therefore receive training signals before the final block.
The first three rows of Table 1 use frozen ViT-L encoders: ADE20K uses a linear probe; SSv2 uses an attentive probe.
| Training recipe | ADE20K mIoU β | SSv2 top-1 β |
|---|---|---|
| V-JEPA 2 baseline | 22.2 | 72.8% |
| + weighted context loss | 33.8 | 62.5% |
| + multi-level prediction | 38.6 | 72.1% |
Context matching improves segmentation but hurts recognition on its own. Deep supervision recovers much of that recognition performance. These ablations precede model scaling and high-resolution cooldown; they are not the final ViT-G scores.
Give Images and Videos Native Tokenizers
The patch embedding modules provide spatial image patches and spatiotemporal video tubelets. The encoder routes the input through its modality-specific embedding, then uses shared transformer blocks with learned modality information. Images need not be repeated into a long static clip. Here βmulti-modalβ means images and videos, not text supervision.
The official release includes 384px ViT-B/L/g/G checkpoints: approximately 80M, 300M, 1B and 2B parameters. B and L are distilled variants. Model size, pretraining data and evaluation inputs matter when comparing these checkpoints with the original V-JEPA 2 results above.
Better features can support downstream prediction, but an encoder alone is not an action-conditioned robot planner. The runnable implementation and native feature inspection above remain original V-JEPA 2; this follow-up section does not relabel their outputs as V-JEPA 2.1.
Key Takeaways
-
Mask denoising predicts missing representations β V-JEPA 2 drops masked tubelets before the encoder, adds learned queries in the predictor, and regresses masked-only teacher features with L1. This continues V-JEPAβs objective; 3D-RoPE and scaling are the key changes.
-
Progressive training saves 8.4Γ compute β training at low resolution for 95% of iterations and scaling up during a brief cooldown transfers coarse understanding to fine detail, using 3D-RoPE coordinates and additional training at the increased resolution.
-
Frozen video classification improves at scale β ViT-g reaches 75.3% SSv2 under the common 256px protocol; ViT-g384 reaches 77.3% with larger inputs and longer clips. These scores and anticipation results do not establish temporal causality.
-
Latent planning is 15Γ faster than pixel generation β V-JEPA 2-AC plans robot actions in 16 seconds by searching latent space, while pixel-generation approaches like Cosmos require 4+ minutes to render and evaluate candidate futures.
-
62 hours of robot interaction data supports zero-shot deployment β the action-conditioned world model learns state transitions from video paired with recorded actions and end-effector states and deploys zero-shot on physical robots it has never seen, achieving mean pick-and-place success across the two labs of 80% for cups and 65% for boxes without task labels or reward signals.
-
V-JEPA 2.1 directly supervises visible features too β weighted context matching and intermediate-layer supervision improve dense representations alongside global recognition.
Related Reading
-
V-JEPA 2.1 paper β Dense prediction, deep supervision and native image/video tokenization
-
V-JEPA β The direct predecessor: latent video prediction with ViT-L on VideoMix2M
-
I-JEPA β Joint-embedding prediction for images, the architectural foundation
-
DINOv2 β Universal frozen image features via DINO + iBOT at scale
-
BEiT β Discrete token prediction that pioneered masked image modeling
-
MAE β Pixel reconstruction for masked image modeling, the approach V-JEPA 2 outperforms
-
DINO β Self-distillation with Vision Transformers and momentum teacher
-
SimCLR β Contrastive learning framework for visual representations
-
MoCo β Momentum contrast, architectural precursor to EMA-based methods
-
BYOL β Non-contrastive learning without negative pairs
-
VICReg β Variance-invariance-covariance regularization for SSL
