Self-supervised masked prediction can ask a network to predict features of a hidden image region instead of its pixels. That leaves a question: who produces those features while the network is still learning?
A target encoder is a second encoder that supplies the comparison features. Its outputs are learned representations, not immutable teacher labels. As its parameters change, the same image can produce a different target. The useful distinction is between keeping a target fixed during one backward pass and keeping it fixed for all of training.
Two branches, two jobs
The context encoder processes the visible information. A predictor transforms its output into the representation being predicted. The target encoder processes the information needed to construct the target. A loss measures the disagreement between prediction and target.
In I-JEPA, Section 3, the context encoder sees context patches with overlapping target regions removed. The target encoder sees the full training image, and target-region features are selected from its outputs afterward. Its target features can therefore include surrounding image context. Encoding a cropped target block in isolation would be a different computation.
Let be the context encoder, the predictor, and the target encoder. For context input and target-producing input , a simplified feature-matching objective is:
Here , and are parameter sets. The symbol means stop-gradient. This equation describes the analytical example below; it omits real model details such as feature normalization and averaging over target patches.
Stop-gradient changes the backward pass
During the forward pass, stop-gradient returns exactly the same numerical feature values:
During the backward pass, it supplies zero derivative along that path:
This is a rule for automatic differentiation, not the ordinary derivative of an identity function. It lets the prediction be compared with a target without letting this loss change the target encoder through backpropagation. Gradients still flow through the predictor into the context encoder. A detached target does not detach the prediction branch.
For example, if prediction is and target is , the half squared error is . The prediction derivative is . Without the cut, the target-value derivative would be ; with stop-gradient, that path contributes zero. The forward loss is identical in both cases.
EMA changes parameters after the optimizer
The optimizer updates the context encoder and predictor using their gradients. The target encoder then follows the updated context encoder through an exponential moving average, or EMA:
The first equations use simple gradient descent to make the distinction visible; an actual training run can use a more elaborate optimizer. The EMA equation averages corresponding encoder parameters, not predictions, gradients, or predictor parameters. The two encoders must have compatible parameter shapes. They commonly begin as copies, so their weights initially agree.
The momentum controls how much old target state survives each update. At , the target copies the newly updated context weights. At , it stays frozen. Between these endpoints it follows more gradually. With constant momentum, an old contribution is multiplied by another factor of each step, which gives the moving average its exponential weighting.
The explorer separates one forward pass from its backward, optimizer, and EMA operations. Switch to two coordinates to see that EMA acts on every parameter coordinate, then compare momentum zero, , and one.
Forward. Both branches compute values. Stop-gradient preserves the target features exactly.
Context → encoder θ → predictor p → prediction
x = [1]; prediction = [0.5]
Target input → encoder ξ → stop-gradient → target
y = [2]; sg(target) = [3]
L = ½ Σ(prediction − target)² = 3.125
Values shown in the forward paths and loss come from the start of this step. Updated parameters affect the next forward pass.
Authored analytical example with diagonal linear encoders and a scalar predictor, not measured training or an I-JEPA simulation. Steps are manual; changing a control recomputes this same single step.
In the scalar example, context weight and predictor produce prediction from input . Target weight produces target from input . The context gradient is and the predictor gradient is . With learning rate , their updated values are and . At momentum , the target weight becomes . It changes despite receiving no loss gradient.
Read the objective and implementation separately
The I-JEPA paper's loss section presents squared L2 feature prediction. The official training implementation instead calls F.smooth_l1_loss(z, h), with PyTorch's default mean reduction. Smooth L1 has a quadratic region near zero and a linear region for larger residuals. It is not the same numerical objective as squared L2 or the half squared sum used in this explorer.
The same code makes the two update paths visible: target computation runs under torch.no_grad(), target parameters have requires_grad = False, the optimizer updates the context encoder and predictor, and a later no-gradient block performs the parameter EMA. Target features are normalized before target positions are selected. These details matter when reproducing a training run; the small example isolates update arithmetic rather than reproducing that run.
A slowly moving target can still collapse
A collapsed representation assigns the same features to unrelated inputs. If the predictor always returns that same feature vector, feature-matching error can be zero even though the representation has discarded useful distinctions.
Neither stop-gradient nor EMA alone rules this out. Stop-gradient specifies where the current loss can send gradients; EMA specifies how one parameter set follows another. Neither equation requires different inputs to remain distinguishable. Even constant identical encoders can satisfy the averaging rule. The predictor, architecture, masking task, normalization and optimization interact, and useful representations require empirical evaluation. I-JEPA's reported behavior concerns its complete method, not a general guarantee from EMA.
The next question is what useful distinctions survive training. Continue with representation collapse for that failure mode, or gradient flow to follow backward paths through larger networks. With these update roles separated, the I-JEPA walkthrough connects target construction, masking and prediction in the full image-learning method.
Related concepts
Understand where self-supervised targets come from, how masked prediction creates a learning task, and why predicting pixels differs from predicting learned features.
Understanding complete, dimensional, and cluster collapse — the failure modes that every self-supervised method must prevent. Learn why collapse happens and how contrastive, asymmetric, regularization, and masking approaches solve it.
Move a pretrained encoder to a downstream task: trace frozen features, head training, partial and full fine-tuning, learning rates, and BatchNorm state.
Understand contrastive loss for representation learning: interactive demos of InfoNCE, triplet loss, and embedding space clustering with temperature tuning.
Learning low-dimensional vector representations of graphs through random walks, DeepWalk, Node2Vec, and skip-gram models
Evaluate frozen image features with nearest neighbors and linear probes, separate training from validation and test data, and understand what each protocol can establish.
