Skip to main content

Target Encoders, EMA, and Stop-Gradient

Follow how a learned target encoder supplies feature targets, how stop-gradient separates backward paths, and how exponential moving averages update its parameters.

Self-supervised masked prediction can ask a network to predict features of a hidden image region instead of its pixels. That leaves a question: who produces those features while the network is still learning?

A target encoder is a second encoder that supplies the comparison features. Its outputs are learned representations, not immutable teacher labels. As its parameters change, the same image can produce a different target. The useful distinction is between keeping a target fixed during one backward pass and keeping it fixed for all of training.

Two branches, two jobs

The context encoder processes the visible information. A predictor transforms its output into the representation being predicted. The target encoder processes the information needed to construct the target. A loss measures the disagreement between prediction and target.

In I-JEPA, Section 3, the context encoder sees context patches with overlapping target regions removed. The target encoder sees the full training image, and target-region features are selected from its outputs afterward. Its target features can therefore include surrounding image context. Encoding a cropped target block in isolation would be a different computation.

Let fθf_\theta be the context encoder, gϕg_\phi the predictor, and fξf_\xi the target encoder. For context input xx and target-producing input yy, a simplified feature-matching objective is:

z^=gϕ(fθ(x)),z=fξ(y),L=12∥z^−sg⁡(z)∥22.\hat z = g_\phi(f_\theta(x)), \qquad z = f_\xi(y), \qquad L = \frac{1}{2}\|\hat z - \operatorname{sg}(z)\|_2^2.

Here θ\theta, ϕ\phi and ξ\xi are parameter sets. The symbol sg⁡\operatorname{sg} means stop-gradient. This equation describes the analytical example below; it omits real model details such as feature normalization and averaging over target patches.

Stop-gradient changes the backward pass

During the forward pass, stop-gradient returns exactly the same numerical feature values:

sg⁡(z)=z.\operatorname{sg}(z) = z.

During the backward pass, it supplies zero derivative along that path:

∂sg⁡(z)∂z=0.\frac{\partial\operatorname{sg}(z)}{\partial z} = 0.

This is a rule for automatic differentiation, not the ordinary derivative of an identity function. It lets the prediction be compared with a target without letting this loss change the target encoder through backpropagation. Gradients still flow through the predictor into the context encoder. A detached target does not detach the prediction branch.

For example, if prediction is 0.50.5 and target is 33, the half squared error is 3.1253.125. The prediction derivative is 0.5−3=−2.50.5-3=-2.5. Without the cut, the target-value derivative would be 2.52.5; with stop-gradient, that path contributes zero. The forward loss is identical in both cases.

EMA changes parameters after the optimizer

The optimizer updates the context encoder and predictor using their gradients. The target encoder then follows the updated context encoder through an exponential moving average, or EMA:

θt+1=θt−η∇θtLt,ϕt+1=ϕt−η∇ϕtLt,\theta_{t+1} = \theta_t - \eta\nabla_{\theta_t}L_t, \qquad \phi_{t+1} = \phi_t - \eta\nabla_{\phi_t}L_t,
ξt+1=mξt+(1−m)θt+1.\xi_{t+1} = m\xi_t + (1-m)\theta_{t+1}.

The first equations use simple gradient descent to make the distinction visible; an actual training run can use a more elaborate optimizer. The EMA equation averages corresponding encoder parameters, not predictions, gradients, or predictor parameters. The two encoders must have compatible parameter shapes. They commonly begin as copies, so their weights initially agree.

The momentum mm controls how much old target state survives each update. At m=0m=0, the target copies the newly updated context weights. At m=1m=1, it stays frozen. Between these endpoints it follows more gradually. With constant momentum, an old contribution is multiplied by another factor of mm each step, which gives the moving average its exponential weighting.

The explorer separates one forward pass from its backward, optimizer, and EMA operations. Switch to two coordinates to see that EMA acts on every parameter coordinate, then compare momentum zero, 0.90.9, and one.

Two update paths
Example
Step
0.90

Forward. Both branches compute values. Stop-gradient preserves the target features exactly.

Context → encoder θ → predictor p → prediction

x = [1]; prediction = [0.5]

Target input → encoder ξ → stop-gradient → target

y = [2]; sg(target) = [3]

L = ½ Σ(prediction − target)² = 3.125

Values shown in the forward paths and loss come from the start of this step. Updated parameters affect the next forward pass.

Authored analytical example with diagonal linear encoders and a scalar predictor, not measured training or an I-JEPA simulation. Steps are manual; changing a control recomputes this same single step.

In the scalar example, context weight 11 and predictor 0.50.5 produce prediction 0.50.5 from input 11. Target weight 1.51.5 produces target 33 from input 22. The context gradient is −1.25-1.25 and the predictor gradient is −2.5-2.5. With learning rate 0.10.1, their updated values are 1.1251.125 and 0.750.75. At momentum 0.90.9, the target weight becomes 0.9(1.5)+0.1(1.125)=1.46250.9(1.5)+0.1(1.125)=1.4625. It changes despite receiving no loss gradient.

Read the objective and implementation separately

The I-JEPA paper's loss section presents squared L2 feature prediction. The official training implementation instead calls F.smooth_l1_loss(z, h), with PyTorch's default mean reduction. Smooth L1 has a quadratic region near zero and a linear region for larger residuals. It is not the same numerical objective as squared L2 or the half squared sum used in this explorer.

The same code makes the two update paths visible: target computation runs under torch.no_grad(), target parameters have requires_grad = False, the optimizer updates the context encoder and predictor, and a later no-gradient block performs the parameter EMA. Target features are normalized before target positions are selected. These details matter when reproducing a training run; the small example isolates update arithmetic rather than reproducing that run.

A slowly moving target can still collapse

A collapsed representation assigns the same features to unrelated inputs. If the predictor always returns that same feature vector, feature-matching error can be zero even though the representation has discarded useful distinctions.

Neither stop-gradient nor EMA alone rules this out. Stop-gradient specifies where the current loss can send gradients; EMA specifies how one parameter set follows another. Neither equation requires different inputs to remain distinguishable. Even constant identical encoders can satisfy the averaging rule. The predictor, architecture, masking task, normalization and optimization interact, and useful representations require empirical evaluation. I-JEPA's reported behavior concerns its complete method, not a general guarantee from EMA.

The next question is what useful distinctions survive training. Continue with representation collapse for that failure mode, or gradient flow to follow backward paths through larger networks. With these update roles separated, the I-JEPA walkthrough connects target construction, masking and prediction in the full image-learning method.

If you found this explanation helpful, consider sharing it with others.

Mastodon