Image patches and patch embeddings explain how an image becomes a sequence of vectors. A vector sequence is a useful input format, but it leaves a learning question unanswered: what should the encoder learn to represent?
In image classification, a human-provided category supplies the answer. The model predicts “dog,” compares that prediction with the recorded label, and changes its parameters to reduce the error. An image collection without category annotations cannot provide that particular training signal. It can still provide other targets. Hide part of an image, for example, and the original image tells us what was hidden.
Self-supervised learning builds a training task from the data itself. Masked prediction is one such task: learn to predict something about a hidden region from visible context. The crucial design choice is what that “something” should be.
Supervision is a comparison, not necessarily a category
A prediction is the model's output. A target is the value it is trained to match. A loss measures their disagreement and supplies the signal used to update trainable parameters. Supervision describes this learning signal; it does not require a human to assign a category to every example.
A class label such as “dog” is one kind of target. Hidden pixel values are another. A vector produced by another network can also serve as a target. Calling all three “labels” can obscure the difference between an external annotation and a target constructed during training.
Self-supervised pretraining therefore does not mean training without an objective or without targets. It means constructing the supervision from the available data rather than requiring task annotations for that pretraining objective. A later classifier can still use human labels, and evaluation may use annotated benchmarks. The method's name does not imply that its entire development or evaluation avoids labels.
The point of pretraining is usually to obtain a useful encoder, rather than to deploy the pretraining prediction head. A representation learned from unlabeled images might later support classification, retrieval, or another task. Whether it actually helps must be evaluated on that task; solving an invented prediction problem is not sufficient evidence by itself.
Hiding patches creates an information gap
Imagine a four-by-four image grid. Choose a block of four patches as the target region and give the context encoder only the other twelve. A predictor receives the context representation and the positions it should predict. The hidden source information supplies targets for those positions.
The separation matters. If the context path directly receives the target patch content, it may learn a copying shortcut rather than infer missing information. Target positions are useful conditioning information: they specify where the requested output belongs without revealing what occupies that location.
Mask shape also changes the task. A tiny hole surrounded by similar texture may be filled using nearby colors. A larger missing region can require relationships across more of the image. This is a motivation for structured masking, not a guarantee that every learned feature will become semantic. Masking, architecture and the objective work together.
Pixels and features ask different questions
A pixel reconstruction target asks for the source values in the hidden region. For a color patch, these might be its red, green and blue channel intensities. A successful reconstruction must account for visual detail, including variation that a downstream category task might not need. Masked Autoencoders are a concrete example of learning through reconstruction of masked pixels. MAE paper
A feature prediction target asks for a representation of the hidden region. Its coordinates are learned quantities rather than a fixed vocabulary such as “dog” or “wheel.” A feature vector can preserve some distinctions while discarding others. This creates room for a training objective that does not demand every detail of the pixels, but a learned vector is not automatically an object description or a human-readable concept.
In the illustration below, switch between pixel and feature targets, then change the source texture. The feature rule deliberately ignores texture so the difference is inspectable. Moving the target block changes which positions are supervised.
Context input: 12 visible patches
Full source: 16 patches
Numbers identify patch positions. Amber marks the target block; teal marks visible context. The symbol ≋ indicates changed source texture.
- Target construction
- Read hidden source pixels
- First target patch
- 4 → [0.20, 0.20, 0.20, 0.20]
- Target scalars
- 4 patches × 4 values = 16
Changing texture changes the pixel values to reconstruct.
Authored masking and feature illustration, not model inference. No predictor is run and no accuracy or efficiency is measured.
The explorer's single feature coordinate is the mean brightness of the four source pixel values in a patch. Balanced bright and dark stripes change individual pixels while preserving their mean. This fixed descriptor demonstrates selective information retention; it is not a learned encoder and does not use surrounding context. I-JEPA instead learns its feature representation. The smaller vector size is also just a teaching choice. Predicting features does not inherently mean fewer output coordinates or measured faster training.
How I-JEPA constructs a learned target
I-JEPA uses a context encoder, a predictor, and a target encoder. The context encoder processes visible patches with target regions excluded. The target encoder processes the full image view, including the regions hidden from the context path. Target-block features are selected from that full-image representation afterward. These features can therefore incorporate information from the surrounding image; they are not encodings of independently cropped hidden patches. I-JEPA paper, official training loop
This asymmetry is intentional. The predictor must infer a representation using incomplete input, while the target branch has the information needed to define what it should match. Training has access to the complete image even though one branch sees only part of it. There is no requirement that both branches have equally restricted inputs.
A compact way to describe the target is:
Here is the full image view, is the target encoder, and identifies the target patch positions. Selection happens after encoding. The predictor produces corresponding vectors from the context representation and target positions; training compares the predictions with these target vectors.
Unlike recorded class annotations, these targets can change as the target encoder's parameters change. “Self-supervised” explains their source, while “learned” explains why their representation is not fixed throughout training.
A moving target needs an update rule
Learning both sides introduces a problem that pixel reconstruction does not have in the same form. If the encoder and predictor output the same constant for every image, the outputs can agree while carrying no useful information. Agreement alone does not establish a good representation. This failure is called representation collapse.
I-JEPA separates the update paths: the target branch does not receive gradients from the prediction loss, and its parameters follow an exponential moving average of the context encoder's parameters. These choices stabilize how the targets evolve. They do not, on their own, prove that collapse is impossible; the complete training design and empirical evaluation matter. I-JEPA paper, official training loop
The next concept, Target Encoders, EMA, and Stop-Gradient, follows those two update paths separately. Once their roles are clear, the I-JEPA paper walkthrough connects masking, learned targets and parameter updates into one training system.
Related concepts
Follow how a learned target encoder supplies feature targets, how stop-gradient separates backward paths, and how exponential moving averages update its parameters.
Understanding complete, dimensional, and cluster collapse — the failure modes that every self-supervised method must prevent. Learn why collapse happens and how contrastive, asymmetric, regularization, and masking approaches solve it.
Understand contrastive loss for representation learning: interactive demos of InfoNCE, triplet loss, and embedding space clustering with temperature tuning.
Learning low-dimensional vector representations of graphs through random walks, DeepWalk, Node2Vec, and skip-gram models
Move a pretrained encoder to a downstream task: trace frozen features, head training, partial and full fine-tuning, learning rates, and BatchNorm state.
Evaluate frozen image features with nearest neighbors and linear probes, separate training from validation and test data, and understand what each protocol can establish.
