Skip to main content

Evaluating Visual Representations

Evaluate frozen image features with nearest neighbors and linear probes, separate training from validation and test data, and understand what each protocol can establish.

Transfer learning and fine-tuning separates reusing a fixed encoder from adapting its features. That distinction also determines what an evaluation can establish. Whether pretraining used category labels or a self-supervised objective, a low training loss does not yet show that the features will recognize a new category, retrieve the same object, or locate a boundary.

Evaluation makes that next claim testable. Start with an encoder, specify a downstream task, and decide how much adaptation that task is allowed. A good representation makes the information needed for that task accessible under the stated protocol. “Good” needs a task, a dataset, a metric, and an adaptation rule.

Hold the encoder fixed

Helpful background

Transfer learning and fine-tuning

Compare a frozen encoder with partial and full fine-tuning, tracing parameter updates separately from gradient propagation and normalization state.

Open full page

An image encoder produces a feature vector z=fθ(x)z = f_{\theta}(x). A downstream classifier turns that vector into a predicted class. In a frozen-feature evaluation, the encoder parameters θ\theta stay fixed while the downstream rule is built and evaluated.

Freezing means more than excluding parameters from gradient updates. Feature extraction also uses the encoder's evaluation behavior: dropout is inactive and batch-normalization layers with tracked running statistics use their stored values. Layers configured to use batch statistics even during evaluation can make an image's features depend on its batch companions; record and control that behavior. Keep the chosen checkpoint, image preprocessing, output layer, and pooling rule fixed too. Otherwise the purportedly fixed representation changes during the comparison.

Labels can still be used to evaluate self-supervised features. They supervise a downstream classifier or score its predictions; they need not have supplied the encoder's pretraining targets. The separate question is how much labeled downstream data the evaluation uses.

Two common frozen-feature protocols make different demands on the same vectors: nearest neighbors ask whether useful examples are close, while a linear probe asks whether a learned linear readout can separate the categories.

Nearest neighbors test local structure

For k-nearest-neighbor classification, store the training features together with their class labels. This labeled reference pool is sometimes called a feature bank. For a new query, find its kk closest reference vectors and combine their labels. The simplest version assigns the majority class, giving each neighbor one vote. No classifier weights are fitted, but labels and a reference pool are still required.

For Euclidean distance, the distance between vectors zz and uu is:

d(z,u)=∑j=1D(zj−uj)2d(z,u) = \sqrt{\sum_{j=1}^{D}(z_j-u_j)^2}

The illustration uses twelve training features and eight separate validation queries. Switch to alternating features and increase kk from 3 to 7: the neighborhood reaches into other clusters. Change the query to inspect which training samples supply the votes.

A frozen-feature neighborhood
Features
3
Q1

Feature 2 ↑

Separated authored features. Query Q1, true class A, predicted class A, with 3 training neighbors.

Feature 1 →

A: training circlesB: training squaresDiamond: selected query

Q1 = [-1.26, -1.16]. Prediction A is correct. Rings mark the 3 neighbors. Background cells sample the kNN prediction; the query never enters the training pool.

Authored two-dimensional features, not pretrained model outputs. Unnormalized Euclidean distance and equal votes; all eight validation queries are scored. No test set is used in this illustration.

These are authored coordinates, not features extracted from images. Each arrangement keeps the same sample identities, labels, and split while changing their feature positions. The displayed prediction, neighbor distances, and validation accuracy are computed from those coordinates. The background samples the classifier's decisions on a grid; it is not a fitted linear boundary.

Accuracy here is the fraction of the eight validation queries classified correctly:

accuracy=correct predictionsnumber of evaluated queries\mathrm{accuracy} = \frac{\text{correct predictions}}{\text{number of evaluated queries}}

Small neighborhoods succeed in both arrangements. That does not make every neighborhood size equally useful: a large neighborhood can mix unrelated groups. The figure lets validation guide this choice. It contains no test set and makes no claim about which pretrained image encoder is better.

Real protocols may use cosine similarity and distance-weighted votes instead of this example's raw Euclidean distance and equal votes. DINO's official kNN evaluator normalizes both feature sets and weights neighbors using exponentiated similarity divided by a temperature. Its reported result therefore describes a particular procedure, rather than an unspecified “kNN score.” DINO official kNN implementation

A linear probe tests a different readout

A linear probe fits class scores from frozen features:

s=Wz+b,y^=arg max⁡cscs = Wz+b, \qquad \hat{y} = \operatorname*{arg\,max}_{c} s_c

Here WW contains one row per class and bb contains class biases. The classifier learns these parameters using training features and labels, commonly by minimizing a classification loss such as cross-entropy. The encoder remains unchanged. The SimCLR paper is one example of evaluating self-supervised representations with a learned linear classifier. SimCLR paper

For a concrete score calculation, consider the prescribed, rather than fitted, weights below. The first row scores A and the second scores B:

W=[−1010],b=[00]W = \begin{bmatrix}-1&0\\1&0\end{bmatrix},\qquad b=\begin{bmatrix}0\\0\end{bmatrix}

For query Q1 in the separated arrangement, z=[−1.26,−1.16]z=[-1.26,-1.16], so sA=1.26s_A=1.26 and sB=−1.26s_B=-1.26: A wins. The equality boundary is z1=0z_1=0. This calculation illustrates the readout; an actual probe fits its weights on training examples. A nonlinear head on a fixed encoder is still frozen-feature evaluation, but it is not a linear probe.

For two classes in two dimensions, the two scores are equal along a line. The separated arrangement above admits a line between its left and right groups. The alternating arrangement places class A at opposite corners and class B at the other two corners: no single line separates those four cluster centers. Local neighbors can still classify points near each cluster correctly.

This is why a nearest-neighbor score and a linear-probe score answer different questions. A representation can have useful local structure without putting each class on one side of a linear boundary. A linear probe can learn coordinate weights and ignore some unhelpful directions; nearest-neighbor distances use the chosen feature geometry directly. Neither score alone measures all information in the features.

The probe's training recipe matters as well. DINOv2's linear evaluation varies the learning rate, which output layers are used, and whether pooled patch features are concatenated with the class token. These choices change the evaluated readout or its input, so comparisons should report them. DINOv2, Appendix B.3

Keep training, selection, and reporting separate

A score is useful only if the evaluated queries were kept separate from the information used to build the predictor. For an ordinary supervised downstream evaluation, give each split a defined role:

SplitAllowed role
TrainingFit the linear head and any learned preprocessing; supply the kNN reference pool.
ValidationChoose k, distance or voting settings, regularization, learning rate, output features, and other candidate settings.
TestScore the locked procedure after those choices are complete.

Do not place a query in its own reference pool. A one-neighbor classifier could then retrieve that same sample at distance zero. Separate files are not enough either: duplicate images, adjacent video frames, or repeated views of the same subject can cross the split boundary. Split at the independent unit relevant to the intended use, such as person, video, camera, or collection site, before fitting or selecting anything.

Preprocessing follows the same boundary. If a scaler, PCA projection, or feature-selection step learns statistics, fit it on training data and apply that fitted transform to validation and test features. Learning it from the combined corpus lets held-out examples influence the predictor. The official scikit-learn guidance documents both this leakage and the need to apply consistent transformations. Data leakage and preprocessing

Per-vector L2 normalization is different: z~=z/∥z∥2\tilde z=z/\lVert z\rVert_2 uses that vector's own length, without fitting statistics across the dataset. Apply the same rule to reference and query vectors. For nonzero unit vectors:

∥z~−u~∥22=2−2z~Tu~\lVert\tilde z-\tilde u\rVert_2^2 = 2-2\tilde z^{\mathsf T}\tilde u

Thus Euclidean neighbor ordering and cosine-similarity ordering agree on unit vectors. Without normalization, magnitude can change the ordering. Normalization is part of the protocol, not a cosmetic plot setting, and zero vectors need an explicit handling rule.

A reproducible result identifies the checkpoint and feature output, data splits and overlap checks, preprocessing, classifier recipe, selection procedure, metric, and number of evaluated examples. For an imbalanced task, overall accuracy can hide poor performance on a rare class; inspect class-level results and choose metrics that reflect the intended use. A tiny validation score is also a noisy estimate, not evidence of broad reliability.

Some published benchmarks use a dataset named “validation” as their reporting split and may select settings on it. Read the authors' actual protocol before comparing numbers. In a new evaluation with a dedicated test set, repeated test-driven changes turn that set into another selection set, even if no encoder gradients ever touch it.

Classification does not establish every capability

Avoiding representation collapse preserves distinctions between examples. A downstream evaluation asks whether the preserved distinctions are useful for a particular task. Nonconstant features can still emphasize backgrounds, camera identity, or another shortcut that fails on new data.

An image-level classification score also does not establish localization quality. As Visual representations and image encoders explains, pooling can discard spatial information. Detection or segmentation needs a suitable spatial output, a specified head, and task-specific evaluation. Image retrieval needs a defined query and gallery split and a relevance metric. Transfer to a new domain needs examples from that domain. DINOv2 accordingly evaluates image classification, instance retrieval, and dense prediction separately. DINOv2, Section 7

Use I-JEPA and DINOv2 to connect these protocols to specific pretraining methods. Their training objectives explain how features are learned; their evaluation protocols explain which downstream claims the evidence supports.

When adapting the features is allowed

A frozen probe measures what a fixed encoder makes accessible to its readout. Fine-tuning changes some or all of the encoder parameters, so it asks how well the pretrained model can adapt under a stated training budget. That is a different experiment, with its own learning rates, trainable layers, validation choices, and test result.

Return to Transfer learning and fine-tuning when deciding which parameters may change. To evaluate a concrete method next, follow the frozen-feature and downstream-task experiments in DINOv2, keeping each result attached to its feature output, readout, and data split.

If you found this explanation helpful, consider sharing it with others.

Mastodon