Skip to main content

Visual Representations and Image Encoders

Follow pixels into image features, compare spatial maps with pooled vectors, and see why image encoders expose different outputs for recognition and localization.

A classifier can recognize that an image contains a car, while a detector must also say where it is. Both can begin with the same kind of intermediate result: a visual representation, a collection of numbers computed from the image. The important choice is what those numbers retain before the final task makes its prediction.

Start before the final class scores: what does an image encoder produce, and when should its output remain spatial? Image classification and CNN feature maps provides a complementary example that follows features through to a prediction.

From pixels to features

Pixels record an image's sampled color or intensity values. An RGB image of height HH and width WW has three channels, so one way to write its shape is H×W×3H \times W \times 3. A feature channel has a different role: it is a coordinate of a computed representation, rather than a red, green, or blue measurement.

An image encoder is the function that computes that representation. For a learned encoder with parameters θ\theta:

Z=fθ(X)Z = f_{\theta}(X)

Here XX is the image and ZZ is its representation. Training changes θ\theta so that the features support the training objective. A separate head can then turn those features into class scores, bounding boxes, or another task's outputs. The boundary between encoder and head is an architectural choice; calling a network an encoder does not specify how it was trained.

A feature vector is an ordered list of numbers. A spatial representation stores one such vector at each output location. Omitting the batch dimension, a CNN feature map might have shape H′×W′×DH' \times W' \times D: H′W′H'W' locations, each with DD feature coordinates. The output grid is often smaller than the input because of striding or downsampling.

The vector at a location can depend on a whole region of input pixels. Deeper layers can combine wider context, and transformer features can exchange information across image locations. A spatial feature's grid position therefore identifies an output location; it does not mean that the vector describes only one isolated pixel.

Inspect what a summary discards

Move the same pattern between the four regions below. The responding feature cell moves, but its average over all four cells stays unchanged. Then switch between vertical and horizontal splits: they have the same average brightness, yet different directional features.

What survives an image summary?
Region pattern
Move region
Feature channel

Input: 4 × 4 × 1

1
1
0
0
1
1
0
0
0
0
0
0
0
0
0
0

0 is dark; 1 is bright. Each nonoverlapping 2 × 2 region produces one feature vector.

Spatial output: 2 × 2 × 3

Top left1
Top right0
Bottom left0
Bottom right0

Showing brightness. Each cell has three values, even though only one channel is colored.

Moved region
Top left
Its feature vector
[1, 0, 0]
Mean-pooled vector
[0.25, 0, 0]

Vector order: brightness, vertical contrast, horizontal contrast.

Moving this pattern changes which spatial cell responds. Its mean-pooled vector stays the same because averaging ignores the order of the four vectors.

Authored grayscale inputs and analytically calculated, fixed descriptors. No learned weights, inference, or measured model outputs. Contrasts are measured only inside each 2 × 2 region; boundaries between regions are ignored.

This example deliberately uses a grayscale 4×4×14 \times 4 \times 1 image so every input value is visible. Its fixed encoder groups pixels into four nonoverlapping 2×22 \times 2 regions. For a region with top row a,ba,b and bottom row c,dc,d, its three features are:

z=[a+b+c+d4,  ∣b+d−a−c2∣,  ∣c+d−a−b2∣]z = \left[ \frac{a+b+c+d}{4},\; \left|\frac{b+d-a-c}{2}\right|,\; \left|\frac{c+d-a-b}{2}\right| \right]

These coordinates measure brightness, left-right contrast, and top-bottom contrast. “Vertical contrast” responds to a vertical boundary between dark and bright columns. The absolute values ignore which side is brighter. A solid bright region produces [1,0,0][1,0,0]; a vertical split produces [0.5,1,0][0.5,1,0]. Region boundaries are excluded from these local calculations.

The resulting spatial tensor has shape 2×2×32 \times 2 \times 3. Flattening just its two spatial dimensions would give a sequence of four three-dimensional vectors. Keeping the row-major order allows each sequence entry to be mapped back to its grid location. Reshaping alone does not average or discard values.

The named channels make this arithmetic easy to inspect. They are authored descriptors, not a claim that a trained network develops three correspondingly named channels. Learned representations usually distribute useful information across many coordinates; a single coordinate need not mean “car,” “wheel,” or even “vertical edge.”

Spatial features and an image-level vector

Global average pooling averages each feature coordinate across all spatial locations. If the spatial vectors are z1,…,zN∈RDz_1,\ldots,z_N \in \mathbb{R}^{D}, it produces one vector:

g=1N∑i=1Nzi∈RDg = \frac{1}{N}\sum_{i=1}^{N} z_i \in \mathbb{R}^{D}

For the solid pattern, one region has [1,0,0][1,0,0] and the other three have [0,0,0][0,0,0]. Their mean is [0.25,0,0][0.25,0,0] wherever the bright region sits. No downstream function receiving only that vector can distinguish these four placements, because its input is identical in every case.

That is an exact property of this example. It is not proof that arbitrary images shifted through a real encoder will produce identical summaries. Borders, sampling, positional information, and context can change the spatial vectors themselves. Even an averaged vector can contain layout information already encoded into its coordinates; averaging simply cannot recover the order of the vectors being averaged.

An image-level vector is useful when the next operation needs one descriptor per image, such as a classification head or image retrieval system. It is still a representation, not automatically a class prediction or a probability. Detection and segmentation need location-dependent evidence, so their heads commonly consume spatial features instead of relying only on an image summary.

Different encoders expose different outputs

CNNs commonly construct spatial feature maps with shared convolutional operations. A Vision Transformer instead begins by representing image patches as vectors and processes their sequence. In the original ViT, learned position embeddings are added to the patch embeddings, and the final state of an extra class token supplies the image-level representation. A class-token summary is learned through attention; it is not the arithmetic mean used above. Vision Transformer, Section 3.1

An encoder can expose both kinds of output. DINOv2's official forward_features method returns normalized class-token features and normalized patch-token features separately. The patch vectors retain their association with patch locations while carrying context acquired during encoding. Choosing the class-token output versus the patch-token output changes what the downstream system receives. DINOv2 official encoder implementation

For a detection example, original DETR takes a CNN backbone's spatial features into a transformer rather than first reducing the image to one classification vector. Its official backbone wrapper extracts the last residual stage, or intermediate stages when requested. DETR paper, official backbone implementation

Follow the representation into the next task

The next question is how a transformer gets those patch vectors in the first place. Image patches and patch embeddings follows an image into a sequence and works through the dimensions of its learned projection.

For detection, keep the spatial features and continue to Object queries in DETR: a set of decoder slots reads image evidence to make object predictions. For representation learning, Self-supervised learning and masked prediction asks how an encoder can learn useful features when the training targets come from the data itself.

To reuse an encoder for a new task, continue with transfer learning and fine-tuning. Then evaluating visual representations separates what frozen features support from what downstream adaptation can learn.

If you found this explanation helpful, consider sharing it with others.

Mastodon