A classifier can recognize that an image contains a car, while a detector must also say where it is. Both can begin with the same kind of intermediate result: a visual representation, a collection of numbers computed from the image. The important choice is what those numbers retain before the final task makes its prediction.
Start before the final class scores: what does an image encoder produce, and when should its output remain spatial? Image classification and CNN feature maps provides a complementary example that follows features through to a prediction.
From pixels to features
Pixels record an image's sampled color or intensity values. An RGB image of height and width has three channels, so one way to write its shape is . A feature channel has a different role: it is a coordinate of a computed representation, rather than a red, green, or blue measurement.
An image encoder is the function that computes that representation. For a learned encoder with parameters :
Here is the image and is its representation. Training changes so that the features support the training objective. A separate head can then turn those features into class scores, bounding boxes, or another task's outputs. The boundary between encoder and head is an architectural choice; calling a network an encoder does not specify how it was trained.
A feature vector is an ordered list of numbers. A spatial representation stores one such vector at each output location. Omitting the batch dimension, a CNN feature map might have shape : locations, each with feature coordinates. The output grid is often smaller than the input because of striding or downsampling.
The vector at a location can depend on a whole region of input pixels. Deeper layers can combine wider context, and transformer features can exchange information across image locations. A spatial feature's grid position therefore identifies an output location; it does not mean that the vector describes only one isolated pixel.
Inspect what a summary discards
Move the same pattern between the four regions below. The responding feature cell moves, but its average over all four cells stays unchanged. Then switch between vertical and horizontal splits: they have the same average brightness, yet different directional features.
Input: 4 × 4 × 1
0 is dark; 1 is bright. Each nonoverlapping 2 × 2 region produces one feature vector.
Spatial output: 2 × 2 × 3
Showing brightness. Each cell has three values, even though only one channel is colored.
- Moved region
- Top left
- Its feature vector
- [1, 0, 0]
- Mean-pooled vector
- [0.25, 0, 0]
Vector order: brightness, vertical contrast, horizontal contrast.
Moving this pattern changes which spatial cell responds. Its mean-pooled vector stays the same because averaging ignores the order of the four vectors.
Authored grayscale inputs and analytically calculated, fixed descriptors. No learned weights, inference, or measured model outputs. Contrasts are measured only inside each 2 × 2 region; boundaries between regions are ignored.
This example deliberately uses a grayscale image so every input value is visible. Its fixed encoder groups pixels into four nonoverlapping regions. For a region with top row and bottom row , its three features are:
These coordinates measure brightness, left-right contrast, and top-bottom contrast. “Vertical contrast” responds to a vertical boundary between dark and bright columns. The absolute values ignore which side is brighter. A solid bright region produces ; a vertical split produces . Region boundaries are excluded from these local calculations.
The resulting spatial tensor has shape . Flattening just its two spatial dimensions would give a sequence of four three-dimensional vectors. Keeping the row-major order allows each sequence entry to be mapped back to its grid location. Reshaping alone does not average or discard values.
The named channels make this arithmetic easy to inspect. They are authored descriptors, not a claim that a trained network develops three correspondingly named channels. Learned representations usually distribute useful information across many coordinates; a single coordinate need not mean “car,” “wheel,” or even “vertical edge.”
Spatial features and an image-level vector
Global average pooling averages each feature coordinate across all spatial locations. If the spatial vectors are , it produces one vector:
For the solid pattern, one region has and the other three have . Their mean is wherever the bright region sits. No downstream function receiving only that vector can distinguish these four placements, because its input is identical in every case.
That is an exact property of this example. It is not proof that arbitrary images shifted through a real encoder will produce identical summaries. Borders, sampling, positional information, and context can change the spatial vectors themselves. Even an averaged vector can contain layout information already encoded into its coordinates; averaging simply cannot recover the order of the vectors being averaged.
An image-level vector is useful when the next operation needs one descriptor per image, such as a classification head or image retrieval system. It is still a representation, not automatically a class prediction or a probability. Detection and segmentation need location-dependent evidence, so their heads commonly consume spatial features instead of relying only on an image summary.
Different encoders expose different outputs
CNNs commonly construct spatial feature maps with shared convolutional operations. A Vision Transformer instead begins by representing image patches as vectors and processes their sequence. In the original ViT, learned position embeddings are added to the patch embeddings, and the final state of an extra class token supplies the image-level representation. A class-token summary is learned through attention; it is not the arithmetic mean used above. Vision Transformer, Section 3.1
An encoder can expose both kinds of output. DINOv2's official forward_features method returns normalized class-token features and normalized patch-token features separately. The patch vectors retain their association with patch locations while carrying context acquired during encoding. Choosing the class-token output versus the patch-token output changes what the downstream system receives. DINOv2 official encoder implementation
For a detection example, original DETR takes a CNN backbone's spatial features into a transformer rather than first reducing the image to one classification vector. Its official backbone wrapper extracts the last residual stage, or intermediate stages when requested. DETR paper, official backbone implementation
Follow the representation into the next task
The next question is how a transformer gets those patch vectors in the first place. Image patches and patch embeddings follows an image into a sequence and works through the dimensions of its learned projection.
For detection, keep the spatial features and continue to Object queries in DETR: a set of decoder slots reads image evidence to make object predictions. For representation learning, Self-supervised learning and masked prediction asks how an encoder can learn useful features when the training targets come from the data itself.
To reuse an encoder for a new task, continue with transfer learning and fine-tuning. Then evaluating visual representations separates what frozen features support from what downstream adaptation can learn.
Related concepts
Evaluate frozen image features with nearest neighbors and linear probes, separate training from validation and test data, and understand what each protocol can establish.
Follow an image through convolution, ReLU, pooling and class scores. Inspect a transparent toy CNN and learn what classification can and cannot predict.
Move a pretrained encoder to a downstream task: trace frozen features, head training, partial and full fine-tuning, learning rates, and BatchNorm state.
Compare anchor-based vs anchor-free object detection: Faster R-CNN and RetinaNet anchors vs FCOS and CenterNet point-based methods.
Learn how DETR assigns unordered object targets to prediction slots, why greedy choices can fail, and how matching differs from training loss and evaluation.
Learn how normalized box coordinates, L1 loss, IoU, and Generalized IoU guide object localization, including disjoint boxes and containment limits.
