Patch content does not identify its location
An image patch carries information about its pixels. After a shared patch projection, its embedding still does not say which grid cell it came from. Moving the same patch from top-left to top-right leaves its content embedding unchanged.
Without position information, full self-attention is permutation equivariant: reorder the input rows and their output rows reorder with them. The output associated with a particular patch stays the same. If the [CLS] row remains fixed while patch rows are reordered, its resulting readout is invariant. These are different statements about the same computation.
Add a position row to each content row
Standard ViT prepends a learned [CLS] vector to the projected patches, then adds a learned absolute position table. For n patches of width D:
The semicolons stack rows into a sequence. The plus sign adds corresponding components; it does not concatenate position onto each feature vector. The first position row belongs to [CLS], which has no image-grid cell. Patch rows follow raster order. ViT, §3.1 defines this input.
An absolute row identifies a slot. When a different patch occupies that slot, it receives that slot’s position vector. The combination changes the queries, keys and values that subsequent attention layers compute.
Follow the same patch through the shuffle
The figure starts with Shuffle patches and patch A selected. Both lanes contain exactly the same patch order, C, A, D, B. The left lane uses content alone. The right adds the position assigned to each slot. Every output is computed from the displayed vectors.
Choose Original to see A in its original top-left slot, then return to Shuffle patches. A keeps content [1,0]. With positional addition, its input changes from [0.25,-0.75] to [1.75,-0.75]. In this authored example, its attention output changes by 1.018 in Euclidean distance; the [CLS] output changes by 0.203. Without positions, both changes are zero up to rounding.
Choose B, C or D under Inspect patch to follow that identity instead. The selected patch stays selected when the arrangement changes. Expand Inspect the selected patch’s attention weights to see how its normalized key weights produce the output. This is one unmasked attention head with identity Q/K/V projections, not a complete trained ViT, classification score or measured accuracy improvement.
Reordering rows is different from moving image content
Reorder pairs moves each content row together with its original position row. It changes the serialization, but preserves which content belongs to which spatial position. The right lane’s selected-patch output and fixed [CLS] readout agree with Original again.
Let P permute patch rows while keeping [CLS] fixed, and let A denote full self-attention with shared projections and no positional bias:
This follows because both axes of the score matrix reorder, row-wise softmax reorders with them, and the same permutation reorders the value rows. Moving already-positioned rows therefore gives:
Shuffling only content against fixed positions instead gives PX+E, which generally differs from P(X+E)=PX+PE. Its outputs can change. Position information breaks the symmetry of assigning the same set of patches to different locations; it does not make row serialization itself meaningful.
The equivariance statement assumes deterministic evaluation and no order-dependent mask or bias. Shared row-wise normalization, MLPs and residuals preserve it; a causal mask or position-dependent operation changes the assumptions. Distinct outputs in this fixture demonstrate the mechanism, not a guarantee that any trained model uses position effectively.
Changing the image resolution
A learned table has a specified number of rows. At fixed patch size, a larger image has more patches. The ViT paper uses 2D interpolation of the patch-position rows for higher-resolution fine-tuning. Keep the [CLS] row separate from that grid. Interpolation supplies values at a new grid size; successful transfer still depends on the model and training. ViT, §3.2.
For a 224×224 image with 16×16 patches, there are 196 patches and 197 rows including [CLS]. At width 768, that table has 151,296 parameters, about 0.18% of the paper’s rounded 86M-parameter ViT-Base. Its limitation is the fixed grid it was trained with, rather than a large share of the model’s parameters.
Other ways to supply position
The original text Transformer uses fixed sinusoidal rows, with the same frequency for each sine/cosine pair:
These require no learned position parameters and can be evaluated at new indices. That availability does not guarantee task quality at longer sequences or encode an image’s 2D grid by itself. Attention Is All You Need, §3.5.
An explicit 2D scheme can combine row and column encodings into one width-D position vector before addition. Relative-position schemes instead model relationships between positions. Rotary position embeddings (RoPE) rotate queries and keys; they are a different mechanism from adding an absolute table to the input.
Further reading
- An Image is Worth 16x16 Words (ViT): Dosovitskiy et al., 2020; input position table and resolution adjustment.
- Attention Is All You Need: Vaswani et al., 2017; attention equations and sinusoidal positional encoding.
Related concepts
How multi-head attention runs scaled dot-product attention in parallel across several representation subspaces to build context-aware token embeddings.
Follow one image patch through Q/K/V projection, scaled scores, row-wise softmax, value mixing, and the residual update inside a Vision Transformer.
Trace how a learned CLS row joins image patches, gathers evidence through self-attention, and becomes the image-level classification readout.
Trace how local windows, shifted cross-window exchange, and patch merging turn one high-resolution token grid into a multi-scale vision hierarchy.
Explore an illustrative region-tiling policy: resize tiles, calculate visual tokens, and distinguish analytical savings from measured model quality.
Interactive guide to convolution in CNNs: visualize sliding windows, kernels, stride, padding, and feature detection with step-by-step demos.
