Skip to main content

Positional Embeddings in Vision Transformers

Move the same image patches with and without positional addition to see how learned absolute positions change a Vision Transformer’s attention inputs.

Patch content does not identify its location

An image patch carries information about its pixels. After a shared patch projection, its embedding still does not say which grid cell it came from. Moving the same patch from top-left to top-right leaves its content embedding unchanged.

Without position information, full self-attention is permutation equivariant: reorder the input rows and their output rows reorder with them. The output associated with a particular patch stays the same. If the [CLS] row remains fixed while patch rows are reordered, its resulting readout is invariant. These are different statements about the same computation.

Add a position row to each content row

Standard ViT prepends a learned [CLS] vector to the projected patches, then adds a learned absolute position table. For n patches of width D:

Z_0 = [x_{\mathrm{CLS}};x_1;\ldots;x_n] + E_{\mathrm{pos}},\qquad E_{\mathrm{pos}}\in\mathbb{R}^{(n+1)\times D}

The semicolons stack rows into a sequence. The plus sign adds corresponding components; it does not concatenate position onto each feature vector. The first position row belongs to [CLS], which has no image-grid cell. Patch rows follow raster order. ViT, §3.1 defines this input.

An absolute row identifies a slot. When a different patch occupies that slot, it receives that slot’s position vector. The combination changes the queries, keys and values that subsequent attention layers compute.

Follow the same patch through the shuffle

The figure starts with Shuffle patches and patch A selected. Both lanes contain exactly the same patch order, C, A, D, B. The left lane uses content alone. The right adds the position assigned to each slot. Every output is computed from the displayed vectors.

Choose Original to see A in its original top-left slot, then return to Shuffle patches. A keeps content [1,0]. With positional addition, its input changes from [0.25,-0.75] to [1.75,-0.75]. In this authored example, its attention output changes by 1.018 in Euclidean distance; the [CLS] output changes by 0.203. Without positions, both changes are zero up to rounding.

Choose B, C or D under Inspect patch to follow that identity instead. The selected patch stays selected when the arrangement changes. Expand Inspect the selected patch’s attention weights to see how its normalized key weights produce the output. This is one unmasked attention head with identity Q/K/V projections, not a complete trained ViT, classification score or measured accuracy improvement.

Reordering rows is different from moving image content

Reorder pairs moves each content row together with its original position row. It changes the serialization, but preserves which content belongs to which spatial position. The right lane’s selected-patch output and fixed [CLS] readout agree with Original again.

Let P permute patch rows while keeping [CLS] fixed, and let A denote full self-attention with shared projections and no positional bias:

A(PX)=P\,A(X)

This follows because both axes of the score matrix reorder, row-wise softmax reorders with them, and the same permutation reorders the value rows. Moving already-positioned rows therefore gives:

A\!\left(P(X+E)\right)=P\,A(X+E)

Shuffling only content against fixed positions instead gives PX+E, which generally differs from P(X+E)=PX+PE. Its outputs can change. Position information breaks the symmetry of assigning the same set of patches to different locations; it does not make row serialization itself meaningful.

The equivariance statement assumes deterministic evaluation and no order-dependent mask or bias. Shared row-wise normalization, MLPs and residuals preserve it; a causal mask or position-dependent operation changes the assumptions. Distinct outputs in this fixture demonstrate the mechanism, not a guarantee that any trained model uses position effectively.

Changing the image resolution

A learned table has a specified number of rows. At fixed patch size, a larger image has more patches. The ViT paper uses 2D interpolation of the patch-position rows for higher-resolution fine-tuning. Keep the [CLS] row separate from that grid. Interpolation supplies values at a new grid size; successful transfer still depends on the model and training. ViT, §3.2.

For a 224×224 image with 16×16 patches, there are 196 patches and 197 rows including [CLS]. At width 768, that table has 151,296 parameters, about 0.18% of the paper’s rounded 86M-parameter ViT-Base. Its limitation is the fixed grid it was trained with, rather than a large share of the model’s parameters.

Other ways to supply position

The original text Transformer uses fixed sinusoidal rows, with the same frequency for each sine/cosine pair:

PE_{p,2i}=\sin\!\left(\frac{p}{10000^{2i/D}}\right),\qquad PE_{p,2i+1}=\cos\!\left(\frac{p}{10000^{2i/D}}\right)

These require no learned position parameters and can be evaluated at new indices. That availability does not guarantee task quality at longer sequences or encode an image’s 2D grid by itself. Attention Is All You Need, §3.5.

An explicit 2D scheme can combine row and column encodings into one width-D position vector before addition. Relative-position schemes instead model relationships between positions. Rotary position embeddings (RoPE) rotate queries and keys; they are a different mechanism from adding an absolute table to the input.

Further reading

If you found this explanation helpful, consider sharing it with others.

Mastodon