Skip to main content

Image Patches and Patch Embeddings

Turn an image into a sequence: count patches, flatten pixels, apply a shared learned projection, and separate token count from embedding width.

Visual representations introduced the idea that an image encoder turns pixels into useful features. A convolutional encoder can retain a grid of features. A Vision Transformer instead processes a sequence of vectors. How does a two-dimensional image become that sequence?

The first step is to cut the image into patches and turn each patch into one vector. This page follows that conversion up to the input of the transformer. The conversion sets the size and meaning of the input; later attention layers learn relationships between its entries.

Count the patches before building the vectors

An image has height HH, width WW, and CC channels. Divide it into square, non-overlapping patches with side length PP. Assume both image dimensions are divisible by PP and there is no padding. The grid then has H/PH/P rows and W/PW/P columns, giving

N=HPWPN = \frac{H}{P}\frac{W}{P}

patches. Number them from left to right across the first row, then continue on the next row. That consistent ordering turns the grid into a sequence without discarding the pixels inside each patch.

For a 224×224224\times224 RGB image and 16×1616\times16 patches, the grid is 14×1414\times14 and the sequence contains 196 patch tokens. A patch is a fixed region, not an object proposal: a face can cross several patch boundaries, and one patch can contain both foreground and background. If a dimension is not divisible by the patch side, a real pipeline must choose a policy such as resizing, cropping, or padding. The formula above assumes that choice has already been made.

Flattening changes the arrangement

Each patch contains P2CP^2 C scalar values. Flattening places these values in a fixed order in one vector. For example, a 2×22\times2 grayscale patch with rows [1,2][1,2] and [3,4][3,4] becomes [1,2,3,4][1,2,3,4]. For RGB, each spatial position contributes three channel values, in a consistent channel order.

Flattening itself has no learned parameters and does not summarize the patch: it rearranges the same values. Its output is a sequence with shape N×(P2C)N\times(P^2 C). This raw width depends on the patch size and number of channels, so the next step maps it to the width the model expects.

A shared projection produces patch embeddings

Let xix_i be flattened patch ii, represented as a row vector. A learned matrix EE and optional bias bb produce its embedding:

zi=xiE+b,E∈R(P2C)×D,b∈RD.z_i = x_i E + b,\qquad E\in\mathbb{R}^{(P^2 C)\times D},\quad b\in\mathbb{R}^{D}.

The result has DD numbers. The same matrix and bias apply to every patch, so equal raw patches receive equal embeddings at this stage, wherever they occur in the image. Training adjusts these weights through the model's learning objective; there is no separate matrix for each image location. The ViT paper, Section 3.1, defines this shared trainable projection.

Change the patch side in the explorer to compare sequence lengths, then change the embedding width. Selecting different patches shows how a shared rule can produce different vectors from different input values.

From a patch to a token
Patch side
2

Authored grayscale pixels, rows: [0, 1, 2, 3]; [1, 2, 3, 0]; [2, 3, 0, 1]; [3, 0, 1, 2]. Select a patch in reading order.

Patch P1 flattens to [0, 1, 1, 2].

Each input value uses weight 1 for output 1, −1 for output 2, and 0.5 for output 3 when present. Bias is zero.

P1 → [4, -4]

Authored image and weights; outputs are exact arithmetic, not measured pretrained features. Every patch uses the same projection. No position vectors or attention are applied.

The explorer deliberately uses simple fixed weights so every output can be checked by hand. A trained projection learns many different weighted combinations of the pixels. The width DD is an architectural choice; it need not equal P2CP^2 C, and the conversion need not be a compression. Even equal input and output widths do not guarantee an invertible mapping.

In the official Google implementation, a convolution with kernel size PP and stride PP performs patch extraction and projection together, followed by a reshape into a sequence. Its filters are shared over positions, just as the matrix above is shared across patches. See the embedding layer in models_vit.py. This is an implementation of the patch projection, rather than a whole convolutional feature hierarchy.

Sequence length and vector width are different choices

The projected sequence has shape N×DN\times D: NN says how many locations enter the model, while DD says how many numbers describe each location. For a batch of BB images, the shape is B×N×DB\times N\times D.

Keeping a 224×224224\times224 input and changing the patch side from 16 to 32 reduces the grid from 14×1414\times14 to 7×77\times7, so NN falls from 196 to 49. The model can keep the same DD. Larger patches place more local pixels into each token and create fewer tokens; smaller patches retain a finer input grid. Neither setting alone establishes better accuracy.

With fixed PP, CC and DD, increasing image resolution increases the token count but does not change the projection's parameter count: P2CDP^2 C D weights, plus DD values if a bias is used. More tokens mean more downstream work, including more pairwise interactions in ordinary global self-attention. That arithmetic is not a measured latency prediction.

Patch embeddings are the beginning of the encoder

At this point, each embedding depends only on the pixels in its own patch. There has been no exchange of information between patches, no object recognition, and no image-level prediction. The initial vectors become contextual representations only as later encoder blocks process them together.

There is also a missing ingredient: where did each patch come from? A sequence is stored in an order, but a shared content projection does not itself encode that order. ViT adds position embeddings to identify sequence locations before its attention blocks. For classification, the original ViT also prepends a learned class token, so 196 patch tokens become 197 input tokens. That extra token is not another image patch. The paper's model overview and input equation show these as separate steps.

Continue with positional embeddings to see how location enters the vectors, then the ViT walkthrough to follow them through attention and an image-level readout. To follow the learning branch, self-supervised learning and masked prediction asks what a model can learn from these image regions when human labels are unavailable.

If you found this explanation helpful, consider sharing it with others.

Mastodon