Skip to main content

BEiT: BERT Pre-Training of Image Transformers

BEiT masked image modeling explained through original paper sources, exact PyTorch code, frozen DALL-E targets, and real masked-token and classification outputs.

TL;DR

  • BEiT trains a Vision Transformer to predict frozen DALL-E dVAE categories at masked patch positions, rather than reconstructing RGB pixels.
  • Aligned views supply complete-image tokenizer targets and masked Transformer inputs; the encoder retains all patch positions, and only masked positions contribute to the loss.
  • Visual code IDs are learned categories, not object labels, and token prediction alone does not guarantee semantic understanding.
  • Original BEiT, later BEiT variants, and MAE report different training and evaluation recipes; their scores do not isolate the effect of target choice.

Paper Overview

BEiT, BERT Pre-Training of Image Transformers, adapts masked categorical prediction to vision. Published at ICLR 2022 by Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei, it uses a frozen discrete variational autoencoder (dVAE) to tokenize images into a vocabulary of 8192 categories. A Vision Transformer learns to predict the labels at masked patch positions from visible context. Earlier work, including iGPT and ViT's masked-patch experiment, already explored prediction-based visual pretraining; BEiT's contribution is its particular discrete-target framework and its strong downstream results.

The tokenizer separates the target representation from the Transformer input. It compresses the complete image into learned categorical labels while BEiT sees selected patch embeddings replaced by a mask vector. This changes the learning objective from RGB regression to classification over visual codes. It does not guarantee semantic understanding, and the code IDs are not object labels.

Original paper Table 1 reports 83.2% ImageNet-1K top-one for BEiT-B/16 at 224px after 800 epochs of pretraining and downstream fine-tuning, compared with 81.8% for supervised DeiT-B in that table. BEiT-L reaches 85.2% at 224px. Table 3 reports base-model ADE20K single-scale SETR-PUP results of 45.6 mIoU, or 47.7 with intermediate ImageNet-1K fine-tuning. Appendix D's 56.7% base linear probe uses average patch features from the best middle layer, not simply the final special token.

The Visual Tokenizer

The main experiments reuse OpenAI's publicly released pretrained DALL-E dVAE. Training a tokenizer on ImageNet-1K is an Appendix C comparison, not the default recipe. The convolutional encoder receives a complete 112px image crop and outputs a 14×14 grid with 8192 categorical logits at each position. Its most likely category supplies the target:

z_i = \arg\max_{v} q_{\phi}(v \mid x)_i

This is vocabulary-axis argmax, not a nearest-codebook-vector lookup. The 112px tokenizer view and 224px Transformer view share one crop and flip, so their grids align. The encoder downsamples by eight, while BEiT uses 16px patches. Its convolutions give each target spatial context; the tokenizer does not independently label a collection of isolated 16px crops.

Only the encoder supplies BEiT's labels. The decoder belongs to the dVAE training objective and is not used for the masked-token results here. An integer category has no guaranteed name such as “sky” or “fur,” and numerical distance between two IDs has no similarity meaning. Later BEiT-2 explores a different target-learning approach, but its downstream gains also involve training and model choices.

Blockwise Masking Strategy

BEiT selects contiguous rectangles on the 14×14 patch grid. Paper Algorithm 1 describes minimum block area 16 and approximately 40% masking; the released sampler defaults to area 4. It samples area uniformly and aspect ratio uniformly in log space between 0.3 and 1/0.3, counting only newly masked positions when blocks overlap.

The requested count is 75 out of 196 patches, or 38.3%. Bounded retries can leave fewer selected patches: the real examples below contain 75 and 74. The executable implementation preserves that behavior instead of filling the remainder with random individual patches.

Contiguous masking removes nearby visual context over larger regions. It changes the geometry of the prediction task relative to scattered masking, but this alone does not establish semantic reasoning or explain cross-method accuracy differences. BEiT keeps every patch position in the encoder; MAE instead encodes only its visible patches. The following 8×8 demo is a geometric illustration, not the actual 14×14 sampler or a measurement of representation quality.

The BEiT Pipeline

Two aligned views play different roles. The complete 112px view reaches the frozen tokenizer and produces 196 integer targets. The 224px view reaches the Transformer, where selected patch embeddings are replaced with a shared learned mask vector. The model retains all 196 patch positions and prepends one special token, making the full encoder sequence length 197.

The encoder's self-attention lets masked and visible positions exchange context. Its attention matrix covers 197×197 positions regardless of the patch mask. MAE's visible-only encoder has a smaller matrix, although the resulting whole-model speed difference also depends on its decoder and implementation.

After encoding, BEiT excludes the special token from its vocabulary head and predicts 8192-category logits at each patch position. Only selected positions contribute to cross entropy. The schematic below shows invented token outputs; actual learned predictions appear after the implementation.

Token Prediction vs Pixel Reconstruction

BEiT predicts learned visual-token categories with cross entropy, while MAE reconstructs per-patch normalized pixels with mean squared error. For selected patch positions \mathcal{M}, the executable mean loss is:

\mathcal{L}_{\text{BEiT}} = -\frac{1}{|\mathcal{M}|}\sum_{i \in \mathcal{M}} \log p(z_i \mid \tilde{x})

The original paper writes a summed log-likelihood objective. Averaging selected positions gives the release's training reduction. Cross entropy trains a distribution over categories and can represent uncertainty; it does not force a semantic commitment. Displaying the argmax category is a separate readout. MSE and categorical prediction make different target and distributional assumptions, rather than providing a general ranking of semantic quality.

Published base-model results are close after fine-tuning: BEiT 83.2% and MAE 83.6%. Their reported linear probes, 56.7% and 68.0%, use different protocols. In particular, BEiT reports the best middle-layer average patch representation. These numbers do not isolate the cause of a feature difference or establish that one objective requires nonlinear adaptation. The next section makes the target, mask and measured outputs concrete.

Implementation

The frozen DALL-E encoder supplies labels from the complete 112px crop. BEiT receives the aligned 224px crop with selected patch embeddings replaced by its mask vector. Follow those two roles through the exact executed Python below.

1. Make two views of one crop

Original paper

Paper excerpt · PDF p. 3Figure 1 · §2.1 and §2.5

One image supplies both patch embeddings and visual-token labels. The released transforms use different resolutions for these two branches.

Math and tensor shapes

N=(224/16)^2=(112/8)^2=196

Crop and flip once, then resize into aligned 14 × 14 grids. Normalize the bicubic 224px view with ImageNet mean and standard deviation. Map the Lanczos 112px view into [0.1, 0.9] for DALL-E. The tokenizer sees the complete crop, including regions that BEiT will not see. These photo examples use a deterministic center-square crop and no flip; they do not recreate the full training augmentation policy.

BEiT: [B, 3, 224, 224]; dVAE: [B, 3, 112, 112]

Pinned release: datasets.py

Python implementation

1. Make two views of one crop
python
def make_views(image, crop_box=None, flip=False):
    """Crop once, then make spatially aligned Transformer and tokenizer views."""
    crop = _crop(image, crop_box)
    if flip:
        crop = crop.transpose(Image.Transpose.FLIP_LEFT_RIGHT)
    beit_crop = crop.resize((224, 224), Image.Resampling.BICUBIC)
    dvae_crop = crop.resize((112, 112), Image.Resampling.LANCZOS)
    mean = torch.tensor(IMAGENET_MEAN).reshape(3, 1, 1)
    std = torch.tensor(IMAGENET_STD).reshape(3, 1, 1)
    return {
        "beit": (_pixels(beit_crop) - mean) / std,
        "dvae": 0.8 * _pixels(dvae_crop) + 0.1,
        "beit_crop": beit_crop,
        "dvae_crop": dvae_crop,
    }

2. Ask the frozen tokenizer for categorical targets

Original paper

§2.1.2 · §2.4 · Equation 3

The original experiments reuse the pretrained DALL-E discrete VAE. Its encoder is fixed while BEiT learns the masked-image prior.

Math and tensor shapes

z_i=\arg\max_v q_\phi(v\mid x)_i

Take argmax across the vocabulary axis, then flatten rows before columns. This is categorical prediction, not a nearest-vector lookup. The convolutional tokenizer has spatial context across the crop; each integer identifies an arbitrary learned category, not a named object. Evaluation mode, frozen parameters, and no_grad keep targets outside the student gradient graph.

[B, 8192, 14, 14] → [B, 196] integer labels

Python implementation

2. Ask the frozen tokenizer for categorical targets
python
def visual_targets(tokenizer, dvae_images):
    """Freeze the tokenizer and flatten vocabulary argmax in row-major order."""
    images = _image_batch(dvae_images)
    if images.shape[2] % 8 or images.shape[3] % 8:
        raise ValueError("DALL-E view dimensions must be divisible by eight")
    tokenizer.eval().requires_grad_(False)
    for parameter in tokenizer.parameters():
        parameter.grad = None
    with torch.no_grad():
        logits = tokenizer(images)
        _validate_tokenizer_logits(logits, images)
        return logits.argmax(dim=1).flatten(1)

3. Select rectangles and retain their actual count

Original paper

Paper excerpt · PDF p. 4

Algorithm 1 · released mask sampler

The paper describes minimum block area 16. The released default is 4, with at most 75 selected patch positions.

Math and tensor shapes

75/196\approx38.3\%
M=\sum_i m_i

Sample area uniformly and aspect ratio uniformly in log space between 0.3 and 1/0.3. Count only newly selected cells in overlapping rectangles. The helper bounds attempts and stops when a rectangle cannot be added; it never silently fills the remainder with random patches. Seeds 0 and 4 below produce 75 and 74 selections. Use the actual count as the loss and agreement denominator.

m: Boolean [B, 196]; special token excluded

Pinned release: masking_generator.py

Python implementation

3. Select rectangles and retain their actual count
python
def block_mask(grid_size=14, num_masked=75, seed=0, min_area=4):
    """Released rectangle algorithm; bounded retries can leave an undershoot."""
    height, width = _grid(grid_size)
    _integer(num_masked, "num_masked", minimum=0)
    _integer(min_area, "min_area")
    if num_masked > height * width:
        raise ValueError("num_masked cannot exceed the patch grid")
    rng = random.Random(seed)
    mask = torch.zeros(height, width, dtype=torch.bool)
    count = 0
    while count < num_masked:
        added = _place_rectangle(mask, num_masked - count, min_area, rng)
        if added == 0:
            break  # No random single-patch completion.
        count += added
    return mask.flatten()

4. Replace embeddings, then prepend the special token

Original paper

Paper excerpt · PDF p. 3§2.2–2.3 · retained masked positions

Selected patches carry a shared learned mask embedding. They stay in the encoder rather than being removed.

Math and tensor shapes

\tilde e_i=m_i e_{mask}+(1-m_i)e_i
s=[e_{cls};\tilde e_1;\ldots;\tilde e_N]

Project non-overlapping 16px patches, replace selected embeddings, and prepend one special token. Only the 196 patch positions have mask indices or visual-token targets. The original MIM checkpoint adds absolute position vectors to all 197 slots. The later classifier uses relative position bias instead.

[B, 196, 768] → [B, 197, 768]

Pinned release: modeling_pretrain.py

Python implementation

4. Replace embeddings, then prepend the special token
python
def embed(self, images, mask=None):
    """Replace selected patch embeddings, then add the special token/positions."""
    patches = self.patch_embed(images)
    batch, count, _ = patches.shape
    if mask is not None:
        if self.profile != "mim":
            raise ValueError(
                "classification uses every patch and accepts no mask"
            )
        mask = _patch_mask(mask, batch, count, patches.device)
        patches = torch.where(mask.unsqueeze(-1), self.mask_token, patches)
    sequence = torch.cat(
        (self.cls_token.expand(batch, -1, -1), patches), dim=1
    )
    return sequence if self.pos_embed is None else sequence + self.pos_embed

5. Let all 197 positions exchange context

Original paper

Paper excerpt · PDF p. 3

§2.2 · §2.5 · released encoder

The original base model has twelve pre-norm blocks, width 768 and twelve attention heads. Released classifier extensions are explicit in the code.

Math and tensor shapes

A=\operatorname{softmax}(QK^\top/\sqrt{d_h}+R)
h'=h+\operatorname{MSA}(\operatorname{LN}(h))
h_{next}=h'+\operatorname{MLP}(\operatorname{LN}(h'))

Attention mixes positions; the 768 → 3072 → 768 GELU MLP mixes channels at each position. For the original 1K MIM profile, R is absent and LayerScale is absent. The later 22K classifier adds relative bias R and learned branch scales. Dropout and stochastic depth are disabled at evaluation. These extensions belong to the release, not to the original paper equations.

Q, K, V: [B, 12, 197, 64]; attention: [B, 12, 197, 197]

Pinned release: modeling_finetune.py

Python implementation

5. Let all 197 positions exchange context
python
def scaled_attention(q, k, v, scale, relative_bias=None):
    """Each query attends to the entire retained sequence, including masked slots."""
    scores = (q * scale) @ k.transpose(-2, -1)
    if relative_bias is not None:
        scores = scores + relative_bias  # Released classification extension.
    return scores.softmax(dim=-1) @ v


def encode_block(sequence, block):
    """Pre-norm residuals; the classifier release adds learned LayerScale."""
    attention = block.attn(block.norm1(sequence))
    if block.gamma_1 is not None:
        attention = block.gamma_1 * attention
    sequence = sequence + block._drop(attention)
    mlp = block.mlp(block.norm2(sequence))
    if block.gamma_2 is not None:
        mlp = block.gamma_2 * mlp
    return sequence + block._drop(mlp)


def encode_sequence(sequence, blocks, final_norm):
    """All 196 patch slots plus the special token remain throughout the encoder."""
    for block in blocks:
        sequence = block(sequence)
    return final_norm(sequence)  # Identity for the mean-pooling classifier.

6. Score only selected patch predictions

Original paper

§2.3 · Equation 1

The paper maximizes a summed log likelihood over masked positions. The executable training loss averages their negative log likelihood.

Math and tensor shapes

\mathcal L=-\frac1M\sum_{i:m_i=1}\log p_\theta(z_i\mid\tilde x)

The vocabulary head follows the final encoder normalization and excludes the special token. Select logits and integer labels with the same Boolean mask. Cross entropy accepts raw logits and computes log-softmax internally. Visible predictions are unscored, and an empty mask is rejected. A distribution can express uncertainty across categories; argmax is only the displayed top-one choice.

[B, 196, 8192] → [M, 8192] → scalar loss

Pinned release: engine_for_pretraining.py

Python implementation

6. Score only selected patch predictions
python
def masked_loss(logits, targets, mask):
    """Average raw-logit cross entropy only over selected patch positions."""
    if logits.ndim != 3 or logits.shape[-1] < 1:
        raise ValueError("logits must have shape [B,N,V]")
    if targets.shape != logits.shape[:2] or targets.dtype != torch.long:
        raise ValueError("targets must be integer [B,N] labels")
    mask = _patch_mask(mask, *logits.shape[:2], logits.device)
    if not mask.any():
        raise ValueError("masked loss requires at least one selected patch")
    return F.cross_entropy(logits[mask], targets.to(logits.device)[mask])

7. Update the student while its targets stay fixed

Original paper

Paper excerpt · PDF p. 5

§2.4–2.5 · two-stage training

The tokenizer supplies fixed targets. The original BEiT pretraining uses unlabeled ImageNet-1K images for 800 epochs.

Math and tensor shapes

\theta\leftarrow\operatorname{AdamW}(\theta,\nabla_\theta\mathcal L)
\nabla_\phi\mathcal L=0

Clear accumulated gradients, run the masked student, backpropagate the finite objective, and update. The patch projection, mask vector, encoder, and vocabulary head learn; the tokenizer does not. The downloadable smoke command uses two synthetic images and a tiny synthetic target provider to verify these mechanics. It does not train the paper model or reproduce its accuracy, schedule, or data pipeline.

Loss: scalar; gradients: BEiT parameters only

Python implementation

7. Update the student while its targets stay fixed
python
def training_step(model, tokenizer, views, mask, optimizer):
    """Update BEiT only; the tokenizer sees the complete aligned crop."""
    _student_optimizer(model, optimizer)
    targets = visual_targets(tokenizer, views["dvae"])
    model.train()
    optimizer.zero_grad(set_to_none=True)
    logits = model(_image_batch(views["beit"]), mask)
    loss = masked_loss(logits, targets, mask)
    if not torch.isfinite(loss):
        raise ValueError(
            "masked loss must be finite before updating parameters"
        )
    loss.backward()
    optimizer.step()
    return loss.item()

8. Replace the objective with an image-class readout

Original paper

Paper excerpt · PDF p. 5

§2.6 · released mean-pooling classifier

Fine-tuning uses labeled images and a class head. The displayed classifier is a separate released 22K-to-1K checkpoint.

Math and tensor shapes

g=\operatorname{LN}(\frac1N\sum_{i=1}^N h_i)
\ell=Wg+b

Keep all image patches visible. Exclude the special token, average patch features, apply fc_norm, and use the 1,000-class head. This checkpoint expects a bicubic shorter-edge resize to 256, center crop to 224, and mean/std 0.5. Softmax is for displaying class probabilities; neither this readout nor its preprocessing is the MIM path.

[B, 197, 768] → [B, 768] → [B, 1000]

Pinned release: modeling_finetune.py

Python implementation

8. Replace the objective with an image-class readout
python
def classification_readout(encoded, pooling_norm, classifier):
    """Released readout: exclude special token, mean patches, normalize, classify."""
    patch_features = encoded[:, 1:]
    image_features = pooling_norm(patch_features.mean(dim=1))
    return classifier(image_features)

Keep these files together. CPU wheels are sufficient; real inference downloads and verifies the weights once, then caches them.

beit.pybeit_tokenizer.pybeit_export.pyrequirements-beit.txtbeit-NOTICE.txt

python -m pip install -r requirements-beit.txt
python beit.py --smoke
python beit.py --mim --image photo.png --seed 0 --output-dir mim
python beit.py --predict --image photo.png --output-dir classification
python beit_export.py --output-dir beit-demo --cache-dir ~/.cache/beit-example

Smoke checks mechanics with synthetic targets. The following photo outputs use learned checkpoints. The excerpts and syntax tokens are generated from the downloadable source.

Real masked-token predictions

The following CPU float32 outputs use beit_base_patch16_224_pt1k_800ep.pth: ImageNet-1K masked image modeling, 800 epochs, absolute positions, no relative bias or LayerScale. Frozen encoder.pkl supplies learned DALL-E labels. These two image/mask cases measure category agreement, not classification accuracy, segmentation, or image reconstruction.

Chelsea the cat

Original photograph · 451 × 300
Exact BEiT crop · 224 × 224
Complete tokenizer crop · 112 × 112
Selected patches, gray overlay · 224 × 224

Seed 0 · Requested 75 · Actual masked 75/196 · Matches 1/75 (1.33%) · Mean cross entropy 6.407495

Frozen DALL-E targets at all positions
BEiT predictions at masked positions

Teal outlines mark selected targets. Predicted cells: teal = same category, amber = different category, dash = visible and unscored. Colors indicate mask/equality only; numerical distance between IDs has no similarity meaning.

Exact token IDs and mask state: Chelsea the cat
All 196 positions in row-major order, one-based row/column. Visible predictions are null and unscored. IDs are arbitrary categories.
Row, colMaskTarget IDPredicted IDAgreement
1, 1visible3782null (unscored)unscored
1, 2visible8045null (unscored)unscored
1, 3visible2346null (unscored)unscored
1, 4visible7934null (unscored)unscored
1, 5visible4406null (unscored)unscored
1, 6visible2152null (unscored)unscored
1, 7visible2389null (unscored)unscored
1, 8visible7164null (unscored)unscored
1, 9visible1414null (unscored)unscored
1, 10visible500null (unscored)unscored
1, 11visible2132null (unscored)unscored
1, 12visible5888null (unscored)unscored
1, 13visible7274null (unscored)unscored
1, 14visible6299null (unscored)unscored
2, 1visible2649null (unscored)unscored
2, 2visible1182null (unscored)unscored
2, 3visible3159null (unscored)unscored
2, 4visible2677null (unscored)unscored
2, 5visible1881null (unscored)unscored
2, 6visible1030null (unscored)unscored
2, 7visible1115null (unscored)unscored
2, 8visible5667null (unscored)unscored
2, 9visible4836null (unscored)unscored
2, 10visible8138null (unscored)unscored
2, 11visible797null (unscored)unscored
2, 12visible2925null (unscored)unscored
2, 13visible7657null (unscored)unscored
2, 14visible2149null (unscored)unscored
3, 1visible2933null (unscored)unscored
3, 2visible5167null (unscored)unscored
3, 3visible3360null (unscored)unscored
3, 4visible3153null (unscored)unscored
3, 5visible1978null (unscored)unscored
3, 6visible6088null (unscored)unscored
3, 7visible7378null (unscored)unscored
3, 8visible256null (unscored)unscored
3, 9visible4229null (unscored)unscored
3, 10visible5146null (unscored)unscored
3, 11visible827null (unscored)unscored
3, 12visible1094null (unscored)unscored
3, 13visible4424null (unscored)unscored
3, 14visible3140null (unscored)unscored
4, 1masked19134990different
4, 2masked2156712different
4, 3masked66486920different
4, 4masked71741654different
4, 5masked59706064different
4, 6masked54906063different
4, 7visible3777null (unscored)unscored
4, 8visible888null (unscored)unscored
4, 9visible3021null (unscored)unscored
4, 10visible6499null (unscored)unscored
4, 11visible658null (unscored)unscored
4, 12visible902null (unscored)unscored
4, 13visible2726null (unscored)unscored
4, 14visible4079null (unscored)unscored
5, 1masked50242947different
5, 2masked46234716different
5, 3masked8474415different
5, 4masked66745193different
5, 5masked50773816different
5, 6masked38926532different
5, 7visible2927null (unscored)unscored
5, 8visible5515null (unscored)unscored
5, 9visible259null (unscored)unscored
5, 10visible4601null (unscored)unscored
5, 11visible8173null (unscored)unscored
5, 12visible7250null (unscored)unscored
5, 13visible5871null (unscored)unscored
5, 14visible7206null (unscored)unscored
6, 1masked13004707different
6, 2masked2530120different
6, 3masked54263231different
6, 4masked54348113different
6, 5masked35851281different
6, 6masked9787413different
6, 7visible1755null (unscored)unscored
6, 8visible3580null (unscored)unscored
6, 9visible3043null (unscored)unscored
6, 10masked55904228different
6, 11visible1104null (unscored)unscored
6, 12visible3747null (unscored)unscored
6, 13visible5172null (unscored)unscored
6, 14visible6408null (unscored)unscored
7, 1masked53485263different
7, 2masked72277281different
7, 3masked51742922different
7, 4masked65144518different
7, 5masked50225546different
7, 6masked49351957different
7, 7visible5988null (unscored)unscored
7, 8visible8119null (unscored)unscored
7, 9visible520null (unscored)unscored
7, 10masked57602235different
7, 11visible396null (unscored)unscored
7, 12visible4480null (unscored)unscored
7, 13masked39463946match
7, 14masked79355103different
8, 1masked3149810different
8, 2masked78957737different
8, 3masked67337664different
8, 4masked15547243different
8, 5masked21685624different
8, 6masked68257766different
8, 7visible2402null (unscored)unscored
8, 8visible7157null (unscored)unscored
8, 9visible6903null (unscored)unscored
8, 10masked63046146different
8, 11visible5145null (unscored)unscored
8, 12visible204null (unscored)unscored
8, 13masked25828084different
8, 14masked31444418different
9, 1masked25525364different
9, 2masked62958103different
9, 3masked39976864different
9, 4masked50991799different
9, 5masked1778142different
9, 6masked24601536different
9, 7visible4525null (unscored)unscored
9, 8visible373null (unscored)unscored
9, 9visible1882null (unscored)unscored
9, 10masked48972729different
9, 11visible6914null (unscored)unscored
9, 12visible3624null (unscored)unscored
9, 13visible4729null (unscored)unscored
9, 14visible7369null (unscored)unscored
10, 1masked3997369different
10, 2masked57653045different
10, 3masked33885515different
10, 4masked8010278different
10, 5masked59238067different
10, 6masked44245970different
10, 7visible7256null (unscored)unscored
10, 8visible2868null (unscored)unscored
10, 9visible477null (unscored)unscored
10, 10visible8145null (unscored)unscored
10, 11visible7389null (unscored)unscored
10, 12visible5625null (unscored)unscored
10, 13visible6624null (unscored)unscored
10, 14visible8142null (unscored)unscored
11, 1masked21245515different
11, 2masked77996299different
11, 3masked7557826different
11, 4masked44863384different
11, 5masked5973086different
11, 6masked58213809different
11, 7visible1667null (unscored)unscored
11, 8visible276null (unscored)unscored
11, 9visible6201null (unscored)unscored
11, 10visible2145null (unscored)unscored
11, 11visible1424null (unscored)unscored
11, 12visible5897null (unscored)unscored
11, 13visible4397null (unscored)unscored
11, 14visible2197null (unscored)unscored
12, 1masked67646299different
12, 2masked335449different
12, 3masked2232923different
12, 4masked31183857different
12, 5masked46397517different
12, 6masked59521130different
12, 7visible2053null (unscored)unscored
12, 8visible311null (unscored)unscored
12, 9visible979null (unscored)unscored
12, 10visible5833null (unscored)unscored
12, 11visible3669null (unscored)unscored
12, 12visible5159null (unscored)unscored
12, 13visible6922null (unscored)unscored
12, 14visible6005null (unscored)unscored
13, 1masked34886802different
13, 2masked43446296different
13, 3masked1707449different
13, 4masked5487449different
13, 5masked51491799different
13, 6masked25257585different
13, 7masked11832341different
13, 8visible5383null (unscored)unscored
13, 9visible1280null (unscored)unscored
13, 10visible3063null (unscored)unscored
13, 11visible7243null (unscored)unscored
13, 12visible8129null (unscored)unscored
13, 13visible4852null (unscored)unscored
13, 14visible3233null (unscored)unscored
14, 1masked66276802different
14, 2masked77756804different
14, 3masked25776804different
14, 4masked45985192different
14, 5masked17114589different
14, 6masked45895192different
14, 7visible4385null (unscored)unscored
14, 8visible3821null (unscored)unscored
14, 9visible5493null (unscored)unscored
14, 10visible3960null (unscored)unscored
14, 11visible5820null (unscored)unscored
14, 12visible7034null (unscored)unscored
14, 13visible5696null (unscored)unscored
14, 14visible2928null (unscored)unscored

Photo: Stefan van der Walt · Pinned source · CC0 · Original photograph

Cup of coffee

Original photograph · 600 × 400
Exact BEiT crop · 224 × 224
Complete tokenizer crop · 112 × 112
Selected patches, gray overlay · 224 × 224

Seed 4 · Requested 75 · Actual masked 74/196 · Matches 7/74 (9.46%) · Mean cross entropy 4.745119

Frozen DALL-E targets at all positions
BEiT predictions at masked positions

Teal outlines mark selected targets. Predicted cells: teal = same category, amber = different category, dash = visible and unscored. Colors indicate mask/equality only; numerical distance between IDs has no similarity meaning.

Exact token IDs and mask state: Cup of coffee
All 196 positions in row-major order, one-based row/column. Visible predictions are null and unscored. IDs are arbitrary categories.
Row, colMaskTarget IDPredicted IDAgreement
1, 1visible418null (unscored)unscored
1, 2visible272null (unscored)unscored
1, 3visible6907null (unscored)unscored
1, 4masked5957386different
1, 5masked59021356different
1, 6masked4998217different
1, 7masked70235881different
1, 8masked217217match
1, 9visible6667null (unscored)unscored
1, 10visible6636null (unscored)unscored
1, 11visible1228null (unscored)unscored
1, 12visible560null (unscored)unscored
1, 13visible6301null (unscored)unscored
1, 14visible6717null (unscored)unscored
2, 1visible2426null (unscored)unscored
2, 2visible760null (unscored)unscored
2, 3visible6580null (unscored)unscored
2, 4masked67243348different
2, 5masked287287match
2, 6masked10586762different
2, 7masked81256286different
2, 8masked70434423different
2, 9visible1405null (unscored)unscored
2, 10visible145null (unscored)unscored
2, 11visible3592null (unscored)unscored
2, 12visible6941null (unscored)unscored
2, 13visible3265null (unscored)unscored
2, 14visible6044null (unscored)unscored
3, 1visible2389null (unscored)unscored
3, 2visible6201null (unscored)unscored
3, 3visible4572null (unscored)unscored
3, 4masked65733097different
3, 5masked58393493different
3, 6masked21961339different
3, 7masked39035054different
3, 8masked80038003match
3, 9visible1334null (unscored)unscored
3, 10visible701null (unscored)unscored
3, 11visible5106null (unscored)unscored
3, 12visible3627null (unscored)unscored
3, 13visible4911null (unscored)unscored
3, 14visible5647null (unscored)unscored
4, 1visible5829null (unscored)unscored
4, 2visible1131null (unscored)unscored
4, 3visible6566null (unscored)unscored
4, 4masked4451223different
4, 5masked4692681different
4, 6masked77767031different
4, 7masked6802586different
4, 8masked33903390match
4, 9visible1871null (unscored)unscored
4, 10visible3802null (unscored)unscored
4, 11visible5841null (unscored)unscored
4, 12visible6614null (unscored)unscored
4, 13visible6614null (unscored)unscored
4, 14visible7585null (unscored)unscored
5, 1visible2395null (unscored)unscored
5, 2visible4672null (unscored)unscored
5, 3visible7005null (unscored)unscored
5, 4masked17275416different
5, 5masked72586262different
5, 6masked32653325different
5, 7masked76622457different
5, 8masked37291551different
5, 9visible5998null (unscored)unscored
5, 10visible6594null (unscored)unscored
5, 11visible2237null (unscored)unscored
5, 12visible111null (unscored)unscored
5, 13visible92null (unscored)unscored
5, 14visible793null (unscored)unscored
6, 1visible1662null (unscored)unscored
6, 2visible6667null (unscored)unscored
6, 3visible2507null (unscored)unscored
6, 4masked29645674different
6, 5masked74177363different
6, 6masked28231340different
6, 7masked13401340match
6, 8masked32864249different
6, 9visible4454null (unscored)unscored
6, 10visible5995null (unscored)unscored
6, 11visible4276null (unscored)unscored
6, 12visible5977null (unscored)unscored
6, 13visible5856null (unscored)unscored
6, 14visible5263null (unscored)unscored
7, 1visible4672null (unscored)unscored
7, 2visible5584null (unscored)unscored
7, 3visible92null (unscored)unscored
7, 4masked374686different
7, 5masked81834682different
7, 6masked1779873different
7, 7masked37897532different
7, 8masked73661990different
7, 9masked25282528match
7, 10masked1533780different
7, 11masked48882856different
7, 12masked54544866different
7, 13masked26303862different
7, 14masked75017232different
8, 1visible2630null (unscored)unscored
8, 2visible7521null (unscored)unscored
8, 3visible4035null (unscored)unscored
8, 4masked29601607different
8, 5masked24895892different
8, 6masked59775720different
8, 7masked81471305different
8, 8masked81663130different
8, 9masked3747649different
8, 10masked2600536different
8, 11masked3868093different
8, 12masked71922778different
8, 13masked69486948match
8, 14masked24601608different
9, 1visible6915null (unscored)unscored
9, 2visible1527null (unscored)unscored
9, 3visible6544null (unscored)unscored
9, 4masked42335129different
9, 5masked22775250different
9, 6masked59992630different
9, 7masked32915977different
9, 8masked23541740different
9, 9masked4621491different
9, 10masked32233947different
9, 11masked13568089different
9, 12masked62642778different
9, 13masked15271435different
9, 14masked7963277different
10, 1visible770null (unscored)unscored
10, 2visible3897null (unscored)unscored
10, 3visible1698null (unscored)unscored
10, 4masked61167405different
10, 5masked6875750different
10, 6masked23096954different
10, 7masked48786456different
10, 8masked2115746different
10, 9visible4327null (unscored)unscored
10, 10visible36null (unscored)unscored
10, 11visible6828null (unscored)unscored
10, 12visible6911null (unscored)unscored
10, 13visible2535null (unscored)unscored
10, 14visible5911null (unscored)unscored
11, 1visible7517null (unscored)unscored
11, 2visible8089null (unscored)unscored
11, 3visible6976null (unscored)unscored
11, 4visible6186null (unscored)unscored
11, 5visible8073null (unscored)unscored
11, 6visible6411null (unscored)unscored
11, 7visible4319null (unscored)unscored
11, 8visible4920null (unscored)unscored
11, 9visible7528null (unscored)unscored
11, 10visible5937null (unscored)unscored
11, 11visible2762null (unscored)unscored
11, 12visible7965null (unscored)unscored
11, 13visible6119null (unscored)unscored
11, 14visible2850null (unscored)unscored
12, 1visible960null (unscored)unscored
12, 2visible6959null (unscored)unscored
12, 3visible4035null (unscored)unscored
12, 4visible6950null (unscored)unscored
12, 5visible3741null (unscored)unscored
12, 6visible1672null (unscored)unscored
12, 7visible4878null (unscored)unscored
12, 8visible3542null (unscored)unscored
12, 9visible5540null (unscored)unscored
12, 10visible934null (unscored)unscored
12, 11visible477null (unscored)unscored
12, 12visible118null (unscored)unscored
12, 13visible6973null (unscored)unscored
12, 14visible2949null (unscored)unscored
13, 1visible4965null (unscored)unscored
13, 2visible7872null (unscored)unscored
13, 3masked5236662different
13, 4masked36691959different
13, 5masked60533326different
13, 6visible5751null (unscored)unscored
13, 7visible5701null (unscored)unscored
13, 8visible7851null (unscored)unscored
13, 9visible507null (unscored)unscored
13, 10visible5162null (unscored)unscored
13, 11visible6704null (unscored)unscored
13, 12visible3004null (unscored)unscored
13, 13visible2157null (unscored)unscored
13, 14visible2862null (unscored)unscored
14, 1visible6274null (unscored)unscored
14, 2visible5057null (unscored)unscored
14, 3masked4569418different
14, 4masked7644569different
14, 5masked67504010different
14, 6visible5231null (unscored)unscored
14, 7visible5499null (unscored)unscored
14, 8visible5231null (unscored)unscored
14, 9visible4265null (unscored)unscored
14, 10visible7872null (unscored)unscored
14, 11visible6677null (unscored)unscored
14, 12visible6415null (unscored)unscored
14, 13visible2299null (unscored)unscored
14, 14visible5811null (unscored)unscored

Photo: Rachel Michetti · Pinned source · CC0 · Original photograph

Classification with a separate checkpoint

beit_base_patch16_224_pt22k_ft1k.pth: Self-supervised ImageNet-22K pretraining then ImageNet-1K fine-tuning; relative position bias, LayerScale, patch mean then fc_norm. This later recipe is distinct from the original 1K MIM experiment. The exact classifier crops below use shorter-edge 256, center crop 224, and normalization with mean/std 0.5. Softmax scores describe the 1,000-class distribution and are not calibrated certainty.

Chelsea the cat

Exact classifier input · 224 × 224
  1. Egyptian cat (class 285)66.03%
  2. tiger cat (class 282)11.79%
  3. tabby (class 281)5.66%
  4. lynx (class 287)0.27%
  5. Angora (class 332)0.06%

Cup of coffee

Exact classifier input · 224 × 224
  1. espresso (class 967)91.15%
  2. cup (class 968)0.96%
  3. coffee mug (class 504)0.10%
  4. consomme (class 925)0.07%
  5. soup bowl (class 809)0.05%

Reproduce both fixed photos with the exporter command above. Every tokenizer, masked-model and classifier logit was compared with the pinned author implementations on identical inputs, with strict weight loading. The complete comparisons, tolerances, checkpoint hashes, licenses and preprocessing are recorded in measured results and provenance. Original equation crop provenance.

Pinned beit_modeling_finetune.pyPinned beit_modeling_pretrain.pyPinned dall_e_utils.pyPinned dall_e_encoder.py

Keeping the tokenizer fixed makes the target representation a separate design choice. That distinction leads to BEiT-2, which changes how those visual targets are learned.

From BEiT to BEiT-3

BEiT-2 changes how visual targets are learned. VQ-KD distills a teacher representation into a quantized vocabulary instead of using the original reconstruction-trained dVAE. The pinned official release reports 85.5% for Base and 87.3% for Large at 224px with ImageNet-1K pretraining. The base result is 2.3 points above original BEiT-B's 83.2%, but the recipes differ; this comparison does not attribute the entire gain to the tokenizer alone.

BEiT-3 extends masked data modeling to vision, language and paired data. Its Multiway Transformer shares attention while using modality-specific experts; Multiway/Magneto describes the architecture, not a tokenizer. Text uses SentencePiece, and image patches and text do not share a single integer vocabulary. The pinned official release reports ImageNet-1K top-one of 85.4% for Base and 87.6% for Large at 224px, with 305M parameters in the Large vision-only classifier. These are released classification protocols, distinct from other BEiT-3 retrieval-formulation results.

How BEiT Compares

Self-Supervised Method Comparison

Published ImageNet fine-tuning results across different recipes, not a controlled benchmark. BEiT and DINO use original BEiT Table 1; MAE uses its official release, SimMIM uses ViT-B 224px/800ep, and iBOT uses released blockmask weights.

BEiT

Approach: Token prediction
Masking: 40% blockwise
Target: Discrete tokens
ViT-B: 83.2%ViT-L: 85.2%

Frozen DALL-E categorical targets, original 800ep recipe

MAE

Approach: Pixel reconstruction
Masking: 75% random
Target: Raw pixels
ViT-B: 83.6%ViT-L: 85.9%

Visible-only encoder; speed depends on recipe

SimMIM

Approach: Pixel reconstruction
Masking: 60% random
Target: Raw pixels
ViT-B: 83.8%ViT-L: —

Simple design, works with Swin Transformer

DINO

Approach: Self-distillation
Masking: None (multi-crop)
Target: Teacher CLS token
ViT-B: 82.8%ViT-L: —

Emergent segmentation in attention maps

iBOT

Approach: Distillation + MIM
Masking: Patch masking
Target: Teacher tokens
ViT-B: 84.0%ViT-L: 84.8%

Combines DINO image-level + BEiT patch-level

Supervised DeiT-B

Approach: Cross-entropy
Masking: N/A
Target: Class labels
ViT-B: 81.8%ViT-L: —

BEiT Table 1 supervised comparator, 224px

BEiT / DINO / DeiT sourceMAE sourceSimMIM sourceiBOT source

BEiT's key insight

  • Learned categorical targets change the task from pixel regression
  • Frozen dVAE maps the complete crop to a categorical grid
  • Blockwise masking encourages reasoning about spatial context

Trade-offs

  • Requires an externally pretrained dVAE tokenizer
  • No MAE-style encoder speedup — all patches processed
  • Target choice matters alongside data and training recipe

Key Results

The following are published recipes, not a controlled head-to-head experiment. Resolution, training, representation selection and model generation differ.

ModelImageNet fine-tuning top-oneLinear probeProtocol
BEiT-B/1683.2%56.7%Original paper, 224px, 800ep; best middle-layer probe
BEiT-L/1685.2%—Original paper, 224px, 800ep
BEiT-2 Base85.5%—Official 1K-pretrained release, 224px
BEiT-3 Large87.6%—Official released classifier, 224px
MAE ViT-B83.6%68.0%MAE's published fine-tuning/probe recipes
Supervised DeiT-B81.8%—Original BEiT Table 1 comparator, 224px

Sources: BEiT Tables 1 and 9, BEiT-2 release, BEiT-3 release, MAE release.

Why BEiT Matters

BEiT helped establish masked image modeling as a practical way to pretrain Vision Transformers. Its distinctive choice was to obtain fixed visual categories from a separately pretrained tokenizer and learn to predict them from corrupted image input. Original Table 1's BEiT-B result exceeds its supervised DeiT-B comparator by 1.4 points under the reported recipes.

The framework separates three decisions: how targets are formed, which input positions are hidden, and how the trained encoder is adapted downstream. MAE and SimMIM explore pixel targets; later BEiT work changes visual targets and extends masked modeling across modalities. The implementation makes that separation visible: target IDs come from the complete tokenizer crop, student predictions come from a masked input, and classification requires its own fine-tuned head and preprocessing.

Key Takeaways

  1. Two views, one spatial crop. The complete 112px tokenizer view and the masked 224px Transformer view both yield a 14×14 grid.
  2. Targets are categorical and frozen. DALL-E vocabulary-axis argmax supplies integer labels; student gradients do not update the tokenizer.
  3. Count selected positions precisely. The release requests 75 patches, may undershoot, and retains 197 encoder positions including the special token.
  4. Score the masked positions only. Mean cross entropy and agreement use the actual selected count; visible positions are unscored.
  5. Separate recipes from claims. Original MIM, later 22K-to-1K classification, and cross-paper benchmarks use different models or protocols. Token prediction alone does not guarantee semantic understanding.
  • MAE — Masked autoencoders that reconstruct raw pixels instead of discrete tokens
  • DINO — Self-distillation with Vision Transformers via momentum teacher
  • SimCLR — Contrastive learning framework that established SSL baselines for vision
  • MoCo — Momentum contrast for building large, consistent negative dictionaries
  • I-JEPA — Joint-embedding predictive architecture that avoids pixel and token reconstruction
  • DINOv2 — Scaling self-supervised ViTs with curated data and distillation
  • BYOL — Bootstrap Your Own Latent — self-supervised learning without negative pairs
  • VICReg — Variance-invariance-covariance regularization for non-contrastive learning

If you found this paper review helpful, consider sharing it with others.

Mastodon