Skip to main content

DINO: Emerging Properties in Self-Supervised Vision Transformers

How self-distillation with no labels produces Vision Transformer attention maps that automatically segment objects — without any pixel-level supervision.

TL;DR

  • DINO learns visual representations without labels by training a student to match an EMA teacher across different image crops; the teacher is built online rather than pretrained separately.
  • The teacher sees global crops while the student sees global and local crops. Centering and sharpening balance the teacher targets to avoid collapsed predictions.
  • Trained DINO ViTs show semantic structure in their final-layer attention maps without segmentation supervision. The attention demo here is schematic, and the tiny CPU reference checks training mechanics rather than reproducing that learned behavior.
  • The reported classification results and ablations show why crop strategy, teacher momentum and patch size matter when interpreting or reproducing the method.

Paper Overview

DINO — self-DIstillation with NO labels — demonstrates that self-supervised Vision Transformers learn features containing explicit information about the semantic layout of images. When you visualize the self-attention maps from the final layer's [CLS] token, the heads naturally segment objects without any pixel-level supervision, bounding boxes, or labels of any kind.

Published at ICCV 2021 by Mathilde Caron, Hugo Touvron, Ishan Misra, and colleagues at Meta AI and Inria, DINO combines knowledge distillation with self-supervised learning through a student-teacher framework where both networks share the same architecture. The teacher is not pretrained — it is built online as an exponential moving average of the student.

The results are striking: a ViT-S/16 trained with DINO achieves 77.0% top-1 accuracy on ImageNet under linear evaluation and 45.9 Jaccard index on PASCAL VOC object segmentation — nearly double the 27.3 achieved by a supervised ViT with the same architecture. These segmentation properties emerge without any segmentation training objective.

DINO Architecture

DINO is built on self-distillation: a student network learns by matching the output distribution of a teacher network, where the teacher is simply an exponential moving average (EMA) of the student's own weights. Both networks share the same architecture — there is no separate, pretrained teacher.

The framework has four key components that work together to produce high-quality representations:

  1. Multi-crop augmentation: The teacher only sees large global crops while the student processes both global and smaller local crops. This asymmetry forces the student to learn local-to-global correspondences.

  2. Shared architecture: Both student and teacher use the same backbone (ViT or ResNet). The teacher's weights are an EMA of the student, not independently trained.

  3. Softmax with temperature: Both networks produce probability distributions via softmax, with the teacher using a lower temperature to produce sharper predictions.

  4. Cross-entropy loss: The student learns by minimizing cross-entropy between its output distribution and the teacher's output distribution across different view pairs.

Multi-Crop Training

DINO's multi-crop strategy creates an asymmetry between what the teacher and student see. The teacher receives only two global crops (sampled from larger image areas), while the student processes all crops — both the global views and several smaller local crops (sampled from smaller image areas).

This asymmetry is the critical design choice. By requiring the student to match the teacher's global-view output while only seeing a small local patch, DINO forces the student to infer global semantic content from local visual information. A local crop of a dog's ear must produce a representation consistent with the teacher's representation of the entire dog.

The paper uses 2 global crops at resolution 224x224 and eight local crops in the official code’s default configuration at resolution 96x96. The teacher only processes the two global views, while the student processes all views. The official code samples global crop area scales from (0.4, 1.0) and local scales from (0.05, 0.4), so these are ranges, not fixed coverage guarantees. The loss has 18 valid teacher-to-student pairs with eight locals: each teacher global supervises nine student views, excluding its matching global identity.

Loss Function

DINO minimizes the cross-entropy between the teacher's output probability distribution and the student's output distribution, computed across all valid pairs of views. Crucially, a view is never compared with itself — the loss only considers pairs where the teacher and student process different crops.

h(x,x')=H(P_t(x),P_s(x'))
\mathcal{L} = \sum_{x \in \{x_1^g, x_2^g\}} \sum_{\substack{x' \in V \\ x' \neq x}} h(x,x')

Here x_1^g and x_2^g are the two global views processed by the teacher, V is the full set of views (global and local), and H(a, b) = -\sum_i a^{(i)} \log b^{(i)} is the standard cross-entropy. The teacher output appears as the target distribution and the student output as the predicted distribution.

Both networks produce K-dimensional probability distributions via softmax with temperature scaling. The teacher probability for dimension i is computed with centering to prevent collapse:

a^{(i)}=(g_{\theta_t}(x)^{(i)}-c^{(i)})/\tau_t
P_t(x)^{(i)}=\frac{\exp(a^{(i)})}{\sum_{k=1}^{K}\exp(a^{(k)})}

The teacher temperature \tau_t is set lower than the student temperature \tau_s (typically \tau_t = 0.04 vs \tau_s = 0.1), producing sharper probability distributions from the teacher. The centering vector c is subtracted before the softmax to prevent any single dimension from dominating.

Preventing Collapse: Centering and Sharpening

Self-distillation without labels faces a fundamental collapse risk: the teacher and student could converge to outputting the same uniform or constant distribution for all inputs. DINO prevents this through two complementary mechanisms that work in opposing directions.

Centering subtracts a running mean from the teacher's output before applying softmax. The center vector c is updated with exponential moving average over batch statistics:

c \leftarrow mc + (1 - m)\frac{1}{B}\sum_{i=1}^{B} g_{\theta_t}(x_i)

where m is the momentum rate (typically 0.9). In this expression, B counts the raw teacher-logit rows being averaged; the code combines both global views, giving twice the image batch size on one worker. Centering prevents any single dimension from dominating the softmax output, which would cause one form of collapse where the teacher always assigns high probability to the same class regardless of input.

Sharpening uses a low teacher temperature \tau_t to produce peaked probability distributions. While centering pushes toward uniformity by preventing dominance of any single dimension, sharpening pushes away from uniformity by amplifying the largest logits. These two forces balance each other: centering alone would cause uniform collapse, and sharpening alone would cause dominance collapse.

The paper's ablation confirms that both are essential — removing either one causes training to collapse.

EMA Teacher Update

The teacher network in DINO is not trained with gradients. Instead, its parameters are updated as an exponential moving average (EMA) of the student parameters after each training step:

\theta_t \leftarrow \lambda \theta_t + (1 - \lambda)\theta_s

The momentum coefficient \lambda follows a cosine schedule from 0.996 to 1.0 over the course of training. Early in training, \lambda = 0.996 means the teacher incorporates more of the student's rapid updates, allowing it to keep pace with the student's fast initial learning. As training progresses, \lambda approaches 1.0, making the teacher increasingly stable — it changes very slowly, providing a consistent target for the student.

This schedule is motivated by the observation that early training requires faster teacher adaptation to track meaningful changes in the student, while late training benefits from a nearly frozen teacher that provides a stable, high-quality reference. The cosine schedule provides a smooth interpolation between these two regimes.

The EMA approach is what makes DINO a self-distillation method: the teacher is not an external model but a temporally smoothed version of the student itself. This means the student is effectively learning from its own past, ensembled over recent training history.

Implementation

The mechanics above can be checked without an ImageNet run. The following reference executes multi-crop pairing, normalized heads, centered teacher targets, student gradients, and the two separate EMA states on tiny synthetic CPU inputs. It follows the official loss and update conventions at a pinned revision; its reduced encoder and crops are explained alongside the code.

Executed, seeded offline CPU mechanics with tiny random-weight networks and synthetic images. This reference measures multi-crop pair selection, loss, gradients, center state, and teacher EMA. It uses no pretrained weights and produces no attention maps, segmentation results, ImageNet accuracy, or native distributed training performance.

1. Preserve view identity across two globals and L locals

Original paper

Paper excerpt · PDF p. 3§3.1 · multi-crop construction and Eq. (3)

Only the two global views enter the teacher; every view enters the student.

Math and tensor shapes

V=\{x_1^g,x_2^g,x_1^l,\ldots,x_L^l\}
|V|=2+L

The first two list entries retain global identities 0 and 1. Seeded offsets and flips crop each synthetic batch together. This reduction uses square source sides 10 and 8 for globals, and 4 for locals; it replaces the release’s random resized crops and photometric augmentations. Original output sizes are 224 and 96, with default area-scale ranges (0.4, 1.0) and (0.05, 0.4).

Smoke: images [3, 3, 12, 12]; globals [3, 3, 8, 8]; locals [3, 3, 4, 4]

Python implementation

1. Preserve view identity across two globals and L locals
python
def make_crops(images, local_count, generator):
    """Make two global crops followed by L local crops, preserving batch identity.

    Args:
        images: Synthetic images [B, 3, H, W] with H and W at least 12.
        local_count: Nonnegative integer number of local views.
        generator: Local CPU torch.Generator controlling offsets and flips.

    Returns:
        List with two [B, 3, 8, 8] globals and L [B, 3, 4, 4] locals.
    """
    _tensor(images, "images", 4)
    if images.shape[1] != 3 or min(images.shape[2:]) < 12:
        raise ValueError(
            "images require three channels and spatial dimensions >= 12"
        )
    if type(local_count) is not int or local_count < 0:
        raise ValueError("local_count must be a nonnegative integer")
    views = []
    for side, output_size in [(10, 8), (8, 8)] + [(4, 4)] * local_count:
        top = int(
            torch.randint(images.shape[2] - side + 1, (), generator=generator)
        )
        left = int(
            torch.randint(images.shape[3] - side + 1, (), generator=generator)
        )
        crop = images[:, :, top : top + side, left : left + side]
        if bool(torch.randint(2, (), generator=generator)):
            crop = crop.flip(-1)
        views.append(
            F.interpolate(
                crop,
                (output_size, output_size),
                mode="bilinear",
                align_corners=False,
            )
        )
    return views

2. Normalize the bottleneck and projection weights

Original paper

Paper excerpt · PDF p. 4§3.1 · architecture; pinned DINOHead, lines 257–291

DINO uses a three-layer MLP, an L2-normalized bottleneck, and a weight-normalized output layer without a predictor.

Math and tensor shapes

b=\operatorname{MLP}(f(x))
\hat b=b/\lVert b\rVert_2
g(x)=\hat W\hat b,\quad g(x)\in\mathbb R^K

A tiny convolution and spatial mean replace the ViT backbone. The BN-free MLP keeps GELU and normalized bottlenecks; normalizing each output-weight row implements weight normalization with gain fixed at 1. It uses random initialization and omits the release’s early epoch freezing of the whole last layer. These seven output dimensions are learned coordinates, not semantic class labels.

Conv features [B, 6] → MLP 6 → 12 → 12 → 4 → logits [B, 7]

Python implementation

2. Normalize the bottleneck and projection weights
python
class TinyDINO(nn.Module):
    """BN-free reduced encoder and three-layer DINO projection head."""

    def __init__(self, output_dim=7):
        super().__init__()
        if type(output_dim) is not int or output_dim < 2:
            raise ValueError("output_dim must be an integer >= 2")
        self.encoder = nn.Conv2d(3, 6, kernel_size=3, padding=1)
        self.mlp = nn.Sequential(
            nn.Linear(6, 12),
            nn.GELU(),
            nn.Linear(12, 12),
            nn.GELU(),
            nn.Linear(12, 4),
        )
        self.direction = nn.Parameter(torch.randn(output_dim, 4))

    def forward(self, view):
        """Map any positive crop resolution to logits [B, K]."""
        _tensor(view, "view", 4)
        if view.shape[1] != 3:
            raise ValueError("view must have three channels")
        features = F.gelu(self.encoder(view)).mean(dim=(-2, -1))
        bottleneck = F.normalize(self.mlp(features), dim=-1)
        # Weight normalization with fixed gain 1, like norm_last_layer=True.
        weights = F.normalize(self.direction, dim=-1)
        return F.linear(bottleneck, weights)

3. Use the old center, sharpen, and stop gradients

Original paper

Paper excerpt · PDF p. 3Algorithm 1; pinned DINOLoss.forward, lines 387–390

The teacher targets are formed before the center update and detached from the gradient path.

Math and tensor shapes

a=(t-c_{old})/\tau_t
q=\operatorname{sg}(\operatorname{softmax}(a))

Centering subtracts a per-coordinate raw-logit running mean, counteracting one-dimension dominance. Lower temperature sharpens predictions, counteracting uniform outputs. Teacher parameters are frozen and its forward runs under no_grad; detaching targets also protects the standalone loss when passed gradient-enabled teacher logits. Neither operation proves that this tiny run learns useful features.

Teacher logits and detached targets [2, B, K]; old center [1, K]

Python implementation

3. Use the old center, sharpen, and stop gradients
python
def teacher_probabilities(logits, old_center, temperature):
    """Center raw teacher logits using the previous state, sharpen, then detach."""
    _tensor(logits, "teacher logits", 3)
    _tensor(old_center, "center", 2)
    _positive(temperature, "teacher temperature")
    if (
        logits.shape[0] != 2
        or logits.shape[2] < 2
        or old_center.shape != (1, logits.shape[2])
        or old_center.dtype != logits.dtype
    ):
        raise ValueError(
            "teacher requires [2, B, K] logits and matching [1, K] center"
        )
    return F.softmax((logits - old_center) / temperature, dim=-1).detach()

4. Keep gradients through the student log probabilities

Original paper

Paper excerpt · PDF p. 3Eq. (1); pinned student scaling and log-softmax, lines 384–399

The release evaluates log-softmax at the student temperature for each crop chunk.

Math and tensor shapes

\log p=\operatorname{logsoftmax}(s/\tau_s)
\tau_s=0.1

Stable log-softmax avoids taking log of tiny softmax values. It preserves the student autograd graph, so matching teacher distributions updates the student encoder and head. Teacher temperature is scheduled separately from the fixed student temperature.

Student logits and log probabilities [2 + L, B, K]

Python implementation

4. Keep gradients through the student log probabilities
python
def student_log_probabilities(logits, temperature=0.1):
    """Use stable log-softmax for all student crops [2 + L, B, K]."""
    _tensor(logits, "student logits", 3)
    _positive(temperature, "student temperature")
    if logits.shape[0] < 2 or logits.shape[2] < 2:
        raise ValueError(
            "student requires at least two crops and two output dimensions"
        )
    return F.log_softmax(logits / temperature, dim=-1)

5. Exclude only equal global identities

Original paper

Paper excerpt · PDF p. 3Eq. (3); pinned pair loop and reduction, lines 392–404

Each of two teacher globals supervises all student views except its matching global crop. The code averages the paper’s sum over valid pairs.

Math and tensor shapes

H(q,p)=-\sum_{k=1}^{K}q_k\log p_k
N_{pairs}=2(L+1)
h_{i,v}=\overline{H(q_i,p_v)}
\mathcal L=\frac{1}{N_{pairs}}\sum_{i=0}^{1}\sum_{v\ne i}h_{i,v}

List order encodes view identity. Student local views are always valid for either teacher global. The two equal-global pairs are excluded; there are no teacher local views. Even L = 0 gives two cross-global terms. The reference returns the pair list so this selection and reduction can be checked independently.

L = 8 → 18 pairs; average over B samples, then over pairs → scalar loss

Python implementation

5. Exclude only equal global identities
python
def cross_view_loss(
    student_logits,
    teacher_logits,
    old_center,
    teacher_temperature,
    student_temperature=0.1,
):
    """Average cross-entropy over batch and exactly 2*(L+1) valid view pairs.

    The first two student crop identities match the teacher global identities.
    This function is pure: the caller updates the center after this objective.
    """
    targets = teacher_probabilities(
        teacher_logits, old_center, teacher_temperature
    )
    predictions = student_log_probabilities(student_logits, student_temperature)
    if (
        predictions.shape[1:] != targets.shape[1:]
        or predictions.dtype != targets.dtype
    ):
        raise ValueError(
            "teacher and student must share batch size, K and dtype"
        )
    terms, pairs = [], []
    for teacher_view in range(2):
        for student_view in range(predictions.shape[0]):
            if student_view == teacher_view:
                continue
            terms.append(
                -(targets[teacher_view] * predictions[student_view])
                .sum(dim=-1)
                .mean()
            )
            pairs.append((teacher_view, student_view))
    return torch.stack(terms).mean(), pairs

6. Advance the raw-logit center for the next objective

Original paper

Paper excerpt · PDF p. 4Eq. (4); pinned center update, lines 403–416

The release updates the center at the end of its loss forward, after constructing the current objective.

Math and tensor shapes

\bar t=\frac{1}{2B}\sum_{i=0}^{1}\sum_{b=1}^{B}t_{i,b}
c_{new}=m_c c_{old}+(1-m_c)\bar t
m_c=0.9

Average raw teacher logits over both globals and all samples, not teacher probabilities or student outputs. This serial CPU mean replaces the distributed all-reduce. The smoke caller updates the center after forming the loss and before backward, matching the release order. The loss already contains targets from the previous center, so the mutation cannot change that objective.

Raw logits [2, B, K] → mean [1, K] → center [1, K]

Python implementation

6. Advance the raw-logit center for the next objective
python
@torch.no_grad()
def update_center(center, raw_teacher_logits, momentum=0.9):
    """EMA the raw-logit mean over both globals and B samples, after the loss."""
    teacher_probabilities(
        raw_teacher_logits, center, 1.0
    )  # Shape/finite checks.
    _momentum(momentum)
    batch_center = raw_teacher_logits.mean(dim=(0, 1)).unsqueeze(0)
    center.mul_(momentum).add_(batch_center, alpha=1 - momentum)

7. Move teacher weights toward the updated student

Original paper

Paper excerpt · PDF p. 4§3.1 · teacher network; pinned optimizer and EMA, lines 326–350

Weight EMA follows the student optimizer step; it is distinct from the center’s logit EMA.

Math and tensor shapes

\Delta=\theta_s^{new}-\theta_t^{old}
\theta_t^{new}=\theta_t^{old}+(1-\lambda)\Delta

Deep-copy the student once, disable teacher gradients, then apply parameter EMA after SGD. The pinned cosine convention uses step/N, with base 0.996, so its final sampled step approaches 1 rather than reaching it. The smoke also exercises a deliberately short teacher-temperature warmup [0.04, 0.055, 0.07, 0.07]; the release defaults to 0.04 with zero warmup epochs. No DINOv2 patch loss, Sinkhorn step, or register tokens enter this DINO reference.

Matching named encoder/head parameters; center is separate state

Python implementation

7. Move teacher weights toward the updated student
python
@torch.no_grad()
def update_teacher(teacher, student, momentum):
    """Update matching teacher parameters after the student's optimizer step."""
    _momentum(momentum)
    teacher_parameters = dict(teacher.named_parameters())
    student_parameters = dict(student.named_parameters())
    if teacher_parameters.keys() != student_parameters.keys():
        raise ValueError("teacher and student parameter names must match")
    for name, parameter in teacher_parameters.items():
        other = student_parameters[name]
        if parameter.shape != other.shape or parameter.dtype != other.dtype:
            raise ValueError(
                "teacher and student parameter shapes/dtypes must match"
            )
        _tensor(parameter, name, parameter.ndim)
        _tensor(other, name, other.ndim)
    for name, parameter in teacher_parameters.items():
        parameter.mul_(momentum).add_(
            student_parameters[name], alpha=1 - momentum
        )

Run the same canonical source as the highlighted excerpts. Smoke JSON reports source SHA-256, seed, reductions, crop/logit shapes, actual loss, 18 pair terms, student gradients, zero teacher gradients, old and new centers, the effect of incorrectly using the new center, and measured teacher EMA error. Independent tests also compare the loss and student logit gradients with a scalar algebra oracle.

dino.pyrequirements-dino.txtPinned temperature schedulePinned cosine convention

python -m pip install -r requirements-dino.txt
python dino.py --smoke --seed 7

These numerical checks establish what a training step computes. The next section concerns a separate phenomenon observed in trained DINO ViTs: semantic structure in their attention maps. The random-weight reference does not demonstrate that phenomenon, and the attention demo is a schematic rather than a pretrained-model export.

Emergent Attention Maps

The most striking finding of the DINO paper is that self-supervised ViTs learn attention maps that explicitly segment objects in images. When visualizing the self-attention of the [CLS] token in the last layer, different attention heads attend to different semantic regions of the image — one head might focus on the foreground object, another on the boundary, and another on the background.

This is DINO's signature discovery. These segmentation-quality attention maps emerge purely from the self-supervised training objective. The model was never shown any segmentation masks, bounding boxes, or pixel-level labels. Yet the attention heads learn to decompose images into semantically meaningful regions.

Quantitatively, DINO ViT-S/16 achieves 45.9 Jaccard index (IoU) on PASCAL VOC object segmentation using only the attention maps — nearly doubling the 27.3 achieved by a supervised ViT-S/16 with the same architecture. On DAVIS 2017 video object segmentation, DINO features enable competitive tracking performance without any video-specific training.

This property is specific to the combination of self-supervised training and the ViT architecture. Supervised ViTs do not produce such clean attention maps, and self-supervised CNNs lack the patch-level attention mechanism entirely.

How DINO Compares

SSL Method Comparison

How DINO compares to other self-supervised and supervised learning methods on ImageNet.

DINO

Collapse:
Centering + sharpening
k-NN Top-1:74.5
Linear Top-1:77.0
Seg. IoU:45.9

Self-distillation with multi-crop

SimCLR

Collapse:
Large negative sets
Linear Top-1:69.3

NT-Xent contrastive loss

BYOL

Collapse:
Momentum + predictor
k-NN Top-1:64.8
Linear Top-1:74.4

Asymmetric architecture with EMA

SwAV

Collapse:
Online clustering
k-NN Top-1:65.7
Linear Top-1:75.3

Swapped online clustering assignments

MoCo v3

Collapse:
Momentum contrastive
Linear Top-1:72.5

Momentum-updated queue encoder

Supervised

Collapse:
Labels prevent it
k-NN Top-1:79.8
Linear Top-1:79.8
Seg. IoU:27.3

Cross-entropy on labeled data

DINO's unique advantage

  • Segmentation IoU of 45.9 vs 27.3 supervised — emergent object boundaries without any pixel labels
  • Strong k-NN performance (74.5%) shows features are well-clustered without fine-tuning
  • No negative pairs or contrastive loss required — self-distillation alone prevents collapse

Trade-offs

  • Momentum encoder (EMA teacher) adds memory overhead and complexity
  • Multi-crop augmentation strategy increases GPU cost during training
  • Strongest results with Vision Transformers — gains are smaller with convolutional backbones

Key Results

ImageNet Classification

Under linear evaluation (frozen backbone, trained linear classifier), DINO achieves strong results across architectures and patch sizes:

Modelk-NN AccuracyLinear Eval
DINO ViT-S/1674.5%77.0%
DINO ViT-B/1676.1%78.2%
DINO ViT-S/878.3%79.7%
BYOL ResNet-5066.6%71.4%
SwAV ResNet-5066.3%73.5%

The k-NN accuracy is particularly noteworthy. DINO's representations are so well-structured that a simple k-nearest-neighbor classifier (no training at all) reaches within 2-3% of the linear probe. This gap is much smaller than for other methods, indicating that DINO features form tighter, more linearly separable clusters in the embedding space.

Patch Size Matters More Than Model Size

One of DINO's counterintuitive findings is that reducing patch size (increasing the number of tokens) improves performance more than increasing model capacity. ViT-S/8 (small model, 8x8 patches) achieves 79.7% linear eval — outperforming ViT-B/16 (base model, 16x16 patches) at 78.2%. More spatial tokens provide finer-grained attention and better feature resolution, which matters more than adding parameters.

Critical Ablations

The paper provides thorough ablation studies that isolate the contribution of each component:

  • No momentum teacher (copying student weights directly): fails to converge in the paper’s ablation, supporting the use of a temporally averaged teacher
  • Multi-crop compute tradeoff: Table 8 reports 72.5% linear accuracy without local views versus 76.1% with ten local views after 300 epochs for ViT-S/16; this comparison is separate from the official code’s eight-local default
  • Adding a predictor (like BYOL's MLP predictor): no improvement, suggesting that centering and sharpening already serve the asymmetry role
  • Centering removed: sharpening favors collapse to a single dominant output dimension
  • Sharpening removed (high teacher temperature): centering favors collapse to uniform outputs

Why DINO Matters

DINO's significance extends beyond its classification and segmentation numbers. The emergent attention properties reveal something fundamental about what self-supervised ViTs learn: without any explicit spatial supervision, the model discovers that attending to coherent object regions is the most efficient strategy for self-distillation. The attention maps are not a designed feature — they are a consequence of the training objective and architecture.

This finding has influenced the design of subsequent models. DINOv2 scaled the approach to larger datasets and model sizes, producing a foundation model with strong performance across dense prediction tasks. The idea that self-supervised ViTs implicitly learn spatial decomposition has informed work on open-vocabulary segmentation, where DINO features serve as a spatial backbone.

The connection between knowledge distillation and self-supervised learning is also conceptually important. DINO shows that distillation does not require a pretrained teacher — the teacher can emerge from the student's own training trajectory. This collapses the distinction between distillation and self-supervised learning, suggesting they are endpoints on a spectrum rather than separate paradigms.

Key Takeaways

  1. Self-distillation works without labels — a student can learn strong representations by matching the output distribution of its own EMA teacher, without any pretrained model or labeled data.

  2. Attention maps emerge as object segmenters — DINO ViTs learn to segment objects through self-attention alone, achieving 45.9 IoU on PASCAL VOC without any segmentation supervision.

  3. Centering and sharpening prevent collapse — these two opposing forces balance each other: centering prevents single-dimension dominance while sharpening prevents uniform collapse.

  4. Spatial resolution trumps model size — reducing patch size from 16 to 8 improves performance more than scaling from ViT-S to ViT-B, because finer-grained tokens enable richer attention patterns.

  5. The EMA schedule matters — cosine annealing the momentum from 0.996 to 1.0 provides fast early adaptation and stable late-training targets, both essential for convergence.

  • VICReg — Variance-invariance-covariance regularization for self-supervised learning without collapse
  • V-JEPA — Joint-embedding predictive architecture for video representation learning
  • Vision Transformer — The ViT architecture that DINO trains in a self-supervised manner
  • CLIP — Contrastive vision-language pretraining using a different self-supervised paradigm
  • Attention Is All You Need — The transformer architecture underlying Vision Transformers

If you found this paper review helpful, consider sharing it with others.

Mastodon