TL;DR
- DINO learns visual representations without labels by training a student to match an EMA teacher across different image crops; the teacher is built online rather than pretrained separately.
- The teacher sees global crops while the student sees global and local crops. Centering and sharpening balance the teacher targets to avoid collapsed predictions.
- Trained DINO ViTs show semantic structure in their final-layer attention maps without segmentation supervision. The attention demo here is schematic, and the tiny CPU reference checks training mechanics rather than reproducing that learned behavior.
- The reported classification results and ablations show why crop strategy, teacher momentum and patch size matter when interpreting or reproducing the method.
Paper Overview
DINO — self-DIstillation with NO labels — demonstrates that self-supervised Vision Transformers learn features containing explicit information about the semantic layout of images. When you visualize the self-attention maps from the final layer's [CLS] token, the heads naturally segment objects without any pixel-level supervision, bounding boxes, or labels of any kind.
Published at ICCV 2021 by Mathilde Caron, Hugo Touvron, Ishan Misra, and colleagues at Meta AI and Inria, DINO combines knowledge distillation with self-supervised learning through a student-teacher framework where both networks share the same architecture. The teacher is not pretrained — it is built online as an exponential moving average of the student.
The results are striking: a ViT-S/16 trained with DINO achieves 77.0% top-1 accuracy on ImageNet under linear evaluation and 45.9 Jaccard index on PASCAL VOC object segmentation — nearly double the 27.3 achieved by a supervised ViT with the same architecture. These segmentation properties emerge without any segmentation training objective.
DINO Architecture
DINO is built on self-distillation: a student network learns by matching the output distribution of a teacher network, where the teacher is simply an exponential moving average (EMA) of the student's own weights. Both networks share the same architecture — there is no separate, pretrained teacher.
The framework has four key components that work together to produce high-quality representations:
-
Multi-crop augmentation: The teacher only sees large global crops while the student processes both global and smaller local crops. This asymmetry forces the student to learn local-to-global correspondences.
-
Shared architecture: Both student and teacher use the same backbone (ViT or ResNet). The teacher's weights are an EMA of the student, not independently trained.
-
Softmax with temperature: Both networks produce probability distributions via softmax, with the teacher using a lower temperature to produce sharper predictions.
-
Cross-entropy loss: The student learns by minimizing cross-entropy between its output distribution and the teacher's output distribution across different view pairs.
Multi-Crop Training
DINO's multi-crop strategy creates an asymmetry between what the teacher and student see. The teacher receives only two global crops (sampled from larger image areas), while the student processes all crops — both the global views and several smaller local crops (sampled from smaller image areas).
This asymmetry is the critical design choice. By requiring the student to match the teacher's global-view output while only seeing a small local patch, DINO forces the student to infer global semantic content from local visual information. A local crop of a dog's ear must produce a representation consistent with the teacher's representation of the entire dog.
The paper uses 2 global crops at resolution 224x224 and eight local crops in the official code’s default configuration at resolution 96x96. The teacher only processes the two global views, while the student processes all views. The official code samples global crop area scales from (0.4, 1.0) and local scales from (0.05, 0.4), so these are ranges, not fixed coverage guarantees. The loss has 18 valid teacher-to-student pairs with eight locals: each teacher global supervises nine student views, excluding its matching global identity.
Loss Function
DINO minimizes the cross-entropy between the teacher's output probability distribution and the student's output distribution, computed across all valid pairs of views. Crucially, a view is never compared with itself — the loss only considers pairs where the teacher and student process different crops.
Here x_1^g and x_2^g are the two global views processed by the teacher, V is the full set of views (global and local), and H(a, b) = -\sum_i a^{(i)} \log b^{(i)} is the standard cross-entropy. The teacher output appears as the target distribution and the student output as the predicted distribution.
Both networks produce K-dimensional probability distributions via softmax with temperature scaling. The teacher probability for dimension i is computed with centering to prevent collapse:
The teacher temperature \tau_t is set lower than the student temperature \tau_s (typically \tau_t = 0.04 vs \tau_s = 0.1), producing sharper probability distributions from the teacher. The centering vector c is subtracted before the softmax to prevent any single dimension from dominating.
Preventing Collapse: Centering and Sharpening
Self-distillation without labels faces a fundamental collapse risk: the teacher and student could converge to outputting the same uniform or constant distribution for all inputs. DINO prevents this through two complementary mechanisms that work in opposing directions.
Centering subtracts a running mean from the teacher's output before applying softmax. The center vector c is updated with exponential moving average over batch statistics:
where m is the momentum rate (typically 0.9). In this expression, B counts the raw teacher-logit rows being averaged; the code combines both global views, giving twice the image batch size on one worker. Centering prevents any single dimension from dominating the softmax output, which would cause one form of collapse where the teacher always assigns high probability to the same class regardless of input.
Sharpening uses a low teacher temperature \tau_t to produce peaked probability distributions. While centering pushes toward uniformity by preventing dominance of any single dimension, sharpening pushes away from uniformity by amplifying the largest logits. These two forces balance each other: centering alone would cause uniform collapse, and sharpening alone would cause dominance collapse.
The paper's ablation confirms that both are essential — removing either one causes training to collapse.
EMA Teacher Update
The teacher network in DINO is not trained with gradients. Instead, its parameters are updated as an exponential moving average (EMA) of the student parameters after each training step:
The momentum coefficient \lambda follows a cosine schedule from 0.996 to 1.0 over the course of training. Early in training, \lambda = 0.996 means the teacher incorporates more of the student's rapid updates, allowing it to keep pace with the student's fast initial learning. As training progresses, \lambda approaches 1.0, making the teacher increasingly stable — it changes very slowly, providing a consistent target for the student.
This schedule is motivated by the observation that early training requires faster teacher adaptation to track meaningful changes in the student, while late training benefits from a nearly frozen teacher that provides a stable, high-quality reference. The cosine schedule provides a smooth interpolation between these two regimes.
The EMA approach is what makes DINO a self-distillation method: the teacher is not an external model but a temporally smoothed version of the student itself. This means the student is effectively learning from its own past, ensembled over recent training history.
Implementation
The mechanics above can be checked without an ImageNet run. The following reference executes multi-crop pairing, normalized heads, centered teacher targets, student gradients, and the two separate EMA states on tiny synthetic CPU inputs. It follows the official loss and update conventions at a pinned revision; its reduced encoder and crops are explained alongside the code.
Executed, seeded offline CPU mechanics with tiny random-weight networks and synthetic images. This reference measures multi-crop pair selection, loss, gradients, center state, and teacher EMA. It uses no pretrained weights and produces no attention maps, segmentation results, ImageNet accuracy, or native distributed training performance.
1. Preserve view identity across two globals and L locals
Original paper
Only the two global views enter the teacher; every view enters the student.
Math and tensor shapes
The first two list entries retain global identities 0 and 1. Seeded offsets and flips crop each synthetic batch together. This reduction uses square source sides 10 and 8 for globals, and 4 for locals; it replaces the release’s random resized crops and photometric augmentations. Original output sizes are 224 and 96, with default area-scale ranges (0.4, 1.0) and (0.05, 0.4).
Smoke: images [3, 3, 12, 12]; globals [3, 3, 8, 8]; locals [3, 3, 4, 4]
Python implementation
def make_crops(images, local_count, generator):
"""Make two global crops followed by L local crops, preserving batch identity.
Args:
images: Synthetic images [B, 3, H, W] with H and W at least 12.
local_count: Nonnegative integer number of local views.
generator: Local CPU torch.Generator controlling offsets and flips.
Returns:
List with two [B, 3, 8, 8] globals and L [B, 3, 4, 4] locals.
"""
_tensor(images, "images", 4)
if images.shape[1] != 3 or min(images.shape[2:]) < 12:
raise ValueError(
"images require three channels and spatial dimensions >= 12"
)
if type(local_count) is not int or local_count < 0:
raise ValueError("local_count must be a nonnegative integer")
views = []
for side, output_size in [(10, 8), (8, 8)] + [(4, 4)] * local_count:
top = int(
torch.randint(images.shape[2] - side + 1, (), generator=generator)
)
left = int(
torch.randint(images.shape[3] - side + 1, (), generator=generator)
)
crop = images[:, :, top : top + side, left : left + side]
if bool(torch.randint(2, (), generator=generator)):
crop = crop.flip(-1)
views.append(
F.interpolate(
crop,
(output_size, output_size),
mode="bilinear",
align_corners=False,
)
)
return views2. Normalize the bottleneck and projection weights
Original paper
DINO uses a three-layer MLP, an L2-normalized bottleneck, and a weight-normalized output layer without a predictor.
Math and tensor shapes
A tiny convolution and spatial mean replace the ViT backbone. The BN-free MLP keeps GELU and normalized bottlenecks; normalizing each output-weight row implements weight normalization with gain fixed at 1. It uses random initialization and omits the release’s early epoch freezing of the whole last layer. These seven output dimensions are learned coordinates, not semantic class labels.
Conv features [B, 6] → MLP 6 → 12 → 12 → 4 → logits [B, 7]
Python implementation
class TinyDINO(nn.Module):
"""BN-free reduced encoder and three-layer DINO projection head."""
def __init__(self, output_dim=7):
super().__init__()
if type(output_dim) is not int or output_dim < 2:
raise ValueError("output_dim must be an integer >= 2")
self.encoder = nn.Conv2d(3, 6, kernel_size=3, padding=1)
self.mlp = nn.Sequential(
nn.Linear(6, 12),
nn.GELU(),
nn.Linear(12, 12),
nn.GELU(),
nn.Linear(12, 4),
)
self.direction = nn.Parameter(torch.randn(output_dim, 4))
def forward(self, view):
"""Map any positive crop resolution to logits [B, K]."""
_tensor(view, "view", 4)
if view.shape[1] != 3:
raise ValueError("view must have three channels")
features = F.gelu(self.encoder(view)).mean(dim=(-2, -1))
bottleneck = F.normalize(self.mlp(features), dim=-1)
# Weight normalization with fixed gain 1, like norm_last_layer=True.
weights = F.normalize(self.direction, dim=-1)
return F.linear(bottleneck, weights)3. Use the old center, sharpen, and stop gradients
Original paper
The teacher targets are formed before the center update and detached from the gradient path.
Math and tensor shapes
Centering subtracts a per-coordinate raw-logit running mean, counteracting one-dimension dominance. Lower temperature sharpens predictions, counteracting uniform outputs. Teacher parameters are frozen and its forward runs under no_grad; detaching targets also protects the standalone loss when passed gradient-enabled teacher logits. Neither operation proves that this tiny run learns useful features.
Teacher logits and detached targets [2, B, K]; old center [1, K]
Python implementation
def teacher_probabilities(logits, old_center, temperature):
"""Center raw teacher logits using the previous state, sharpen, then detach."""
_tensor(logits, "teacher logits", 3)
_tensor(old_center, "center", 2)
_positive(temperature, "teacher temperature")
if (
logits.shape[0] != 2
or logits.shape[2] < 2
or old_center.shape != (1, logits.shape[2])
or old_center.dtype != logits.dtype
):
raise ValueError(
"teacher requires [2, B, K] logits and matching [1, K] center"
)
return F.softmax((logits - old_center) / temperature, dim=-1).detach()4. Keep gradients through the student log probabilities
Original paper
The release evaluates log-softmax at the student temperature for each crop chunk.
Math and tensor shapes
Stable log-softmax avoids taking log of tiny softmax values. It preserves the student autograd graph, so matching teacher distributions updates the student encoder and head. Teacher temperature is scheduled separately from the fixed student temperature.
Student logits and log probabilities [2 + L, B, K]
Python implementation
def student_log_probabilities(logits, temperature=0.1):
"""Use stable log-softmax for all student crops [2 + L, B, K]."""
_tensor(logits, "student logits", 3)
_positive(temperature, "student temperature")
if logits.shape[0] < 2 or logits.shape[2] < 2:
raise ValueError(
"student requires at least two crops and two output dimensions"
)
return F.log_softmax(logits / temperature, dim=-1)5. Exclude only equal global identities
Original paper
Each of two teacher globals supervises all student views except its matching global crop. The code averages the paper’s sum over valid pairs.
Math and tensor shapes
List order encodes view identity. Student local views are always valid for either teacher global. The two equal-global pairs are excluded; there are no teacher local views. Even L = 0 gives two cross-global terms. The reference returns the pair list so this selection and reduction can be checked independently.
L = 8 → 18 pairs; average over B samples, then over pairs → scalar loss
Python implementation
def cross_view_loss(
student_logits,
teacher_logits,
old_center,
teacher_temperature,
student_temperature=0.1,
):
"""Average cross-entropy over batch and exactly 2*(L+1) valid view pairs.
The first two student crop identities match the teacher global identities.
This function is pure: the caller updates the center after this objective.
"""
targets = teacher_probabilities(
teacher_logits, old_center, teacher_temperature
)
predictions = student_log_probabilities(student_logits, student_temperature)
if (
predictions.shape[1:] != targets.shape[1:]
or predictions.dtype != targets.dtype
):
raise ValueError(
"teacher and student must share batch size, K and dtype"
)
terms, pairs = [], []
for teacher_view in range(2):
for student_view in range(predictions.shape[0]):
if student_view == teacher_view:
continue
terms.append(
-(targets[teacher_view] * predictions[student_view])
.sum(dim=-1)
.mean()
)
pairs.append((teacher_view, student_view))
return torch.stack(terms).mean(), pairs6. Advance the raw-logit center for the next objective
Original paper
The release updates the center at the end of its loss forward, after constructing the current objective.
Math and tensor shapes
Average raw teacher logits over both globals and all samples, not teacher probabilities or student outputs. This serial CPU mean replaces the distributed all-reduce. The smoke caller updates the center after forming the loss and before backward, matching the release order. The loss already contains targets from the previous center, so the mutation cannot change that objective.
Raw logits [2, B, K] → mean [1, K] → center [1, K]
Python implementation
@torch.no_grad()
def update_center(center, raw_teacher_logits, momentum=0.9):
"""EMA the raw-logit mean over both globals and B samples, after the loss."""
teacher_probabilities(
raw_teacher_logits, center, 1.0
) # Shape/finite checks.
_momentum(momentum)
batch_center = raw_teacher_logits.mean(dim=(0, 1)).unsqueeze(0)
center.mul_(momentum).add_(batch_center, alpha=1 - momentum)7. Move teacher weights toward the updated student
Original paper
Weight EMA follows the student optimizer step; it is distinct from the center’s logit EMA.
Math and tensor shapes
Deep-copy the student once, disable teacher gradients, then apply parameter EMA after SGD. The pinned cosine convention uses step/N, with base 0.996, so its final sampled step approaches 1 rather than reaching it. The smoke also exercises a deliberately short teacher-temperature warmup [0.04, 0.055, 0.07, 0.07]; the release defaults to 0.04 with zero warmup epochs. No DINOv2 patch loss, Sinkhorn step, or register tokens enter this DINO reference.
Matching named encoder/head parameters; center is separate state
Python implementation
@torch.no_grad()
def update_teacher(teacher, student, momentum):
"""Update matching teacher parameters after the student's optimizer step."""
_momentum(momentum)
teacher_parameters = dict(teacher.named_parameters())
student_parameters = dict(student.named_parameters())
if teacher_parameters.keys() != student_parameters.keys():
raise ValueError("teacher and student parameter names must match")
for name, parameter in teacher_parameters.items():
other = student_parameters[name]
if parameter.shape != other.shape or parameter.dtype != other.dtype:
raise ValueError(
"teacher and student parameter shapes/dtypes must match"
)
_tensor(parameter, name, parameter.ndim)
_tensor(other, name, other.ndim)
for name, parameter in teacher_parameters.items():
parameter.mul_(momentum).add_(
student_parameters[name], alpha=1 - momentum
)Run the same canonical source as the highlighted excerpts. Smoke JSON reports source SHA-256, seed, reductions, crop/logit shapes, actual loss, 18 pair terms, student gradients, zero teacher gradients, old and new centers, the effect of incorrectly using the new center, and measured teacher EMA error. Independent tests also compare the loss and student logit gradients with a scalar algebra oracle.
dino.pyrequirements-dino.txtPinned temperature schedulePinned cosine convention
python -m pip install -r requirements-dino.txt python dino.py --smoke --seed 7
These numerical checks establish what a training step computes. The next section concerns a separate phenomenon observed in trained DINO ViTs: semantic structure in their attention maps. The random-weight reference does not demonstrate that phenomenon, and the attention demo is a schematic rather than a pretrained-model export.
Emergent Attention Maps
The most striking finding of the DINO paper is that self-supervised ViTs learn attention maps that explicitly segment objects in images. When visualizing the self-attention of the [CLS] token in the last layer, different attention heads attend to different semantic regions of the image — one head might focus on the foreground object, another on the boundary, and another on the background.
This is DINO's signature discovery. These segmentation-quality attention maps emerge purely from the self-supervised training objective. The model was never shown any segmentation masks, bounding boxes, or pixel-level labels. Yet the attention heads learn to decompose images into semantically meaningful regions.
Quantitatively, DINO ViT-S/16 achieves 45.9 Jaccard index (IoU) on PASCAL VOC object segmentation using only the attention maps — nearly doubling the 27.3 achieved by a supervised ViT-S/16 with the same architecture. On DAVIS 2017 video object segmentation, DINO features enable competitive tracking performance without any video-specific training.
This property is specific to the combination of self-supervised training and the ViT architecture. Supervised ViTs do not produce such clean attention maps, and self-supervised CNNs lack the patch-level attention mechanism entirely.
How DINO Compares
SSL Method Comparison
How DINO compares to other self-supervised and supervised learning methods on ImageNet.
| Method | Collapse Prevention | k-NN Top-1 | Linear Top-1 | Seg. IoU | Key Mechanism |
|---|---|---|---|---|---|
| DINO | Centering + sharpening | 74.5 | 77.0 | 45.9 | Self-distillation with multi-crop |
| SimCLR | Large negative sets | — | 69.3 | — | NT-Xent contrastive loss |
| BYOL | Momentum + predictor | 64.8 | 74.4 | — | Asymmetric architecture with EMA |
| SwAV | Online clustering | 65.7 | 75.3 | — | Swapped online clustering assignments |
| MoCo v3 | Momentum contrastive | — | 72.5 | — | Momentum-updated queue encoder |
| Supervised | Labels prevent it | 79.8 | 79.8 | 27.3 | Cross-entropy on labeled data |
DINO
Self-distillation with multi-crop
SimCLR
NT-Xent contrastive loss
BYOL
Asymmetric architecture with EMA
SwAV
Swapped online clustering assignments
MoCo v3
Momentum-updated queue encoder
Supervised
Cross-entropy on labeled data
DINO's unique advantage
- Segmentation IoU of 45.9 vs 27.3 supervised — emergent object boundaries without any pixel labels
- Strong k-NN performance (74.5%) shows features are well-clustered without fine-tuning
- No negative pairs or contrastive loss required — self-distillation alone prevents collapse
Trade-offs
- Momentum encoder (EMA teacher) adds memory overhead and complexity
- Multi-crop augmentation strategy increases GPU cost during training
- Strongest results with Vision Transformers — gains are smaller with convolutional backbones
Key Results
ImageNet Classification
Under linear evaluation (frozen backbone, trained linear classifier), DINO achieves strong results across architectures and patch sizes:
| Model | k-NN Accuracy | Linear Eval |
|---|---|---|
| DINO ViT-S/16 | 74.5% | 77.0% |
| DINO ViT-B/16 | 76.1% | 78.2% |
| DINO ViT-S/8 | 78.3% | 79.7% |
| BYOL ResNet-50 | 66.6% | 71.4% |
| SwAV ResNet-50 | 66.3% | 73.5% |
The k-NN accuracy is particularly noteworthy. DINO's representations are so well-structured that a simple k-nearest-neighbor classifier (no training at all) reaches within 2-3% of the linear probe. This gap is much smaller than for other methods, indicating that DINO features form tighter, more linearly separable clusters in the embedding space.
Patch Size Matters More Than Model Size
One of DINO's counterintuitive findings is that reducing patch size (increasing the number of tokens) improves performance more than increasing model capacity. ViT-S/8 (small model, 8x8 patches) achieves 79.7% linear eval — outperforming ViT-B/16 (base model, 16x16 patches) at 78.2%. More spatial tokens provide finer-grained attention and better feature resolution, which matters more than adding parameters.
Critical Ablations
The paper provides thorough ablation studies that isolate the contribution of each component:
- No momentum teacher (copying student weights directly): fails to converge in the paper’s ablation, supporting the use of a temporally averaged teacher
- Multi-crop compute tradeoff: Table 8 reports 72.5% linear accuracy without local views versus 76.1% with ten local views after 300 epochs for ViT-S/16; this comparison is separate from the official code’s eight-local default
- Adding a predictor (like BYOL's MLP predictor): no improvement, suggesting that centering and sharpening already serve the asymmetry role
- Centering removed: sharpening favors collapse to a single dominant output dimension
- Sharpening removed (high teacher temperature): centering favors collapse to uniform outputs
Why DINO Matters
DINO's significance extends beyond its classification and segmentation numbers. The emergent attention properties reveal something fundamental about what self-supervised ViTs learn: without any explicit spatial supervision, the model discovers that attending to coherent object regions is the most efficient strategy for self-distillation. The attention maps are not a designed feature — they are a consequence of the training objective and architecture.
This finding has influenced the design of subsequent models. DINOv2 scaled the approach to larger datasets and model sizes, producing a foundation model with strong performance across dense prediction tasks. The idea that self-supervised ViTs implicitly learn spatial decomposition has informed work on open-vocabulary segmentation, where DINO features serve as a spatial backbone.
The connection between knowledge distillation and self-supervised learning is also conceptually important. DINO shows that distillation does not require a pretrained teacher — the teacher can emerge from the student's own training trajectory. This collapses the distinction between distillation and self-supervised learning, suggesting they are endpoints on a spectrum rather than separate paradigms.
Key Takeaways
-
Self-distillation works without labels — a student can learn strong representations by matching the output distribution of its own EMA teacher, without any pretrained model or labeled data.
-
Attention maps emerge as object segmenters — DINO ViTs learn to segment objects through self-attention alone, achieving 45.9 IoU on PASCAL VOC without any segmentation supervision.
-
Centering and sharpening prevent collapse — these two opposing forces balance each other: centering prevents single-dimension dominance while sharpening prevents uniform collapse.
-
Spatial resolution trumps model size — reducing patch size from 16 to 8 improves performance more than scaling from ViT-S to ViT-B, because finer-grained tokens enable richer attention patterns.
-
The EMA schedule matters — cosine annealing the momentum from 0.996 to 1.0 provides fast early adaptation and stable late-training targets, both essential for convergence.
Related Reading
- VICReg — Variance-invariance-covariance regularization for self-supervised learning without collapse
- V-JEPA — Joint-embedding predictive architecture for video representation learning
- Vision Transformer — The ViT architecture that DINO trains in a self-supervised manner
- CLIP — Contrastive vision-language pretraining using a different self-supervised paradigm
- Attention Is All You Need — The transformer architecture underlying Vision Transformers
