BYOL: Bootstrap Your Own Latent

Paper Overview

BYOL — Bootstrap Your Own Latent — demonstrates that self-supervised visual representation learning does not require negative pairs. Prior contrastive methods like SimCLR and MoCo relied on pushing apart representations of different images (negatives) to prevent the network from collapsing to a trivial constant output. BYOL discards this mechanism entirely and still learns representations that surpass its contrastive predecessors.

Published at NeurIPS 2020 by Jean-Bastien Grill, Florian Strub, and colleagues at DeepMind, BYOL achieves 74.3% top-1 accuracy on ImageNet with a ResNet-50 (1000 epochs) — surpassing SimCLR's 69.3% by a significant margin. With a wider and deeper backbone (ResNet-200, 2x width), BYOL reaches 79.6%, approaching supervised baselines.

The central question BYOL raises is deceptively simple: if you only train on positive pairs (two augmented views of the same image), what prevents the network from outputting the same constant vector for every input? The answer lies in architectural asymmetry — a predictor MLP that exists only in the online branch, combined with an exponential moving average (EMA) target network that receives no gradients. The paper reports non-collapsed representations empirically and offers a hypothesis for these dynamics in §3.2; it does not prove that collapsed solutions are impossible.

BYOL Architecture

BYOL uses a two-network design with deliberate asymmetry between the networks. The online network consists of three components: an encoder (e.g., ResNet-50), a projector MLP, and a predictor MLP. The target network mirrors the first two components — encoder and projector — but critically lacks the predictor.

This asymmetry is the core mechanism. The predictor MLP exists only in the online branch, meaning the online network must learn an additional mapping on top of its projection. The target network, lacking a predictor, produces a simpler output that serves as the regression target.

The target network receives no gradient updates. Instead, its parameters are updated as an exponential moving average of the online network's parameters after each training step. This means the target evolves slowly and smoothly, providing a stable reference that the online network learns to predict.

Both networks process different augmented views of the same image. The online network produces a prediction from one view, and the target network produces a projection from the other view. The loss minimizes the distance between these two outputs, then the views are swapped and the loss is symmetrized.

Why No Negative Pairs?

Prior self-supervised methods relied on contrastive losses that serve two purposes simultaneously: pulling together representations of augmented views of the same image (positive pairs), and pushing apart representations of different images (negative pairs). Without negatives, there is nothing to prevent the network from mapping every input to the same point — a degenerate solution that trivially minimizes any positive-pair-only loss.

SimCLR requires large batch sizes (4096+) specifically to provide enough negative pairs per step. MoCo maintains a momentum-updated queue of negatives. SwAV uses cluster assignments as implicit negatives. The entire field assumed that some form of negative signal was necessary.

BYOL removes negatives entirely. The loss function operates exclusively on positive pairs — two views of the same image. No other images participate in the loss computation for a given pair.

A shared constant nonzero output can achieve zero normalized regression loss, even with a predictor and EMA. The question is therefore about the trajectory of training, not whether the objective excludes constants. Section 3.2 treats the explanation as a hypothesis under an idealized optimal predictor.

The Collapse Prevention Mechanism

The evidence comes from controlled ablations. In the paper's Table 5(b), removing the predictor yields 0.2% top-1 accuracy, while the predictor-only configuration without a momentum target yields 0.3%, at 300 epochs. These results establish the importance of both components in that training setup, rather than a theorem for every architecture or optimizer.

The following animation is a schematic of those outcomes. Its cluster motion is prescribed, not an executed training run. The implementation below will expose the actual gradient and state updates separately.

The Predictor's Role

The predictor MLP sits atop the online network's projector and learns to map the online projection to the target projection. Formally, it learns to approximate the conditional expectation: given the online network's representation of one view, predict what the target network would produce for the other view.

For a fixed, informative target, this motivates an online representation that helps predict the other view. The conditional-expectation argument assumes an optimal predictor in an idealized unnormalized, unsymmetrized setting; it is an intuition for BYOL's dynamics, not a claim that a trained MLP always reaches that optimum or that a constant target is excluded.

The BYOL loss is the mean squared error between the L2-normalized prediction and the L2-normalized target projection:

where \bar{q}_\theta(z_\theta) and \bar{z}'_\xi denote L2-normalized vectors. The loss reduces to a cosine similarity term, and is symmetrized by computing it for both orderings of the two views and summing the results.

EMA Target Network

The target network's parameters \xi are updated after each training step as an exponential moving average of the online network's parameters \theta:

The momentum coefficient \tau follows a cosine schedule from a base value of \tau_\text{base} = 0.996 to 1.0 over the course of training. Early in training, \tau = 0.996 means the target incorporates 0.4% of the online network's weights at each step — enough to track the online network's rapid initial learning. As training progresses, \tau approaches 1.0, and the target becomes increasingly frozen, providing a stable high-quality reference.

The EMA provides a delayed reference rather than a second gradient descent update. This temporal asymmetry is part of the authors' explanation for the observed training behavior; it does not eliminate constant-output equilibria. Freezing target weights also does not freeze every state variable: the official target forward runs in training mode and returns updated normalization state. The subsequent EMA changes parameters, while retaining that separately computed target state.

Implementation

The full step is now explicit: compute both view orderings with a stopped target graph, optimize the online encoder/projector/predictor, then interpolate the target weights from the updated online branch. This reduced executable reference measures those operations and their normalization-state boundary before we examine the paper's batch-size evidence.

def make_views(images, seed):
    """Make reproducible flip/noise views without changing global RNG state."""
    if (not isinstance(images, torch.Tensor) or images.ndim != 4
            or images.shape[0] < 2 or min(images.shape[1:]) < 1
            or not images.is_floating_point() or images.device.type != "cpu"
            or not torch.isfinite(images).all()):
        raise ValueError("images must be finite floating CPU [B,C,H,W], B >= 2")
    generator = torch.Generator().manual_seed(seed)
    noise1 = torch.randn(images.shape, generator=generator, dtype=images.dtype)
    noise2 = torch.randn(images.shape, generator=generator, dtype=images.dtype)
    return images + 0.05 * noise1, images.flip(-1) + 0.05 * noise2

class TinyBranch(nn.Module):
    """Reduced encoder f and BN projector g with output [B, D]."""

    def __init__(self, input_dim=48, hidden_dim=12, projection_dim=6):
        super().__init__()
        if any(not isinstance(n, int) or n < 1
               for n in (input_dim, hidden_dim, projection_dim)):
            raise ValueError("branch dimensions must be positive integers")
        self.encoder = nn.Sequential(
            nn.Flatten(), nn.Linear(input_dim, hidden_dim), nn.ReLU())
        self.projector = nn.Sequential(
            nn.Linear(hidden_dim, hidden_dim), nn.BatchNorm1d(hidden_dim),
            nn.ReLU(), nn.Linear(hidden_dim, projection_dim))

    def forward(self, images):
        """Project image features; training BN needs at least two samples."""
        return self.projector(self.encoder(images))


class TinyOnline(nn.Module):
    """Online f, g and predictor q; only f and g have target counterparts."""

    def __init__(self, input_dim=48, hidden_dim=12, projection_dim=6):
        super().__init__()
        self.branch = TinyBranch(input_dim, hidden_dim, projection_dim)
        self.predictor = nn.Sequential(
            nn.Linear(projection_dim, hidden_dim), nn.BatchNorm1d(hidden_dim),
            nn.ReLU(), nn.Linear(hidden_dim, projection_dim))

    def forward(self, images):
        """Return predictions [B, D] and projections [B, D]."""
        projection = self.branch(images)
        return self.predictor(projection), projection

def make_target(online):
    """Copy f/g weights and buffers independently, then freeze parameters."""
    target = copy.deepcopy(online.branch)
    target.requires_grad_(False)
    target.train()  # Freeze weights, not running normalization state.
    return target


def target_forward(target, view1, view2):
    """Stop target graphs while updating BN state on each training view."""
    with torch.no_grad():
        z1 = target(view1)
        z2 = target(view2)
    return z1, z2

def normalized_regression(prediction, target):
    """Return per-example 2 - 2 cosine, with no cross-example comparisons."""
    validate_matrix(prediction, "prediction")
    validate_matrix(target, "target")
    if prediction.shape != target.shape or prediction.dtype != target.dtype:
        raise ValueError("prediction and target shapes and dtypes must match")
    # Reject undefined directions rather than silently normalizing zero vectors.
    for value in (prediction, target):
        norms = torch.linalg.vector_norm(value, dim=-1)
        if not torch.isfinite(norms).all() or (norms <= 1e-6).any():
            raise ValueError("regression requires finite norms greater than 1e-6")
    p = F.normalize(prediction, dim=-1)
    z = F.normalize(target.detach(), dim=-1)
    return 2 - 2 * (p * z).sum(dim=-1)


def byol_loss(p1, p2, z1, z2):
    """Sum both view orderings before the batch mean, as paper section 3.1."""
    forward = normalized_regression(p1, z2)
    reverse = normalized_regression(p2, z1)
    if forward.shape != reverse.shape:
        raise ValueError("both view orderings must have the same batch size")
    return (forward + reverse).mean()

def online_update(online, target, optimizer, view1, view2):
    """Backpropagate the symmetric loss and update only online parameters."""
    online_ids = {id(p) for p in online.parameters()}
    optimized_ids = {id(p) for group in optimizer.param_groups
                     for p in group["params"]}
    if optimized_ids != online_ids:
        raise ValueError("optimizer must contain exactly the online parameters")
    expected = online.branch.encoder[1].in_features
    for view in (view1, view2):
        if (view.ndim != 4 or view.shape[0] < 2
                or math.prod(view.shape[1:]) != expected):
            raise ValueError("views must be matching [B,C,H,W] with B >= 2")
        validate_matrix(view.flatten(1), "view")
    if view1.shape != view2.shape or view1.dtype != view2.dtype:
        raise ValueError("views must have matching shapes and dtypes")
    if view1.dtype != online.branch.encoder[1].weight.dtype:
        raise ValueError("view dtype must match online weights")
    if (not online.training or not target.training
            or any(p.requires_grad for p in target.parameters())):
        raise ValueError("train both branches with frozen target parameters")
    optimizer.zero_grad(set_to_none=True)
    p1, projection1 = online(view1)
    p2, _ = online(view2)
    z1, z2 = target_forward(target, view1, view2)
    loss = byol_loss(p1, p2, z1, z2)
    loss.backward()
    optimizer.step()  # Target EMA is a separate, subsequent operation.
    return loss.detach(), projection1.detach(), z1, z2

def cosine_tau(step, total_steps, base_tau=0.996):
    """Increase tau from base_tau to one, including both endpoints."""
    if (not isinstance(step, int) or not isinstance(total_steps, int)
            or total_steps < 1 or not 0 <= step <= total_steps
            or not math.isfinite(base_tau) or not 0 <= base_tau <= 1):
        raise ValueError("invalid EMA schedule arguments")
    return 1 - (1 - base_tau) * (1 + math.cos(math.pi * step / total_steps)) / 2


def update_target(target, online_branch, tau):
    """Apply post-optimizer EMA to parameters, leaving BN buffers alone."""
    if not math.isfinite(tau) or not 0 <= tau <= 1:
        raise ValueError("tau must be finite and in [0, 1]")
    target_params = dict(target.named_parameters())
    online_params = dict(online_branch.named_parameters())
    if target_params.keys() != online_params.keys() or any(
            target_params[k].shape != online_params[k].shape
            for k in target_params):
        raise ValueError("target and online branch parameter structures differ")
    with torch.no_grad():
        for key, target_param in target_params.items():
            target_param.mul_(tau).add_(online_params[key], alpha=1 - tau)
    # BN running mean/variance already changed in target_forward, not via EMA.

python -m pip install -r requirements-byol.txt
python byol.py --smoke --seed 7

Batch Size Robustness

Contrastive methods like SimCLR depend on having many negative pairs per training step to provide a strong signal for pushing apart representations of different images. This creates a direct dependency on batch size: more images per batch means more negatives, which means better contrastive learning. SimCLR's performance degrades sharply below batch sizes of 4096.

BYOL's regression loss contains no cross-example negative comparisons. Its networks still couple samples through training-mode batch normalization, so absence of negatives does not imply independence from batch size.

The paper's Appendix F.2, Table 16 compares BYOL with its SimCLR reproduction at 300 epochs. BYOL is fairly stable from batch size 256 to 4096: at 256 it achieves 71.8% versus SimCLR's 64.3%; at 4096, 72.5% versus 67.9%. At 64, both drop substantially, to 59.7% and 59.2% respectively. These experiments average gradients over more steps as batch size falls and update the target after the accumulated online update. They do not establish that a one-step batch-64 run on a single GPU reproduces the large-batch result.

The authors attribute the low-batch degradation mainly to BN behavior. The plot below uses those reported values, rather than a synthetic extrapolation of the 1000-epoch headline result.

Augmentation Robustness

SimCLR's contrastive loss relies on augmentations to create "hard negatives" — pairs of different images that happen to look similar after augmentation. The difficulty of distinguishing positives from hard negatives is what drives learning. This creates a strong dependency on augmentation choice: the right augmentations produce informative hard negatives, while the wrong ones produce either trivially easy or impossibly hard pairs.

BYOL's self-prediction objective is inherently less sensitive to augmentation choice. The online network must predict the target's representation of a differently-augmented view, which requires learning features that are invariant to the augmentations. But because there are no negatives, there are no "hard negatives" whose quality depends on augmentation-induced similarity.

The paper's ablation on augmentation sensitivity confirms this. Removing color jittering and grayscale conversion together from the augmentation pipeline causes SimCLR's accuracy to drop by 22.2 percentage points — a catastrophic degradation showing strong sensitivity to this combined removal. BYOL's accuracy drops by only 9.1 points — still a meaningful drop, but far more graceful. Similar patterns hold for other augmentations: BYOL consistently degrades less when individual augmentation components are removed.

How BYOL Compares

Key Results

ImageNet Classification

Under linear evaluation (frozen backbone, trained linear classifier on top), BYOL achieves the following results:

Semi-Supervised Performance

BYOL's representations are effective even with very few labels. With only 1% of ImageNet labels (approximately 12,800 images), BYOL achieves 53.2% top-1 accuracy. With 10% of labels, it reaches 68.8%. These results demonstrate that BYOL's self-supervised features capture meaningful semantic information that transfers effectively with minimal supervision.

Critical Ablations

The paper's ablation studies reveal how tightly coupled BYOL's components are. Removing the momentum encoder entirely (setting \tau = 0, so the target is always an exact copy of the online network) causes complete collapse — accuracy drops to 0.3%. Using a random, fixed target (\tau = 1, so the target never updates) yields 18.8%, compared with 1.4% for the initial random representation. It improves on random features but remains well below the adaptive target’s 72.5% in Table 5(a).

Counterintuitively, adding negative pairs to BYOL's loss actually hurts performance — 70.9% compared to 72.5% without negatives (at 300 epochs). This comparison did not retune the temperature. Section 5 and Appendix F.4 report that properly tuning temperature lets the variant with negatives match BYOL; negatives do not inherently hurt the method.

Removing the predictor gives 0.2% in the reported Table 5 setup, reinforcing the empirical value of that asymmetry. The optimal momentum coefficient is \tau = 0.99 at 300 epochs, with the cosine schedule from 0.996 to 1.0 providing the best results at longer training durations.

Why BYOL Matters

BYOL was the first method to demonstrate that competitive self-supervised visual representations can be learned without any form of negative signal. Prior to BYOL, the field treated negatives as a necessary component — the question was how to provide them (large batches, memory banks, clustering), not whether they were needed at all.

By proving that negatives are sufficient but not necessary, BYOL opened a new research direction. A family of non-contrastive methods followed: SimSiam showed that even the momentum update could be removed with careful design; VICReg replaced the predictor with explicit variance-invariance-covariance regularization; Barlow Twins used a redundancy-reduction objective; and DINO extended the teacher-student paradigm to Vision Transformers with self-distillation.

The practical impact is that BYOL removes explicit negative mining and memory queues from the objective. Its predictor and EMA offer a different training design, while normalization, augmentation, optimizer settings, and update timing remain important engineering choices.

Key Takeaways

1. Negative pairs are not necessary for competitive self-supervised learning. BYOL learns without a cross-example contrastive term, although its loss still admits constant solutions.

2. Predictor and EMA matter in the reported setup. Their removal gives near-random Table 5 results; these are empirical ablations, not universal guarantees against collapse.

3. The conditional-expectation argument is an intuition. It assumes an optimal predictor under simplifying conditions, rather than proving that every encoder must remain informative.

4. Batch size still matters through normalization. Table 16 shows strong results across 256–4096 but substantial degradation at 64, with gradient accumulation used in the smaller-batch comparisons.

5. The update order and state boundary matter. Optimize the online weights first, then apply cosine-scheduled target EMA; target normalization state evolves separately during training forwards.

Related Reading

• VICReg — Variance-invariance-covariance regularization, another non-contrastive approach
• DINO — Self-distillation building on BYOL's teacher-student paradigm
• V-JEPA — Joint-embedding predictive architecture for video
• CLIP — Contrastive vision-language pretraining
• Attention Is All You Need — The transformer architecture