TL;DR
- SimCLR learns from two independently augmented views using a shared encoder, a nonlinear projection head and symmetric NT-Xent loss, with negatives drawn from the current batch.
- Crop and color-distortion composition helps prevent a color-statistics shortcut; augmentation choices define what the contrastive task asks the representation to ignore.
- Contrastive learning operates on the projected embedding, while downstream tasks use the encoder representation before projection.
- Larger batches provide more negatives per step, but training duration and optimizer settings also matter. Read linear evaluation separately from whole-network fine-tuning and compare backbone widths explicitly.
Paper Overview
SimCLR — A Simple Framework for Contrastive Learning of Visual Representations — demonstrates that a carefully designed but structurally minimal contrastive learning pipeline can surpass all prior self-supervised methods by a wide margin. No memory bank, no momentum encoder, no special architecture — just stochastic data augmentation, a shared encoder, a nonlinear projection head, and the NT-Xent contrastive loss.
Published at ICML 2020 by Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton at Google Brain, SimCLR achieves 69.3% top-1 accuracy on ImageNet linear evaluation with a standard ResNet-50 — 5.5 percentage points above CPC v2’s 63.8% with the same backbone (MoCo: 60.6%; PIRL: 63.6%). With a wider ResNet-50 (4x width), SimCLR reaches 76.5%, matching a standard-width supervised ResNet-50 trained on all of ImageNet’s labeled data. This is not an equal-width supervised comparison.
The paper’s contribution is not a single clever trick but a systematic empirical study that identifies three critical design decisions: (1) the composition of data augmentations matters far more than any individual augmentation, (2) a nonlinear projection head between the encoder and the contrastive loss provides a massive accuracy boost, and (3) larger batch sizes provide more negative pairs per step, directly improving representation quality. Each of these findings individually advances the field; together, they define a new baseline for self-supervised visual learning.
SimCLR Architecture
SimCLR’s architecture consists of four components arranged in a linear pipeline: stochastic data augmentation, a shared encoder, a projection head, and the contrastive loss.
Given an input image, SimCLR draws two independent augmentations to produce two views. Both views pass through the same encoder — a standard ResNet-50 — producing a 2048-dimensional representation \mathbf{h} for each view. There is no separate target network, no momentum encoder, no asymmetry between the two branches. The encoder is shared and receives gradients from both views symmetrically.
Each encoder output is then mapped through a projection head — a 2-layer MLP (2048 → 2048 → 128 with ReLU activation) — producing a 128-dimensional embedding \mathbf{z} in the space where the contrastive loss operates. After pretraining, the projection head is discarded entirely; only the encoder representations \mathbf{h} are used for downstream tasks. This separation between the contrastive objective space and the downstream representation space turns out to be one of SimCLR’s most important design choices.
The simplicity is the point. SimCLR showed that a carefully tuned combination of simple, well-understood ingredients outperforms more architecturally complex methods like MoCo (which requires a momentum encoder and a memory queue) and PIRL (which requires pretext task heads and memory banks).
The NT-Xent Loss
SimCLR uses the Normalized Temperature-scaled Cross-Entropy (NT-Xent) loss, a form of contrastive loss. For a batch of N images, SimCLR generates 2N augmented views. Each image produces exactly one positive pair (its two augmented views), and all remaining 2(N-1) views serve as negatives.
For a positive pair (i, j), the loss is:
where similarity is cosine similarity: \text{sim}(\mathbf{u}, \mathbf{v}) = \mathbf{u}^\top \mathbf{v} / (\|\mathbf{u}\| \|\mathbf{v}\|).
The temperature parameter \tau controls the sharpness of the softmax distribution. SimCLR uses \tau = 0.1 by default — a relatively sharp distribution that forces the model to focus on the hardest negatives. Too high a temperature spreads the probability mass across all negatives uniformly, reducing the learning signal. Too low a temperature makes the gradient dominated by a single hardest negative, causing instability.
The key difference from prior contrastive losses is the absence of any external negative storage. MoCo maintains a queue of 65,536 encoded negatives from previous batches. SimCLR draws all its negatives from the current batch alone. The candidate count therefore grows with batch size, while representation quality also depends on the training duration, augmentations, and optimizer settings.
The final loss averages \ell_{i,j} over all positive pairs in both directions (swapping i and j), yielding a symmetric objective.
The Augmentation Recipe
SimCLR’s most important empirical finding concerns data augmentation. The paper conducts a systematic ablation of every pairwise combination of augmentation operations — random cropping, color distortion (jitter, drop, grayscale), Gaussian blur, rotation, cutout, and Sobel filtering — measuring downstream accuracy for each pair.
The critical insight: random crop + color distortion is an especially effective combination. Figure 5 studies an asymmetric ablation, with the targeted transformations applied to one branch after an initial crop/resize. Those results should not be mixed with the symmetric default recipe: Table 1 reports 64.5% for strong color distortion with blur, on top of crop/resize and flip.
Why this particular pair? The answer reveals a subtle failure mode of contrastive learning. Different random crops of the same image tend to share similar color histograms. Without color distortion, the network can solve the contrastive task by simply matching global color statistics — a trivial shortcut that requires no understanding of visual content. Color jitter destroys this shortcut by randomizing the color distribution of each view independently, forcing the network to rely on spatial structure, texture, and semantic content to identify positive pairs.
Random cropping provides spatial diversity: the two views may depict different regions of the image, requiring the network to learn features that capture the shared semantic content rather than pixel-level patterns. Color distortion removes the color histogram shortcut. Together, they force the network to learn genuine visual concepts — object shape, texture, spatial relationships — rather than low-level statistical regularities.
The strength of color distortion must match the strength of the crop. Weak color jitter is insufficient to remove the color shortcut from aggressively cropped views. SimCLR uses strong color distortion (strength 1.0), which is significantly more aggressive than the augmentations used by prior methods.
Batch Size: More Negatives, Better Learning
SimCLR’s main practical limitation is its strong dependence on batch size. Because the contrastive loss draws all negatives from the current batch, the number of negatives per sample scales linearly with batch size: at batch size 256, each positive pair has only 510 negatives; at batch size 8192, it has 16,382 negatives.
More negatives create a harder discrimination task. With 510 negatives, the network can distinguish its positive pair using relatively coarse features — the odds of any negative being very similar to the positive are low. With 16,382 negatives, the probability of encountering hard negatives (images that look similar to the anchor after augmentation) increases substantially. The network must learn finer-grained, more discriminative features to maintain high accuracy on this harder task.
The empirical results confirm this relationship. With linear learning-rate scaling, batch size 256 achieves 57.5% top-1 accuracy at 100 epochs, while batch size 4096 reaches 64.6%. Table B.1 also reports square-root scaling: 62.8% and 64.5% respectively. Batch size 8192 provides a further marginal gain at that duration.
However, longer training partially compensates for smaller batches. At 800 epochs, batch size 256 reaches 66.6%, closing much of the gap with batch size 4096 at 100 epochs. The intuition is that over many epochs, a small-batch model encounters a similar total number of negative pairs as a large-batch model does in fewer epochs — though the per-step diversity is lower.
The default SimCLR configuration uses batch size 4096, trained on 128 TPU v3 cores. This is a significant computational requirement and one of the primary motivations behind subsequent methods like MoCo v2 (which uses a memory queue to decouple negatives from batch size) and BYOL (which eliminates negatives entirely).
The Projection Head Mystery
The nonlinear projection head is SimCLR’s most surprising and counterintuitive finding. Adding a 2-layer MLP (2048 → 2048 → 128 with ReLU) between the encoder and the contrastive loss improves linear evaluation accuracy by over 10 percentage points compared to applying the loss directly to the encoder output. A linear projection provides a more modest improvement; no projection at all performs worst.
The puzzle: the contrastive loss operates on the projected representation \mathbf{z} (128-dimensional), and this is the space that training explicitly optimizes. Yet the encoder representation \mathbf{h} (2048-dimensional) — which the loss never sees directly — consistently performs far better on downstream tasks. Why would an intermediate representation outperform the one that was actually optimized?
The explanation lies in what the contrastive loss encourages. The NT-Xent loss rewards invariance to the applied augmentations — two views of the same image should produce similar embeddings regardless of differences in crop region, color, or blur. This invariance is desirable for the contrastive task but destructive for downstream tasks. A downstream classifier might need to know about color (to distinguish a red car from a blue car) or spatial location (to detect object positions) — exactly the information that augmentation-invariance discards.
The projection head acts as an information bottleneck. It selectively discards augmentation-specific information as it maps from \mathbf{h} to \mathbf{z}, allowing \mathbf{h} to retain this information. The contrastive loss then optimizes \mathbf{z} for invariance without forcing \mathbf{h} to be invariant.
The paper tests this interpretation with transformation prediction. Its rotation experiment uses additional MLP classifiers and a special projection head with both h and g(h) of width 2048, while adding random rotations during pretraining. Table 3 reports 67.6% from h versus 25.6% from g(h), near the four-way chance rate. This supports information preservation before projection; it is distinct from the default 128-dimensional head.
Implementation
The distinction between h and z now becomes a concrete program. Follow the two view batches through their shared network, the positive indices, and the symmetric objective, then inspect how the encoder is frozen for a downstream classifier. The reduced CPU run makes these operations testable before returning to the paper’s large-scale evaluation.
This executable PyTorch reference follows SimCLR’s pairing and loss mechanics with a reduced CNN and random weights. It runs offline on CPU. The official checkpoints use TensorFlow; this example does not load pretrained SimCLR features or reproduce ImageNet training.
Download the Python program · Pinned requirements
python -m pip install -r requirements-simclr.txt
python simclr_training.py --smoke
1. Draw two independent views of each image
Original paper
The released input pipeline combines crop/resize, flip, color jitter, grayscale, and blur. This CPU reference uses torchvision equivalents; its crop is reduced to 32 pixels.
Math and tensor shapes
Calling the transform twice gives separate draws. Reusing one transformed tensor would make the positive task trivial. Jitter changes brightness, contrast, saturation, and hue before tensors enter the encoder. Pixel values remain in [0, 1].
Two [N, 3, 32, 32] batches; independently sampled transforms
Python implementation
def two_views(images, size=32):
"""Draw independent crop/color/blur transformations for every RGB image."""
if not images or not isinstance(size, int) or size < 8:
raise ValueError("Provide images and a crop size of at least 8.")
transform = transforms.Compose(
[
transforms.RandomResizedCrop(
size,
scale=(0.08, 1.0),
interpolation=transforms.InterpolationMode.BICUBIC,
),
transforms.RandomHorizontalFlip(),
transforms.RandomApply(
[transforms.ColorJitter(0.8, 0.8, 0.8, 0.2)], p=0.8
),
transforms.RandomGrayscale(p=0.2),
transforms.RandomApply(
[transforms.GaussianBlur(3, sigma=(0.1, 2.0))], p=0.5
),
transforms.ToTensor(),
]
)
first = torch.stack([transform(image.convert("RGB")) for image in images])
second = torch.stack([transform(image.convert("RGB")) for image in images])
return first, second2. Share the encoder and nonlinear projection head
Original paper
One encoder f and one projection head g process both branches. The paper uses ResNet representations and applies its contrastive objective after nonlinear projection.
Math and tensor shapes
The small CNN pools spatial positions into 32 features. A 32 → 32 → 16 MLP adds BatchNorm and ReLU, following the released head structure at reduced width. The final BatchNorm here omits both learned scale and offset; the official head retains a learned scale. These are educational dimensions; the paper uses h of width 2048 and z of width 128.
[2N, 3, 32, 32] → h [2N, 32] → z [2N, 16]
Python implementation
class SimCLR(nn.Module):
"""Share one reduced encoder and one nonlinear head across both views."""
def __init__(self, representation_dim=32, projection_dim=16):
super().__init__()
if any(
not isinstance(dim, int) or dim < 1
for dim in (representation_dim, projection_dim)
):
raise ValueError("Feature dimensions must be positive.")
self.encoder = nn.Sequential(
nn.Conv2d(3, 16, 3, padding=1),
nn.ReLU(),
nn.Conv2d(16, representation_dim, 3, stride=2, padding=1),
nn.ReLU(),
nn.AdaptiveAvgPool2d(1),
nn.Flatten(),
)
self.head = nn.Sequential(
nn.Linear(representation_dim, representation_dim, bias=False),
nn.BatchNorm1d(representation_dim),
nn.ReLU(),
nn.Linear(representation_dim, projection_dim, bias=False),
nn.BatchNorm1d(projection_dim, affine=False),
)
def forward(self, views):
if views.ndim != 4 or views.shape[1] != 3 or min(views.shape[2:]) < 8:
raise ValueError("Expected [B, 3, H, W] with H and W at least 8.")
h = self.encoder(views)
return h, self.head(h)3. Compute cosine similarity in projection space
Original paper
The official objective normalizes the projected vectors before computing temperature-scaled pairwise dot products.
Math and tensor shapes
Normalization makes the dot product depend on direction rather than feature magnitude. A matrix multiplication computes all candidates at once. Zero or nonfinite vectors trigger an explicit error because their cosine directions are undefined.
z [2N, 16] → S [2N, 2N]; Sᵢᵢ = 1 before masking
Python implementation
def normalized_similarity(z):
"""Return cosine similarities after checking nonzero finite embeddings."""
if z.ndim != 2 or z.shape[0] < 4 or z.shape[0] % 2 or z.shape[1] < 1:
raise ValueError("Expected [2N, D] with N >= 2 and D >= 1.")
if not torch.is_floating_point(z) or not torch.isfinite(z).all():
raise ValueError("Embeddings must be finite floating-point values.")
if torch.any(z.norm(dim=1) <= 1e-12):
raise ValueError("Cosine similarity requires nonzero embeddings.")
unit_z = F.normalize(z, dim=1)
return unit_z @ unit_z.T4. Map each anchor to the other view of its image
Original paper
The released objective splits a concatenated batch into its two view batches. Matching positions in those halves are positives.
Math and tensor shapes
Here subscripts label view number. The paper algorithm interleaves views; this program groups all first views before all second views. The ordering changes indices, not the objective. Apply the same image permutation to both halves to preserve pairs.
N = 2: order [A₁, B₁, A₂, B₂], positives [2, 3, 0, 1]
Python implementation
def positive_indices(batch_size, device=None):
"""Map [all first views, all second views] to the other view of each image."""
if not isinstance(batch_size, int) or batch_size < 2:
raise ValueError("At least two images are required for negatives.")
indices = torch.arange(2 * batch_size, device=device)
return (indices + batch_size) % (2 * batch_size)5. Exclude self, retaining the positive and every negative
Original paper
The diagonal of each same-view similarity block is masked in the released objective. Cross-view positives stay among its softmax candidates.
Math and tensor shapes
Masking only the diagonal prevents matching a vector to itself. Masking the positive as well would remove the target from the denominator. Temperature must be positive and finite. The single matrix is equivalent to the official four similarity blocks.
Each anchor: 2N − 1 candidates = 1 positive + 2N − 2 negatives
Python implementation
def contrastive_logits(z, temperature=0.1):
"""Exclude only self-similarity; the positive remains in the denominator."""
if not math.isfinite(temperature) or temperature <= 0:
raise ValueError("Temperature must be finite and positive.")
similarity = normalized_similarity(z)
logits = similarity / temperature
self_mask = torch.eye(z.shape[0], dtype=torch.bool, device=z.device)
return logits.masked_fill(self_mask, -torch.inf)6. Average NT-Xent over both directions
Original paper
Algorithm 1 includes both directions of every positive pair, then averages over 2N anchors.
Math and tensor shapes
Cross-entropy evaluates the log-sum-exp stably and treats p(i) as the target class. This implements the paper’s mean; the released TensorFlow code sums the two directional means, giving twice this scalar. For identical nonzero embeddings, every candidate has equal probability and the loss is log(2N − 1).
Logits [2N, 2N], targets [2N] → one scalar
Python implementation
def nt_xent(z, temperature=0.1, reduction="mean"):
"""Average both positive directions as in Algorithm 1, not just one half."""
logits = contrastive_logits(z, temperature)
targets = positive_indices(z.shape[0] // 2, z.device)
return F.cross_entropy(logits, targets, reduction=reduction)7. Update encoder and head together
Original paper
Both f and g are updated to reduce the symmetric objective. No target network or momentum copy is introduced.
Math and tensor shapes
Concatenating views into one forward pass makes shared parameters explicit. Backpropagation reaches both the head and the encoder. The offline smoke uses SGD for one update; it omits distributed global BN, LARS, large batches, and the paper’s training schedule.
N = 4 smoke check: h [8, 32], z [8, 16]
Python implementation
def train_step(model, optimizer, first, second, temperature=0.1):
"""Update the same encoder and head jointly from both view branches."""
if first.ndim != 4 or first.shape != second.shape or first.shape[0] < 2:
raise ValueError(
"Both view batches must have matching shapes and N >= 2."
)
model.train()
optimizer.zero_grad(set_to_none=True)
h, z = model(torch.cat([first, second], dim=0))
loss = nt_xent(z, temperature)
loss.backward()
optimizer.step()
return {
"loss": loss.item(),
"h_shape": list(h.shape),
"z_shape": list(z.shape),
}8. Train a downstream classifier on frozen h
Original paper
Linear evaluation trains a classifier on the frozen base network. The projection head is discarded for this downstream representation.
Math and tensor shapes
Evaluation mode freezes BatchNorm statistics as well as parameter gradients. Only the classifier optimizer steps. The behavior test verifies that encoder tensors and buffers stay unchanged. A probe update here demonstrates freezing, without claiming useful learned features or benchmark accuracy.
h [B, 32] → classifier logits [B, C]; labels [B]
Python implementation
def linear_probe_step(encoder, classifier, optimizer, images, labels):
"""Train only a classifier on h; freeze parameters and BN statistics."""
if images.ndim != 4 or images.shape[1] != 3 or min(images.shape[2:]) < 8:
raise ValueError("Expected images [B, 3, H, W], H and W at least 8.")
if (
labels.ndim != 1
or labels.shape[0] != images.shape[0]
or labels.dtype != torch.long
):
raise ValueError("Labels must be int64 [B] matching the image batch.")
encoder.requires_grad_(False)
encoder.zero_grad(set_to_none=True)
encoder.eval()
classifier.train()
with torch.no_grad():
h = encoder(images)
logits = classifier(h)
if torch.any(labels < 0) or torch.any(labels >= logits.shape[1]):
raise ValueError("Labels must index the classifier classes.")
optimizer.zero_grad(set_to_none=True)
loss = F.cross_entropy(logits, labels)
loss.backward()
optimizer.step()
return loss.item()Inspect a real two-image batch
Seed 7 produces these independent 32 × 32 views of two licensed photographs. The saved similarities come from the reduced randomly initialized CNN, in training mode with batch statistics. They show the arithmetic, not semantic quality.
View 1
View 2
View 1
View 2
Rows and columns follow A₁, B₁, A₂, B₂ (cat, coffee). Bold cells are positives; diagonal self-matches are excluded from the loss.
| Anchor | A₁ | B₁ | A₂ | B₂ |
|---|---|---|---|---|
| A₁ | self | -0.541 | -0.132 | -0.008 |
| B₁ | -0.541 | self | -0.165 | -0.358 |
| A₂ | -0.132 | -0.165 | self | -0.735 |
| B₂ | -0.008 | -0.358 | -0.735 | self |
At τ = 0.1, per-anchor losses are 1.4987, 2.0910, 0.5445, 3.5339. Their symmetric mean is 1.9170. With four identical views the reference value would be log(3) ≈ 1.0986.
Download full precision values and provenance
Reproduce this inspection
From the repository root, run python examples/simclr_training.py --export public/examples/simclr-demo. The exporter reads the existing ViT demo’s original cat and coffee photographs and verifies their SHA-256 hashes before sampling. It uses Python 3.13.13, PyTorch 2.12.0+cpu, and torchvision 0.27.0+cpu; seed 7 and two CPU threads. No checkpoint is loaded.
How SimCLR Compares
Self-Supervised Method Comparison
How SimCLR compares to other self-supervised learning frameworks on ImageNet linear evaluation.
| Method | Memory Bank | Batch Requirement | Top-1 (%) | Top-5 (%) | Key Mechanism |
|---|---|---|---|---|---|
| SimCLR | Not needed | Default: 4096 | 69.3 | 89.0 | NT-Xent contrastive loss |
| MoCo | Queue | Any size | 60.6 | — | Momentum queue encoder |
| PIRL | Memory bank | Moderate | 63.6 | — | Pretext-invariant representations |
| CPC v2 | Not needed | Moderate | 63.8 | 85.3 | Autoregressive prediction |
| BYOL | Not needed | Any size | 74.3 | 91.6 | Predictor + EMA target |
| Supervised | N/A | Any size | 76.5 | — | Cross-entropy + labels |
SimCLR
NT-Xent contrastive loss
MoCo
Momentum queue encoder
PIRL
Pretext-invariant representations
CPC v2
Autoregressive prediction
BYOL
Predictor + EMA target
Supervised
Cross-entropy + labels
SimCLR's key insight
- Simplicity: no memory bank, no momentum encoder, no special architecture
- Just augmentation + large batch + nonlinear projection head
- Improves on CPC v2 by 5.5 points with the same ResNet-50 backbone
Trade-offs
- Studies batch sizes 256–8192; default ImageNet batch: 4096
- Uses 32–128 TPU cores depending on batch size in the paper
- Later methods (BYOL, DINO) change the objective and computational trade-offs
Key Results
ImageNet Classification
Under linear evaluation (frozen backbone, trained linear classifier on top), SimCLR achieves the following results:
| Model | Top-1 | Top-5 |
|---|---|---|
| SimCLR ResNet-50 | 69.3% | 89.0% |
| SimCLR ResNet-50 (2x) | 74.2% | 92.0% |
| SimCLR ResNet-50 (4x) | 76.5% | 93.2% |
| Supervised ResNet-50 | 76.5% | — |
Semi-Supervised Performance
SimCLR’s representations transfer effectively with minimal labels. With only 1% of ImageNet labels, fine-tuning achieves 75.5% top-5 accuracy (ResNet-50) and 85.8% top-5 (ResNet-50 4x). With 10% of labels, the 4x model reaches 92.6% top-5. These semi-supervised experiments fine-tune the whole network; they are separate from frozen-backbone linear evaluation.
Training Configuration
SimCLR uses the LARS optimizer with a learning rate of 4.8 (computed as 0.3 × 4096/256), with linear warmup for 10 epochs followed by cosine decay to zero. The default configuration trains for 100 epochs with batch size 4096 on 128 TPU v3 cores. Extended training up to 800–1000 epochs further improves results, particularly for smaller batch sizes where more epochs compensate for fewer negatives per step.
Why SimCLR Matters
SimCLR established that contrastive learning with a simple, unified framework could match the accuracy of fully supervised pretraining on ImageNet. Prior self-supervised methods operated in a regime where they trailed supervised baselines by 10–20 percentage points; SimCLR closed this gap to zero (with a sufficiently wide backbone), fundamentally shifting the field’s expectations for what unsupervised learning could achieve.
Three findings from SimCLR became standard practice across the field. First, the importance of augmentation composition — specifically the crop + color jitter pair — was adopted wholesale by subsequent methods. MoCo v2 explicitly credits SimCLR’s augmentation pipeline for much of its improvement over MoCo v1. Second, nonlinear projection heads became ubiquitous; virtually every SSL method published after SimCLR includes one. Third, the systematic analysis of batch size and negative sampling informed the design of methods that sought to decouple representation quality from computational scale.
SimCLR’s direct influence on subsequent work is extensive. MoCo v2 adopted SimCLR’s augmentation pipeline and projection head, immediately improving upon MoCo v1. BYOL built on SimCLR’s framework while removing negative pairs entirely, raising the question of whether contrastive negatives were ever necessary. SimCLR v2 extended the framework with knowledge distillation for further improvement in the semi-supervised setting. The emphasis on thorough, systematic empirical study — ablating each component in isolation to understand its contribution — set a template for how self-supervised learning methods are evaluated and compared.
Key Takeaways
-
Simplicity wins — no memory bank, no momentum encoder, just a shared encoder with a nonlinear projection head and contrastive loss improves the same-backbone ImageNet linear-evaluation result by 5.5 points over CPC v2.
-
Augmentation composition is critical — random crop + color distortion is the essential pair, preventing the color histogram shortcut that would otherwise allow the network to solve the contrastive task without learning semantic features.
-
The nonlinear projection head provides 10+ points improvement by decoupling the contrastive objective from downstream utility — augmentation-specific information is selectively discarded in \mathbf{z} while preserved in \mathbf{h}, as confirmed by the rotation prediction experiment.
-
Batch size sets the number of in-batch negatives — larger batches help especially at shorter training durations, while longer training can reduce the gap.
-
Temperature \tau = 0.1 is the sweet spot — sharp enough to focus learning on hard negatives, soft enough to maintain gradient flow across the full batch.
Related Reading
- BYOL — Removes negative pairs entirely while building on SimCLR’s augmentation pipeline and projection head design
- DINO — Self-distillation approach for Vision Transformers that also eliminates explicit negatives
- VICReg — Alternative non-contrastive objective using variance-invariance-covariance regularization
- V-JEPA — Joint-embedding predictive architecture for video representation learning
- CLIP — Scales contrastive learning to vision-language pairs with natural language supervision
