Skip to main content

Contrastive Loss for Representation Learning

Understand contrastive loss for representation learning: interactive demos of InfoNCE, triplet loss, and embedding space clustering with temperature tuning.

Contrastive Loss: Learning Representations by Comparison

Most loss functions compare a model's output to a fixed target -- cross-entropy compares predictions to labels, MSE compares values to ground truth. Contrastive loss does something fundamentally different: it compares data points to each other. Instead of asking "did you classify this correctly?", it asks "did you place similar things close together and different things far apart?"

This idea is the foundation of modern self-supervised learning. In SimCLR and MoCo, two augmentations of the same image define a positive pair without class labels. Other images provide candidate negatives, which can include semantically related false negatives. CLIP instead uses paired image-text supervision. Supervised contrastive learning can use class labels to define positives; these are distinct sources of supervision.

The Magnet Analogy

Attraction and repulsion are a useful analogy, but physical magnets attract at unlike poles and repel at like poles. In the authored figure, class labels explicitly define positive and negative pairs. This illustrates supervised pair selection; augmentation-based self-supervision does not know those class labels.

Pairs attract and repel
Step 0 / 5
CatsDogsBirds

Mathematical Foundation

Pair-based Contrastive Loss

The original contrastive loss (Chopra et al. 2005) operates on pairs. Given two embeddings z_i and z_j with a label y indicating whether they are from the same class (y=0) or different classes (y=1):

\mathcal{L} = (1-y) \cdot \frac{1}{2} d(z_i, z_j)^2 + y \cdot \frac{1}{2} \max(0, m - d(z_i, z_j))^2

When the pair is positive (y=0), the loss penalizes distance -- pulling them together. When negative (y=1), it only penalizes if the distance is less than the margin m -- pushing them apart until they are at least m units away.

Why Margin Matters

Without a margin, the loss would try to push negative pairs infinitely far apart. The margin sets a "good enough" threshold: once two dissimilar embeddings are separated by at least m, the loss for that pair drops to zero. This prevents the model from wasting capacity on already-separated pairs and focuses learning on the hard cases.

Embedding Space Explorer

The authored embedding animation illustrates the intended attraction and repulsion for class-defined pairs. It does not record a trained encoder or guarantee semantic clusters; the real result depends on pair construction, data, and optimization.

Embedding space
Training
Classes
Before training: 48 points, 12 per class
Class AClass BClass CClass D

Before training, embeddings are scattered randomly with no meaningful structure. Points from the same class have no reason to be near each other -- the separation ratio is close to 1.0.

Triplet Loss

Triplet loss (Schroff et al. 2015, FaceNet) improves on pair-based loss by considering three points simultaneously: an anchor, a positive (same class as anchor), and a negative (different class). Instead of absolute distances, it optimizes the relative ordering -- the positive should always be closer to the anchor than the negative, by at least a margin:

\mathcal{L}_{triplet} = \max(0, \; d(a, p) - d(a, n) + \text{margin})

The critical challenge is triplet mining -- choosing which triplets to train on. With N samples, there are O(N^3) possible triplets, but most are trivial (negative is already far away, loss is zero). The three mining strategies are:

Easy triplets have d(a,n) > d(a,p) + \text{margin}. The constraint is already satisfied, so loss is zero and gradients are zero -- no learning happens. Hard triplets have d(a,n) < d(a,p). The negative is closer than the positive, producing strong gradients but potentially unstable training. Semi-hard triplets have d(a,p) < d(a,n) < d(a,p) + \text{margin}. The negative is farther than the positive but still within the margin boundary -- these provide the most stable and informative gradients.

Click near any point to reposition it and watch the loss update in real-time. The selected point can also be moved with the coordinate controls.

Triplet loss
Preset
0.20
Move
0.40
0.50
Anchor, positive and negative points with their distances
A: AnchorP: Positive (same class)N: Negative (different class)
Tight (0.05)Wide (0.50)

Semi-hard Triplet: The negative is farther than the positive but still within the margin boundary. These triplets are the most informative for stable, efficient training.

InfoNCE and Modern Contrastive Learning

InfoNCE (Noise Contrastive Estimation) reframes contrastive learning as a classification problem: given an anchor, identify the positive pair from a set of N-1 negatives. The loss function is:

\mathcal{L}_{InfoNCE} = -\log \frac{\exp(\text{sim}(z_i, z_i^+) / \tau)}{\sum_{k=1}^{N} \exp(\text{sim}(z_i, z_k) / \tau)}

Where \text{sim}(\cdot, \cdot) is typically cosine similarity and \tau is the temperature parameter.

The Temperature Parameter

Temperature is arguably the most important hyperparameter in contrastive learning. It controls how sharply the model distinguishes between similar and dissimilar pairs. Low temperature (like CLIP's 0.07) makes the softmax distribution extremely peaked -- the model becomes very confident about which pairs are positive and focuses learning on the hardest negatives. High temperature (like SimCLR's 0.5) softens the distribution, giving all negatives a more equal vote and producing smoother gradients.

The mathematical intuition: dividing similarities by \tau before softmax amplifies differences. A similarity gap of 0.1 becomes a gap of 1.43 at \tau = 0.07 but only 0.2 at \tau = 0.5. Lower temperature means the model needs finer-grained discrimination.

NT-Xent: SimCLR's Variant

NT-Xent (Normalized Temperature-scaled Cross Entropy) is InfoNCE symmetrized over both augmented views. For each image, two random augmentations produce views z_i and z_j. The loss is computed from both directions and averaged. SimCLR showed that with strong augmentations and large batch sizes (4096+), this simple approach produces state-of-the-art representations.

MoCo: Decoupling Batch Size from Negatives

MoCo (Momentum Contrast) solves a practical problem: InfoNCE benefits from more negatives, but large batches require large GPUs. MoCo maintains a memory queue of 65,536 negative embeddings from previous batches, encoded by a slowly-updated momentum encoder. This decouples the number of negatives from the batch size, enabling contrastive learning on a single GPU.

Adjust temperature and the number of negatives to compare similarity scores with their softmax probabilities.

InfoNCE and temperature
0.07
Number of negatives
Common temperature values
Similarity scores and softmax probabilities
Positivesim: 0.8585.3%
Neg 1 (hard)sim: 0.7213.3%
Neg 2sim: 0.551.2%
Neg 3sim: 0.400.1%
Neg 4sim: 0.300.0%
Neg 5sim: 0.200.0%
Neg 6 (easy)sim: 0.100.0%
Sharp (0.01)Uniform (1.0)

Low temperature sharpens the similarity distribution, emphasizing the hardest negatives. This is the standard operating range for most contrastive learning methods.

Comparing Contrastive Methods

Each contrastive method makes different trade-offs between negative sampling efficiency, scalability, and training stability.

Contrastive methods
Contrastive Loss
Siamese / Pair loss
Chopra et al. 2005
Pulls pairs together, pushes non-pairs apart with margin
Negatives
moderate: One negative per pair
Scalability
moderate: Pair mining can be slow
Symmetry
excellent: Symmetric by design
Hard Mining
poor: No explicit hard mining
Best for: Face verification, signature matching
Triplet Loss
FaceNet loss
Schroff et al. 2015 (FaceNet)
max(0, d(a,p) - d(a,n) + margin)
Negatives
moderate: One negative per triplet
Scalability
moderate: O(N^3) triplet mining
Symmetry
poor: Anchor-dependent
Hard Mining
excellent: Semi-hard mining built-in
Best for: Face recognition, image retrieval
InfoNCE
NCE / Noise contrastive
van den Oord et al. 2018
Softmax over one positive vs N negatives
Negatives
excellent: Scales with batch or queue
Scalability
excellent: Batch-efficient
Symmetry
moderate: Can be symmetrized
Hard Mining
moderate: All negatives weighted by softmax
Best for: CLIP, MoCo, general representation learning
NT-Xent
SimCLR loss
Chen et al. 2020 (SimCLR)
Symmetric InfoNCE over augmented pairs
Negatives
excellent: 2(N-1) in-batch negatives
Scalability
moderate: Needs large batches (4096+)
Symmetry
excellent: Loss symmetrized both directions
Hard Mining
moderate: Temperature controls hardness focus
Best for: Self-supervised pretraining (SimCLR)
MoCo Loss
Momentum contrast
He et al. 2020 (MoCo)
InfoNCE with momentum encoder + queue
Negatives
excellent: 65536 via memory queue
Scalability
excellent: Decoupled from batch size
Symmetry
poor: Query-key asymmetric
Hard Mining
excellent: Queue provides diverse negatives
Best for: Self-supervised pretraining with limited compute
For large-scale pretraining:
  • Use InfoNCE or NT-Xent for in-batch training
  • Use MoCo when batch size is limited
  • Temperature in 0.05-0.1 range for discriminative features
For metric learning:
  • Use triplet loss with semi-hard mining
  • Consider pair loss for simple binary verification
  • Set margin based on embedding space geometry

Common Pitfalls

1. Representation Collapse

Positive-only matching has a trivial minimum when every input maps to the same vector. InfoNCE with negatives does not: if all N similarities are equal, its loss is log N. For N=4 that is about 1.386 nats; a separated positive can achieve a much lower loss. The denominator penalizes failure to distinguish positives from negatives. Contrastive training can still have optimization failures or dimensional collapse, so inspect covariance and downstream behavior. Non-contrastive methods such as BYOL and SimSiam use different stabilization mechanisms; stop-gradient alone is not a universal guarantee.

2. Insufficient Negatives

The number and diversity of negatives change the learning signal. Large batches and memory queues are two ways to supply negatives, but there is no universal required batch size or negative-count threshold. More negatives can also introduce false negatives; evaluate both representation quality and sampling assumptions.

3. Weak Augmentations

In self-supervised contrastive learning, augmentations define what "positive pair" means. If augmentations are too weak (only small crops and flips), the model learns to match trivial features like color histograms. Strong augmentations -- aggressive cropping, color jitter, Gaussian blur, grayscale conversion -- force the model to learn semantic features that survive transformation.

4. Temperature Too Low or Too High

Temperature below 0.01 causes the softmax to saturate, producing near-zero gradients for all but the single hardest negative. Temperature above 1.0 flattens the distribution so much that the model cannot distinguish hard from easy negatives. Start with 0.07--0.1 and tune based on the similarity distribution of your data.

5. False Negatives

In-batch negatives assume that different samples in the batch are semantically different. But in large datasets, the batch may contain two images of the same dog breed or two sentences with the same meaning. These "false negatives" receive repulsive gradients when they should receive attractive ones. Solutions include debiased contrastive loss, supervised contrastive loss, or simply using larger and more diverse batches.

If you found this explanation helpful, consider sharing it with others.

Mastodon