VICReg: Self-Supervised Learning Without Collapse

Paper Overview

Self-supervised learning (SSL) aims to learn useful representations from unlabeled data by training models to be invariant to different augmented views of the same input. The central challenge: if you only optimize for invariance, the model discovers a trivial shortcut — map everything to the same constant vector. Loss drops to zero, but the representation is useless.

Previous methods prevent this "representation collapse" through architectural tricks: SimCLR uses large batches of negative pairs, BYOL adds a momentum-updated teacher network, and Barlow Twins minimizes redundancy in a cross-correlation matrix. VICReg takes a different approach — it directly attacks collapse through three explicit regularization terms applied to the embedding space, requiring no negative pairs or momentum encoder. It is less reliant on large batches for negatives, but its variance and covariance estimates still depend on batch size.

Published at ICLR 2022 by Adrien Bardes, Jean Ponce, and Yann LeCun (Meta AI / NYU), VICReg achieves competitive performance with a remarkably simple and principled design.

The Collapse Problem

Representation collapse is the fundamental failure mode of self-supervised learning. When a model is trained to produce similar embeddings for augmented views of the same image, the easiest solution is to ignore the input entirely and output a constant vector. The invariance loss becomes zero — but the model has learned nothing.

In the paper’s ablations, invariance alone collapses. The timing depends on architecture and optimization; the animation below illustrates the failure mode rather than reporting a measured training trajectory.

VICReg Architecture

VICReg follows the standard joint-embedding framework used across SSL methods, but the loss computation is where it diverges from everything else.

The architecture has four stages:

1. Data augmentation: Each input image generates two views through random cropping, color jitter, Gaussian blur, and horizontal flipping. The two augmentation pipelines are sampled independently.

2. Shared encoder: Both views pass through the same backbone (typically ResNet-50), producing representations h and h'. The encoder weights are shared — not copied with momentum like BYOL.

3. Expander MLP: A three-layer MLP (typically 8192 units per layer) projects representations into a higher-dimensional space where the loss is computed. This separation is crucial — the loss operates on the expanded space while downstream tasks use the encoder output.

4. VICReg loss: Three terms computed on the expander outputs Z and Z', each targeting a specific failure mode.

The Three Loss Terms

VICReg's key insight is decomposing the representation quality problem into three separately measurable, interacting objectives.

Variance

The variance term prevents collapse by ensuring that embedding dimensions maintain sufficient variance across the batch:

This is a hinge loss: when the standard deviation of dimension j drops below the threshold γ (set to 1), the loss activates and pushes it back up. If all dimensions maintain healthy variance, this term contributes zero — it only intervenes when collapse begins.

A constant representation has zero variance and a positive hinge penalty. With epsilon 0.0001, that penalty is 0.99 per branch. At an exactly constant batch its gradient is nevertheless zero, so the penalty is not a theorem of guaranteed escape from every collapsed stationary point.

Invariance

The invariance term is straightforward — minimize the mean squared error between paired embeddings:

This is the standard objective shared across all joint-embedding methods. Two augmented views of the same image should produce similar representations. Without the other two terms, optimizing invariance alone leads directly to collapse.

Covariance

The covariance term decorrelates embedding dimensions, preventing redundancy:

where C(Z) is the sample covariance matrix across the batch, using denominator n − 1. Shrinking off-diagonal entries reduces linear correlation between dimensions. Uncorrelated dimensions can still be statistically dependent, so this does not guarantee independent information or maximum capacity.

Combined Loss

The total VICReg loss combines all three terms with weighting coefficients:

The paper uses λ = 25, μ = 25, ν = 1. Both branches receive variance and covariance regularization. The released code averages the two variance terms and uses elementwise mean squared error over n × d; the equations above display the paper’s branch-sum and feature-sum notation. The implementation below follows the released reductions exactly, so its coefficients and numeric losses must be read with that scaling.

Understanding Covariance Regularization

The covariance term deserves deeper attention because it addresses a subtle problem. Even without collapse, embedding dimensions can become highly correlated — multiple dimensions encoding the same information. This wastes representational capacity.

If many dimensions move together across the batch, the representation can contain redundant linear directions. Covariance regularization discourages that redundancy without guaranteeing that each coordinate corresponds to a separate visual concept.

The paper motivates decorrelation as a way to preserve information. Covariance alone does not establish an entropy or mutual-information optimum: those quantities also depend on the distribution and constraints. The diagram below illustrates changing covariance, while the implementation makes the actual penalty inspectable.

How VICReg Compares

VICReg sits within a family of methods that prevent collapse without negative pairs. Each takes a fundamentally different approach to the same problem.

The closest relative is Barlow Twins, which operates on a cross-correlation matrix between branches. VICReg instead regularizes within-branch covariance matrices. The key difference: Barlow Twins pushes the entire cross-correlation matrix toward the identity (on-diagonal toward 1, off-diagonal toward 0), while VICReg decomposes this into separate variance and covariance objectives. This decomposition gives VICReg more flexibility — the variance threshold γ provides explicit control over how much spread is required.

Implementation

The three loss terms now translate into a concrete two-view training step. Follow the source, tensor shapes, and exact executed Python together, then inspect genuine frozen features from the official release before turning to benchmark results.

The offline PyTorch program executes the released VICReg loss with a reduced random-weight CNN. A separate inspection below loads the genuine official pretrained ResNet-50 backbone. The smoke update demonstrates loss mechanics; the frozen features demonstrate checkpoint inference. Neither reproduces the paper’s ImageNet benchmark.

Download Python program · Pinned requirements

python -m pip install -r requirements-vicreg.txt
python vicreg_training.py --smoke

1. Sample two augmented views

Original paper

Enlarge figure

Math and tensor shapes

x=t(i)

x'=t'(i)

The CPU profile draws each transform independently. Both branches use the same reduced augmentation distribution here, omitting the official asymmetric solarization and using a 3-pixel blur kernel. This is a mechanics profile, not the original 224-pixel training recipe.

Two [N, 3, 32, 32] batches; N >= 2

Python implementation

python

def two_views(images, size=32):
    """Sample two view batches with reduced crop/color/blur augmentation."""
    if len(images) < 2 or not isinstance(size, int) or size < 8:
        raise ValueError("Provide at least two images and size >= 8.")
    transform = transforms.Compose(
        [
            transforms.RandomResizedCrop(
                size,
                scale=(0.08, 1.0),
                interpolation=transforms.InterpolationMode.BICUBIC,
            ),
            transforms.RandomHorizontalFlip(),
            transforms.RandomApply(
                [transforms.ColorJitter(0.4, 0.4, 0.2, 0.1)], p=0.8
            ),
            transforms.RandomGrayscale(p=0.2),
            transforms.RandomApply(
                [transforms.GaussianBlur(3, (0.1, 2.0))], p=0.5
            ),
            transforms.ToTensor(),
            transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)),
        ]
    )
    first = torch.stack([transform(image.convert("RGB")) for image in images])
    second = torch.stack([transform(image.convert("RGB")) for image in images])
    return first, second

2. Share the encoder and expander

Original paper

Enlarge figure

Math and tensor shapes

h=f(x)

z=g(h)

The small CNN retains separate h and z tensors so the loss acts after expansion. Hidden width and output width are reduced to 16. There is no predictor, momentum teacher, stop-gradient branch, or embedding normalization. The trained expander is discarded for downstream tasks.

Reference h [N, 32], z [N, 16]; paper h [N, 2048], z [N, 8192]

Python implementation

python

class VICReg(nn.Module):
    """Small shared encoder with the released three-linear-layer head pattern."""

    def __init__(self, representation_dim=32, projection_dim=16):
        super().__init__()
        if any(
            not isinstance(d, int) or d < 1
            for d in (representation_dim, projection_dim)
        ):
            raise ValueError("Feature dimensions must be positive integers.")
        self.encoder = nn.Sequential(
            nn.Conv2d(3, 16, 3, padding=1),
            nn.ReLU(),
            nn.Conv2d(16, representation_dim, 3, stride=2, padding=1),
            nn.ReLU(),
            nn.AdaptiveAvgPool2d(1),
            nn.Flatten(),
        )
        self.expander = nn.Sequential(
            nn.Linear(representation_dim, projection_dim),
            nn.BatchNorm1d(projection_dim),
            nn.ReLU(),
            nn.Linear(projection_dim, projection_dim),
            nn.BatchNorm1d(projection_dim),
            nn.ReLU(),
            nn.Linear(projection_dim, projection_dim, bias=False),
        )

    def forward(self, images):
        if (
            images.ndim != 4
            or images.shape[1] != 3
            or min(images.shape[2:]) < 8
        ):
            raise ValueError("Expected images [N, 3, H, W], H and W >= 8.")
        if self.training and images.shape[0] < 2:
            raise ValueError("Training BatchNorm needs at least two images.")
        h = self.encoder(images)
        return h, self.expander(h)

3. Match corresponding projected vectors

Original paper

Enlarge figure

Math and tensor shapes

s=\frac{1}{ND}\sum_{i,j}(z_{ij}-z'_{ij})^2

Matching row indices must identify views of the same image. Equation 5 in the paper displays an average of squared Euclidean distances over N; the released elementwise mean divides that expression by D. This program follows the released reduction, including its gradient scaling.

z and z′ [N, D] → scalar s

Python implementation

python

def invariance_loss(first, second):
    """Mean squared difference over both N and D, as in the released code."""
    validate_embeddings(first, second)
    return F.mse_loss(first, second)

4. Keep sample standard deviations above the hinge

Original paper

Enlarge figure

Math and tensor shapes

\sigma_j^2=\operatorname{Var}(z^j)+10^{-4}

v(Z)=\operatorname{mean}_j[1-\sigma_j]_+

PyTorch correction=1 states the unbiased sample reduction explicitly. A constant batch has standard deviation 0.01 and a variance penalty 0.99. Its gradient is zero at exactly constant embeddings; positive loss does not guarantee escape from that stationary point. Near-constant gradients remain finite with epsilon.

[N, D] → std [D] → scalar; V = (v(Z) + v(Z′))/2

Python implementation

python

def variance_loss(first, second, epsilon=1e-4):
    """Average branch hinge penalties with unbiased sample variance."""
    validate_embeddings(first, second)
    if not isinstance(epsilon, (int, float)) or not 0 < epsilon <= 1:
        raise ValueError("Epsilon must be in (0, 1].")
    first = first - first.mean(dim=0)
    second = second - second.mean(dim=0)
    std_first = torch.sqrt(first.var(dim=0, correction=1) + epsilon)
    std_second = torch.sqrt(second.var(dim=0, correction=1) + epsilon)
    return F.relu(1 - std_first).mean() / 2 + F.relu(1 - std_second).mean() / 2

5. Penalize both halves of off-diagonal covariance

Original paper

Enlarge figure

Math and tensor shapes

X=Z-\bar Z

C=X^\top X/(N-1)

c=\frac{1}{D}\sum_{j\ne k}C_{jk}^2

Both (j,k) and (k,j) count. Dividing by the number of off-diagonal entries would change the objective. The two branch penalties are added, not averaged. Lower covariance means less linear correlation; it does not establish statistical independence.

[N, D] → [D, D]; D(D − 1) off-diagonal entries

Python implementation

python

def off_diagonal(matrix):
    """Select every off-diagonal entry, including both symmetric halves."""
    if matrix.ndim != 2 or matrix.shape[0] != matrix.shape[1]:
        raise ValueError("Expected a square covariance matrix.")
    mask = ~torch.eye(matrix.shape[0], dtype=torch.bool, device=matrix.device)
    return matrix[mask]


def covariance_loss(first, second):
    """Sum both branch covariance penalties, dividing each by D, not D²."""
    validate_embeddings(first, second)
    n, d = first.shape
    first = first - first.mean(dim=0)
    second = second - second.mean(dim=0)
    cov_first = first.T @ first / (n - 1)
    cov_second = second.T @ second / (n - 1)
    return (
        off_diagonal(cov_first).square().sum()
        + off_diagonal(cov_second).square().sum()
    ) / d

6. Combine the released 25 / 25 / 1 terms

Original paper

Enlarge figure

Math and tensor shapes

V=(v(Z)+v(Z\prime))/2

K=c(Z)+c(Z\prime)

\mathcal L=25s+25V+K

Paper Equation 6 displays a sum of variance branches; this executable uses the released branch average. Keep those conventions distinct when comparing numbers. A matching constant pair has s=0, K=0 and total loss 24.75 in this implementation.

Three scalar terms → weighted scalar loss

Python implementation

python

def vicreg_loss(first, second):
    """Return released 25/25/1 total and individually inspectable terms."""
    invariance = invariance_loss(first, second)
    variance = variance_loss(first, second)
    covariance = covariance_loss(first, second)
    total = 25 * invariance + 25 * variance + covariance
    return total, {
        "invariance": invariance,
        "variance": variance,
        "covariance": covariance,
    }

7. Backpropagate through both views

Original paper

Enlarge figure

Math and tensor shapes

\theta\leftarrow\theta-\eta\nabla_\theta\mathcal L

Two forwards call the same model. Autograd reaches both the CNN and MLP, and the optimizer applies one update. This single-process reference computes statistics on its local batch; it omits the released differentiable all-gather, mixed precision, SyncBatchNorm, LARS, and ImageNet schedule.

Smoke: N = 4, h [4, 32], z [4, 16]; one SGD update

Python implementation

python

def train_step(model, optimizer, first, second):
    """Backpropagate through both branches and update shared parameters."""
    if first.shape != second.shape or first.ndim != 4 or first.shape[0] < 2:
        raise ValueError("View batches must match with N >= 2.")
    model.train()
    optimizer.zero_grad(set_to_none=True)
    h_first, z_first = model(first)
    _, z_second = model(second)
    loss, terms = vicreg_loss(z_first, z_second)
    loss.backward()
    gradient_norm = torch.sqrt(
        sum(
            p.grad.square().sum()
            for p in model.parameters()
            if p.grad is not None
        )
    )
    optimizer.step()
    return {
        "loss": loss.item(),
        "terms": {name: value.item() for name, value in terms.items()},
        "gradientNorm": gradient_norm.item(),
        "hShape": list(h_first.shape),
        "zShape": list(z_first.shape),
    }

8. Inspect the released frozen backbone

Original paper

Enlarge figure

Math and tensor shapes

h=f_{\mathrm{frozen}}(x)

\operatorname{cos}(h_i,h_j)=u_i^\top u_j

Check the checkpoint SHA-256, load every tensor strictly, freeze parameters and BatchNorm buffers, and apply the official evaluation transform. Feature normalization happens only after extraction for cosine inspection. No linear head is trained or ImageNet accuracy computed here.

Four licensed-photo inputs [4, 3, 224, 224] → h [4, 2048]

Python implementation

python

def load_frozen_backbone(checkpoint):
    """Strictly load all 318 official tensors, with no classifier or expander."""
    checkpoint = Path(checkpoint)
    if (
        hashlib.sha256(checkpoint.read_bytes()).hexdigest()
        != CHECKPOINT_SHA256
    ):
        raise ValueError("Official checkpoint SHA-256 mismatch.")
    backbone = models.resnet50(weights=None)
    backbone.fc = nn.Identity()
    backbone.load_state_dict(
        torch.load(checkpoint, map_location="cpu", weights_only=True),
        strict=True,
    )
    backbone.requires_grad_(False)
    return backbone.eval()


def frozen_features(backbone, images):
    """Apply the official evaluation transform; return unnormalized h."""
    if not images:
        raise ValueError("Provide at least one RGB image.")
    transform = transforms.Compose(
        [
            transforms.Resize(
                256,
                interpolation=transforms.InterpolationMode.BILINEAR,
                antialias=True,
            ),
            transforms.CenterCrop(224),
            transforms.ToTensor(),
            transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)),
        ]
    )
    backbone.eval()
    backbone.requires_grad_(False)
    batch = torch.stack([transform(image.convert("RGB")) for image in images])
    with torch.no_grad():
        return backbone(batch), batch

Genuine frozen feature inspection

The official ResNet-50 checkpoint produces 2048-dimensional features from these two CC0 photographs and their deterministic horizontal flips. Cosine scores compare these four inputs only. They are neither class probabilities nor an estimate of benchmark quality.

Original

Flip

Original

Flip

Rows and columns: A = cat, B = coffee; A′ and B′ = horizontal flips.

Cosine similarity of frozen official VICReg features

Input	A	B	A′	B′
A	1.000	0.258	0.995	0.265
B	0.258	1.000	0.259	0.989
A′	0.995	0.259	1.000	0.267
B′	0.265	0.989	0.267	1.000

Unnormalized feature norms: 26.685, 18.452, 26.411, 18.341. These are backbone h, not expander z.

Separately, the same photos in a random small CNN give invariance 0.1584, variance 0.6163, covariance 1.1789 and weighted loss 20.5458 at seed 7. This untrained 32-pixel example checks arithmetic and has no semantic-quality claim.

Full precision values and provenance · All four feature vectors

Checkpoint, preprocessing, and reproduction

Architecture: ResNet-50, fc Identity; 318 strictly loaded tensors. Official source revision: 4e12602fd495af83efd1631fbe82523e6db092e0.

Official backbone checkpoint SHA-256: c843e7652491ff2f712734619fd74d85e5e92ac902615665ad3fd50dc6ada591.

RGB PIL; short-side Resize(256), bilinear, antialias=True; CenterCrop(224); ToTensor [0,1]; Normalize mean [0.485,0.456,0.406], std [0.229,0.224,0.225]. Deterministic horizontal flip before preprocessing for second pair.

CPU float32; Python 3.13.13, PyTorch 2.12.0+cpu, torchvision 0.27.0+cpu, Pillow 12.2.0; two threads, seed 7. Photos are hash-verified against the existing licensed ViT demo sources.

python examples/vicreg_training.py --export public/examples/vicreg-demo --checkpoint .context/paper-series/weights/vicreg-resnet50.pth

Run from the repository root with the checkpoint saved at the shown path. No weights are downloaded automatically. The JSON records program, checkpoint, photo, crop, and feature-artifact hashes.

Key Results

ImageNet Linear Evaluation

The paper’s Table 1 reports frozen-backbone ImageNet linear evaluation. VICReg uses a ResNet-50 pretrained for 1000 epochs, then a supervised linear classifier. Comparator results follow their cited protocols; this is not a matched 100-epoch experiment for every method.

VICReg matches Barlow Twins in top-1 accuracy and exceeds the listed SimCLR result; BYOL is higher in this table. See Table 1 and §5.1 for the protocols. The checkpoint inspection above has no classifier and does not measure any of these accuracies.

Transfer Learning

On downstream tasks including PASCAL VOC detection, COCO detection, and iNaturalist classification, VICReg transfers competitively with state-of-the-art methods. The quality of learned representations generalizes well beyond ImageNet.

Critical Ablations

The paper's ablation study reveals important insights:

• Removing variance term: Performance drops to near-random — collapse occurs. This is the most critical component.
• Removing covariance term: Table 7 reports 57.5% for invariance + variance with coefficients 1/1/0, versus 68.6% for VICReg at 25/25/1. These changed coefficients prevent interpreting the difference as a matched-weight 1–2% ablation.
• Removing invariance term: Variance and covariance still supply gradients, but do not align paired views; the paper reports collapse in the tested configuration.
• Expander architecture: The released model uses three linear layers, with BatchNorm/ReLU on the first two. This is the tested recipe, not a universal optimum over depths.
• Expander width: Table 12 reports 68.6% at width 8192 and 68.8% at width 16384 after 100 epochs. Wider heads improve this tested range with diminishing gains; 8192 is a practical recipe, not an exact optimum.

Why VICReg Matters

VICReg's contribution is not just another SSL method — it is a conceptual framework for understanding representation quality. By decomposing the objective into variance, invariance, and covariance, it provides interpretable diagnostics: you can independently monitor each property during training to understand what is happening in the representation space.

This decomposition also enables targeted interventions. If your model's embeddings are collapsing, increase the variance weight. If dimensions are redundant, increase the covariance weight. These diagnostics guide experiments; the resulting accuracy still depends on data, augmentation, batch statistics, and optimization.

The simplicity of VICReg — no momentum encoder, no negative mining, no online clustering — makes it an excellent starting point for practitioners exploring self-supervised learning. The barrier to implementation is low, and the three loss terms provide clear intuition about what the model is optimizing.

Key Takeaways

1. Collapse is a central challenge of self-supervised learning. VICReg counters it with an explicit variance penalty, with the exact-constant stationary-point caveat above.

2. Three interacting objectives (variance, invariance, covariance) expose complementary diagnostics, without guaranteeing high-quality features for arbitrary architectures or datasets.

3. No negative pairs or momentum required — VICReg achieves competitive results with a simpler architecture than SimCLR or BYOL.

4. Covariance regularization discourages linear redundancy; decorrelation does not prove statistical independence.

5. The expander network is critical — loss is computed in a high-dimensional projected space while downstream tasks use the encoder's lower-dimensional output.

Related Reading

• Attention Is All You Need — The transformer architecture that powers modern encoders
• CLIP — Contrastive learning applied to vision-language alignment
• EfficientNet — Efficient backbone architectures commonly used with SSL
• Vision Transformer — ViT patch-based architecture often used as SSL encoder