SigLIP: Sigmoid Loss for Language Image Pre-Training

TL;DR

• SigLIP replaces CLIP’s softmax (InfoNCE) loss with a pairwise sigmoid loss that treats every image–text pair as an independent binary classification, removing the need for a global normalization step.
• No global softmax denominator is needed. The paper computes blocks of pair losses while exchanging text features across devices, reducing matrix memory without eliminating communication for negatives.
• In the paper’s matched experiments, SigLIP improves on the softmax baseline below 32K batch size and peaks at 32K. This is a result for those model/data/schedule settings, not a universal minimum batch size.
• A learnable bias and positive logit scale address the natural 1-positive vs. (N−1)-negatives imbalance and reduce the initial correction driven by negatives.

The cost of softmax

CLIP’s contrastive loss is the InfoNCE objective. For a batch of N image–text pairs, the image-side loss is:

where z_i and z_j' are L2-normalised image and text embeddings, and t is a learned positive logit scale (inverse temperature). A symmetric text-side loss is averaged in.

The denominator is the critical problem: computing \sum_{j} \exp(\cdot) requires every text embedding in the batch. When training is distributed across many accelerators, this means performing an expensive all-gather communication collective to assemble the full batch on every device before the loss can be computed. The dense pairwise similarity matrix grows quadratically with batch size; communication cost depends on the collective, embedding size and hardware, and is not generally quadratic. The result is a tight coupling between batch size, number of devices, and per-step training time.

Sigmoid loss

SigLIP proposes a replacement that removes the global normalization requirement. Instead of an N-way classification, it casts each of the N^2 possible image–text pairs as an independent binary classification: does this pair match (+1) or not (−1)?

where y_{ij} = +1 if i = j (matching pair) and y_{ij} = -1 otherwise, t is a learned positive logit scale, and b is a learned bias. Each term depends only on a single pair of embeddings — there is no normalization across rows or columns. A chunked implementation avoids an all-gather of the full batch, but exchanges features so cross-device negative pairs are still evaluated. The objective sums all N² contributions and divides by N, rather than averaging over N².

The demo above makes the structural difference concrete. In softmax mode, shading any one cell requires the full row (all text embeddings) to be present before the denominator can be evaluated. In sigmoid mode, each cell’s value \sigma(t z_i^\top z_j' + b) is determined solely by the embeddings of image i and text j, with no dependence on any other pair. This makes pair terms independently accumulable. Shared encoders still receive the combined gradients from all their pairs.

Batch size, freed

The loss of one fixed pair has no denominator coupling it to other pairs. Changing batch size still changes how many negative pairs are sampled and therefore changes the full loss and gradient. The paper studies this separately in §4.7; independence of each term does not mean batch-independent representation quality.

In §4.2, the B/16 image and B-sized text models trained on WebLI favor SigLIP over the softmax baseline below 32K. SigLIP peaks at 32K, while the softmax baseline peaks at 98K without surpassing it; 307K hurts both. On four TPU-v4 chips, the authors fit a batch of 4096 with Base SigLIP versus 2048 with the corresponding CLIP model. These are paper results, separate from the exact pair counts in the demo.

Taming the imbalance

Removing the softmax normalization introduces a new challenge. At initialization, the model assigns roughly equal similarities to all pairs. But in a batch of N pairs there is exactly 1 positive per row and N−1 negatives. With zero bias and similarities near zero, each pair scores near 0.5 under the sigmoid, so each positive and negative has logit-gradient magnitude 0.5 before the shared 1/N reduction. The much larger number of negatives therefore dominates the aggregate and drives a large initial correction.

SigLIP addresses this with the learnable bias b. Initialised to a large negative value (around −10), it shifts the sigmoid so that all pairs start with probabilities close to zero. The zero-similarity prior depends on N: for N>1, an exactly prior-matched constant bias would be b=-\log(N-1). The -10 initialization is a training recipe, not an analytic optimum for every N. Negative pairs near zero similarity then start close to probability 0, and positive pairs receive a strong gradient signal right from the start. The positive logit scale t=\exp(a) controls sharpness and is learned jointly. The executable reference explicitly initializes t=10 and b=-10, storing a=log(10).

The bias slider in the demo shows this effect directly. At b = 0, the decision boundary sits at s = 0 (cosine similarity = 0), and the curve is broadly centred — negatives receive large gradients. At b = -10, the decision boundary shifts right to s=-b/t, so higher similarity is required to score above 0.5. This suppresses negative logit gradients near zero similarity while keeping positive logit gradients strong. The demo uses an explicitly illustrative reciprocal-temperature convention, t=1/τ.

Implementation

With normalized features, signed labels, scale and bias in place, the original objective can be checked directly. The following reduced reference compares dense all-pairs loss with serial chunk accumulation and their derivatives.

def normalize_features(image, text):
    """Normalize paired [N, D] features; zero rows stay zero with epsilon."""
    _features(image, text, paired=True)
    image_norm = torch.linalg.vector_norm(image, dim=-1, keepdim=True)
    text_norm = torch.linalg.vector_norm(text, dim=-1, keepdim=True)
    if not torch.isfinite(image_norm).all() or not torch.isfinite(text_norm).all():
        raise ValueError("feature norms must be finite")
    # Match the pinned two_towers.py norm + 1e-8 convention.
    return image / (image_norm + NORM_EPS), text / (text_norm + NORM_EPS)

def pair_logits(image, text, log_scale, bias):
    """Map normalized [R, D], [C, D] features to logits [R, C]."""
    _features(image, text)
    _parameters(log_scale, bias, image)
    logits = log_scale.exp() * (image @ text.T) + bias
    if not torch.isfinite(logits).all():
        raise ValueError("computed logits must be finite")
    return logits

def signed_labels(image_indices, text_indices, dtype=torch.float64):
    """Use global indices, including chunks with no positive pairs."""
    for indices in (image_indices, text_indices):
        if (not isinstance(indices, torch.Tensor) or indices.ndim != 1
                or indices.numel() == 0 or indices.dtype != torch.int64
                or indices.device.type != "cpu"):
            raise ValueError("indices must be nonempty CPU int64 vectors")
        if (indices < 0).any() or indices.unique().numel() != indices.numel():
            raise ValueError("indices must be nonnegative and unique")
    if dtype not in (torch.float32, torch.float64):
        raise ValueError("label dtype must be float32/64")
    matches = image_indices[:, None] == text_indices[None, :]
    return 2 * matches.to(dtype) - 1

def stable_loss(logits, labels, batch_size):
    """Sum stable signed pair losses and divide by global N, never N squared."""
    _positive_int(batch_size, "batch_size")
    if (not isinstance(logits, torch.Tensor) or logits.ndim != 2
            or min(logits.shape) < 1
            or logits.dtype not in (torch.float32, torch.float64)
            or logits.device.type != "cpu" or not torch.isfinite(logits).all()):
        raise ValueError("logits must be a nonempty finite CPU float matrix")
    if (not isinstance(labels, torch.Tensor) or labels.shape != logits.shape
            or labels.dtype != logits.dtype or labels.device != logits.device
            or not ((labels == 1) | (labels == -1)).all()):
        raise ValueError("labels must match logits and contain only -1 or +1")
    return F.softplus(-labels * logits).sum() / batch_size

def chunked_loss(image, text, log_scale, bias, chunk_size):
    """Accumulate global-index chunks with the same total sum/N reduction."""
    _positive_int(chunk_size, "chunk_size")
    zimage, ztext = normalize_features(image, text)
    n = image.shape[0]
    total = image.new_zeros(())
    for i in range(0, n, chunk_size):
        rows = torch.arange(i, min(i + chunk_size, n))
        for j in range(0, n, chunk_size):
            cols = torch.arange(j, min(j + chunk_size, n))
            logits = pair_logits(zimage[rows], ztext[cols], log_scale, bias)
            labels = signed_labels(rows, cols, image.dtype)
            total = total + stable_loss(logits, labels, n)
    return total

python -m pip install -r requirements-siglip.txt
python siglip.py --smoke --seed 7

Why it mattered

SigLIP arrived at a moment when the community had assumed large batch sizes were a fundamental requirement for contrastive vision–language training. Its matched experiments demonstrated stronger performance at smaller batch sizes and a simpler memory-efficient loss implementation. Avoiding a full similarity matrix and global softmax normalization supports distributed chunking, while cross-device feature exchange remains necessary for negatives. The numerical reference above verifies the loss mechanics, not those measured training outcomes.

SigLIP has become a default replacement for CLIP encoders in downstream vision–language models. PaLI-3 trains its 2B-parameter ViT-G vision backbone with the SigLIP recipe, and open VLMs such as PaliGemma, Idefics2 and LLaVA-OneVision initialise their vision encoders from the released SigLIP So400m checkpoint. The simplicity of the sigmoid formulation also makes it straightforward to extend: the loss can be applied asymmetrically, combined with masked prediction heads, or adapted to multi-label settings without the structural constraints imposed by softmax normalisation.

This page covers the original SigLIP objective. Later SigLIP2 models add their own training changes and should not be inferred from this reference.

Related Reading

• CLIP — the softmax contrastive predecessor that SigLIP replaces, establishing the dual-encoder vision–language framework
• CoCa — combines contrastive and captioning losses in a single model, complementing SigLIP’s efficiency-focused approach
• BLIP-2 — bridges frozen vision encoders (CLIP ViT-L/14 or EVA-CLIP ViT-g/14) with large language models via a lightweight Q-Former