Skip to main content

The Modality Gap in Multimodal AI

Measure the centroid gap between normalized image and text embeddings, then check why a smaller gap alone does not guarantee correct retrieval.

Shared space does not mean identical distributions

A contrastive image–text model compares the direction of an image embedding with the directions of candidate caption embeddings. The encoders share a coordinate system, but their outputs can have different average directions. That separation is one form of modality gap.

The useful question is what that separation tells us about matching an image to its caption. A global distance between two collections does not identify which individual pairs are correct. We will measure both.

For how contrastive models such as CLIP compare with models that project vision into an LLM (BLIP-2, LLaVA, Flamingo), see vision-language alignment compared.

Define the gap on normalized embeddings

For n paired, nonzero image and text vectors, normalize each embedding before taking the means:

v_i = \frac{f_I(x_i)}{\lVert f_I(x_i)\rVert_2},\qquad t_i = \frac{f_T(y_i)}{\lVert f_T(y_i)\rVert_2}
\mu_V = \frac{1}{n}\sum_{i=1}^{n}v_i,\qquad \mu_T = \frac{1}{n}\sum_{i=1}^{n}t_i,\qquad g = \lVert\mu_V - \mu_T\rVert_2

Here, g is the Euclidean distance between the two centroids. The means need not have unit length; do not normalize them again when computing this measure. Because each mean lies in the unit ball, 0 \le g \le 2.

This definition gives a common scale for a specified set of normalized embeddings. It does not measure distribution overlap, the minimum distance between two manifolds, or the correctness of the pairing. A 2D projection of real embeddings also need not preserve this full-dimensional distance.

Shift the centroids, then inspect the same query

The figure uses four fixed, invented pairs on the 2D unit circle, so the plotted geometry and computed scores use the same coordinates. Start with the Train query. Its caption ranks third before the shift.

The single intervention uses the original centroid difference \Delta = \mu_V - \mu_T:

v_i' = \operatorname{normalize}(v_i - \tfrac{1}{2}\Delta),\qquad t_i' = \operatorname{normalize}(t_i + \tfrac{1}{2}\Delta)

Before the final normalization, both centroids meet at their midpoint. Normalizing each shifted vector changes the means again, so the remaining gap is 0.290, not zero. In this fixture it falls from 1.182, and correct top-one retrieval rises from 1 of 4 queries to 2 of 4. Train becomes correct; Boat and Cat remain incorrect.

These results describe this constructed example. The shift changes individual directions and relative cosine scores; it is not a trained adapter or a general recipe for improving a real model.

A zero gap can still give the wrong matches

Consider four image embeddings at the cardinal directions: right, up, left, down. Put the four caption embeddings at exactly the same points, but assign each caption to the next image around the circle. Both centroids are zero, so g=0. Yet every image retrieves the wrong caption: 0 of 4 top-one matches.

Permuting caption identities leaves their mean unchanged. It can change every positive pair. Evaluate the gap alongside named positive-pair ranks and retrieval on held-out examples, using the same candidate set before and after an intervention.

Why score rescaling is different

For a fixed query and all its candidates, any common positive scale a>0 and offset b preserve their order:

s_j > s_k\quad\Longleftrightarrow\quad a s_j + b > a s_k + b

Changing a temperature at inference can change softmax probabilities or a threshold decision, but cannot improve top-one retrieval by itself. Adding a uniform “gap correction” has the same limitation. Clipping adjusted scores can create ties rather than recover the correct correspondence. Changing a temperature during training can affect gradients and the learned embeddings, which is a different intervention.

Reproduce the measurement on CPU

This standalone example measures normalized centroid distance and positive ranks before and after the same shift. Equal scores use caption index as the tie breaker, matching the figure.

python
# src/pages/concepts/transformers/modality-gap/index.mdx
import math

import torch
import torch.nn.functional as F


def unit_directions(angles):
    """Create deterministic 2D unit embeddings from angles in degrees."""
    radians = torch.tensor(angles, dtype=torch.float64) * math.pi / 180
    return torch.stack((radians.cos(), radians.sin()), dim=1)


def measure(images, texts):
    """Measure normalized centroid distance and paired image-to-text ranks."""
    if (images.norm(dim=1) == 0).any() or (texts.norm(dim=1) == 0).any():
        raise ValueError("Zero vectors have no direction to normalize.")
    images = F.normalize(images, dim=1)
    texts = F.normalize(texts, dim=1)
    image_mean = images.mean(dim=0)
    text_mean = texts.mean(dim=0)
    scores = images @ texts.T
    order = torch.argsort(scores, dim=1, descending=True, stable=True)
    ranks = order.argsort(dim=1).diagonal() + 1
    return {
        "images": images,
        "texts": texts,
        "image_mean": image_mean,
        "text_mean": text_mean,
        "gap": (image_mean - text_mean).norm().item(),
        "scores": scores,
        "ranks": ranks,
        "correct": (ranks == 1).sum().item(),
    }


images = unit_directions([-25, -10, 15, 35])
texts = unit_directions([50, 70, 100, 120])
before = measure(images, texts)
delta = before["image_mean"] - before["text_mean"]
after = measure(before["images"] - 0.5 * delta,
                before["texts"] + 0.5 * delta)

for name, result in (("before", before), ("after", after)):
    print(name, f"gap={result['gap']:.3f}",
          f"correct={result['correct']}/4", result["ranks"].tolist())

# Identical centroids can conceal entirely incorrect positive pairs.
cardinal = torch.tensor([[1, 0], [0, 1], [-1, 0], [0, -1]],
                        dtype=torch.float64)
permuted = measure(cardinal, cardinal.roll(1, dims=0))
print("permuted", permuted["gap"], permuted["correct"])

The output is before gap=1.182 correct=1/4 [1, 2, 3, 4], then after gap=0.290 correct=2/4 [1, 2, 1, 2]. The final case prints a zero gap and zero correct matches.

What the research supports

Liang et al. investigate modality separation in multimodal contrastive learning. Their experiments connect distinct embedding cones to encoder initialization and study how optimization and temperature preserve or reduce the gap. In their CLIP embedding-shift experiment on MSCOCO captions, closing the gap at the learned temperature increased contrastive loss; temperature changes altered that landscape.

Those findings explain why closing a centroid gap is not automatically desirable. They do not establish a universal gap size, an information-theoretic penalty, or a performance guarantee for every vision–language architecture. For a real intervention, record the model checkpoint, dataset, normalization, gap definition, candidate set, and task results before drawing a conclusion.

References

If you found this explanation helpful, consider sharing it with others.

Mastodon