Shared space does not mean identical distributions
A contrastive image–text model compares the direction of an image embedding with the directions of candidate caption embeddings. The encoders share a coordinate system, but their outputs can have different average directions. That separation is one form of modality gap.
The useful question is what that separation tells us about matching an image to its caption. A global distance between two collections does not identify which individual pairs are correct. We will measure both.
For how contrastive models such as CLIP compare with models that project vision into an LLM (BLIP-2, LLaVA, Flamingo), see vision-language alignment compared.
Define the gap on normalized embeddings
For n paired, nonzero image and text vectors, normalize each embedding before taking the means:
Here, g is the Euclidean distance between the two centroids. The means need not have unit length; do not normalize them again when computing this measure. Because each mean lies in the unit ball, 0 \le g \le 2.
This definition gives a common scale for a specified set of normalized embeddings. It does not measure distribution overlap, the minimum distance between two manifolds, or the correctness of the pairing. A 2D projection of real embeddings also need not preserve this full-dimensional distance.
Shift the centroids, then inspect the same query
The figure uses four fixed, invented pairs on the 2D unit circle, so the plotted geometry and computed scores use the same coordinates. Start with the Train query. Its caption ranks third before the shift.
The single intervention uses the original centroid difference \Delta = \mu_V - \mu_T:
Before the final normalization, both centroids meet at their midpoint. Normalizing each shifted vector changes the means again, so the remaining gap is 0.290, not zero. In this fixture it falls from 1.182, and correct top-one retrieval rises from 1 of 4 queries to 2 of 4. Train becomes correct; Boat and Cat remain incorrect.
These results describe this constructed example. The shift changes individual directions and relative cosine scores; it is not a trained adapter or a general recipe for improving a real model.
A zero gap can still give the wrong matches
Consider four image embeddings at the cardinal directions: right, up, left, down. Put the four caption embeddings at exactly the same points, but assign each caption to the next image around the circle. Both centroids are zero, so g=0. Yet every image retrieves the wrong caption: 0 of 4 top-one matches.
Permuting caption identities leaves their mean unchanged. It can change every positive pair. Evaluate the gap alongside named positive-pair ranks and retrieval on held-out examples, using the same candidate set before and after an intervention.
Why score rescaling is different
For a fixed query and all its candidates, any common positive scale a>0 and offset b preserve their order:
Changing a temperature at inference can change softmax probabilities or a threshold decision, but cannot improve top-one retrieval by itself. Adding a uniform “gap correction” has the same limitation. Clipping adjusted scores can create ties rather than recover the correct correspondence. Changing a temperature during training can affect gradients and the learned embeddings, which is a different intervention.
Reproduce the measurement on CPU
This standalone example measures normalized centroid distance and positive ranks before and after the same shift. Equal scores use caption index as the tie breaker, matching the figure.
# src/pages/concepts/transformers/modality-gap/index.mdx
import math
import torch
import torch.nn.functional as F
def unit_directions(angles):
"""Create deterministic 2D unit embeddings from angles in degrees."""
radians = torch.tensor(angles, dtype=torch.float64) * math.pi / 180
return torch.stack((radians.cos(), radians.sin()), dim=1)
def measure(images, texts):
"""Measure normalized centroid distance and paired image-to-text ranks."""
if (images.norm(dim=1) == 0).any() or (texts.norm(dim=1) == 0).any():
raise ValueError("Zero vectors have no direction to normalize.")
images = F.normalize(images, dim=1)
texts = F.normalize(texts, dim=1)
image_mean = images.mean(dim=0)
text_mean = texts.mean(dim=0)
scores = images @ texts.T
order = torch.argsort(scores, dim=1, descending=True, stable=True)
ranks = order.argsort(dim=1).diagonal() + 1
return {
"images": images,
"texts": texts,
"image_mean": image_mean,
"text_mean": text_mean,
"gap": (image_mean - text_mean).norm().item(),
"scores": scores,
"ranks": ranks,
"correct": (ranks == 1).sum().item(),
}
images = unit_directions([-25, -10, 15, 35])
texts = unit_directions([50, 70, 100, 120])
before = measure(images, texts)
delta = before["image_mean"] - before["text_mean"]
after = measure(before["images"] - 0.5 * delta,
before["texts"] + 0.5 * delta)
for name, result in (("before", before), ("after", after)):
print(name, f"gap={result['gap']:.3f}",
f"correct={result['correct']}/4", result["ranks"].tolist())
# Identical centroids can conceal entirely incorrect positive pairs.
cardinal = torch.tensor([[1, 0], [0, 1], [-1, 0], [0, -1]],
dtype=torch.float64)
permuted = measure(cardinal, cardinal.roll(1, dims=0))
print("permuted", permuted["gap"], permuted["correct"])
The output is before gap=1.182 correct=1/4 [1, 2, 3, 4], then after gap=0.290 correct=2/4 [1, 2, 1, 2]. The final case prints a zero gap and zero correct matches.
What the research supports
Liang et al. investigate modality separation in multimodal contrastive learning. Their experiments connect distinct embedding cones to encoder initialization and study how optimization and temperature preserve or reduce the gap. In their CLIP embedding-shift experiment on MSCOCO captions, closing the gap at the learned temperature increased contrastive loss; temperature changes altered that landscape.
Those findings explain why closing a centroid gap is not automatically desirable. They do not establish a universal gap size, an information-theoretic penalty, or a performance guarantee for every vision–language architecture. For a real intervention, record the model checkpoint, dataset, normalization, gap definition, candidate set, and task results before drawing a conclusion.
References
- Liang et al., “Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning”. The direct source for the initialization, optimization, and embedding-shift analysis above.
- Geirhos et al., “ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness”. Context on learned visual representations, rather than a measurement of the image–text centroid gap.
- Thrush et al., “Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality”. Context on evaluating image–caption correspondence beyond global geometry.
Related concepts
Trace one image-caption pair through cosine scores, contrastive positives and negatives, a trainable projection, and retrieval evaluation.
Explore a fixed compute budget with Chinchilla’s text-only loss fit, then see why multimodal scaling needs its own data, objectives, and evaluation metrics.
Trace frozen weights and trainable LoRA matrices, calculate adapter parameter counts, and compare bottleneck adapters, prefix tuning and BitFit for vision-language models.
CLIP, BLIP-2, LLaVA and Flamingo align images with text by different routes. Compare their objectives, bridges, data, and which parts train or stay frozen.
Understanding complete, dimensional, and cluster collapse — the failure modes that every self-supervised method must prevent. Learn why collapse happens and how contrastive, asymmetric, regularization, and masking approaches solve it.
Trace a target query through source keys, attention weights, and a weighted value mix. Understand cross-attention origins, shapes, masks, and applications.
