Skip to main content

The Vision-Language Alignment Problem

Trace one image-caption pair through cosine scores, contrastive positives and negatives, a trainable projection, and retrieval evaluation.

The Vision-Language Alignment Problem

An image encoder can represent a dog in a park, and a text encoder can represent “a dog playing in a park.” Those vectors need a learned relationship before their dot product can tell us that they describe the same scene. Equal dimensions alone do not give the two encoders a shared meaning.

Images begin as spatial measurements; text begins as a sequence of discrete tokens. Each encoder also chooses what to preserve. An image might make the dog's fur and the grass prominent, while its caption emphasizes playing. Alignment asks the representations to agree on the information a task needs, such as retrieving a matching caption. Combining visual tokens with a language model is a related problem, but it need not use the same global retrieval objective.

Start with a single image query. Its positive caption must compete against other captions before we can decide whether alignment helped.

Follow One Named Pair

Select Dog in the explorer. Its positive is “A dog playing in a park.” The other five captions are negatives for this toy batch. At the starting position, the positive cosine is −0.946 and its rank is 5 of 6. The meal caption scores 0.936 and wins retrieval, even though the labels tell us it describes a different scene.

Move Illustration progress to 50%. The dog caption's cosine becomes 0.164 and its rank becomes 3 of 6. At the scripted endpoint it scores 1.000 and ranks first. The rings, highlighted candidate and selected matrix row always refer to the image you chose.

The slider rotates unit vectors toward assigned targets. It does not optimize a loss, learn from images, or measure a model's training progress. These constructed 2D positions make a narrower point: retrieval depends on the positive's score relative to every candidate, rather than an absolute cosine threshold. The next step is to express that competition as a trainable objective.

Positives and Negatives in a Contrastive Batch

For a batch of N paired images and captions, let the image and text embeddings be v_i and t_j. Normalize them before forming cosine scores:

\hat{v}_i = \frac{v_i}{\lVert v_i \rVert_2}, \qquad \hat{t}_j = \frac{t_j}{\lVert t_j \rVert_2}, \qquad s_{ij} = \hat{v}_i^T \hat{t}_j

Row i compares one image with every caption. With one positive per image, the diagonal entry s_{ii} is the positive and the other entries are in-batch negatives. Divide scores by a positive temperature \tau and apply cross-entropy:

\mathcal{L}_{I\to T} = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(s_{ii}/\tau)}{\sum_{j=1}^{N} \exp(s_{ij}/\tau)}

For the dog image, the numerator uses the dog caption. The denominator includes that positive and all five negatives. A competing caption with a large score increases the loss; moving every caption close to the image does not solve the competition.

Caption-to-image retrieval asks the reverse question. The same score matrix is normalized down each column, so the dog caption must select the dog image over the other images:

\mathcal{L}_{T\to I} = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(s_{ii}/\tau)}{\sum_{k=1}^{N} \exp(s_{ki}/\tau)}, \qquad \mathcal{L} = \frac{\mathcal{L}_{I\to T} + \mathcal{L}_{T\to I}}{2}

CLIP uses this symmetric contrastive setup with separate image and text encoders. Its original experiment used 400 million image-text pairs and learned the logit scale. The fixed temperature below keeps the small example explicit; that data scale is not a minimum requirement for every adaptation task.

This diagonal-label formulation assumes each row and column has exactly one labeled positive. Real batches can contain duplicate scenes or several valid captions. Treating another valid description as a negative creates a false negative; the labels or objective must reflect multiple positives when the task has them. Hard negatives can help distinguish similar scenes, but only when they really are incorrect matches.

Give the Loss Something to Train

A loss cannot update two frozen encoders if every operation between their outputs and the loss is also frozen. A trainable projection provides one possible bridge:

z_i = W v_i + b

It can change the dimension and orientation of a fixed feature space. Whether a linear map has enough capacity depends on what those features preserve and what the task requires. Nonlinear adapters and token-level fusion offer different capacity and computation trade-offs.

The standalone example below takes six synthetic frozen image features of width 8 and text features of width 4. It trains an 8-to-4 linear image projection for one genuine SGD step. Only that projection has gradients; normalization and symmetric cross-entropy connect it to the loss. The synthetic features demonstrate the computation, not a trained image encoder or a generalization result.

python
# src/pages/concepts/transformers/alignment-problem/index.mdx
import torch
from torch import nn
from torch.nn import functional as F


def contrastive_loss(image_embeddings, text_embeddings, temperature=0.07):
    """Return symmetric loss and logits for one positive per paired row."""
    images = F.normalize(image_embeddings, dim=-1)
    texts = F.normalize(text_embeddings, dim=-1)
    logits = images @ texts.T / temperature
    labels = torch.arange(logits.shape[0], device=logits.device)
    loss = 0.5 * (
        F.cross_entropy(logits, labels)
        + F.cross_entropy(logits.T, labels)
    )
    return loss, logits, labels


def retrieval_metrics(similarities):
    """Rank diagonal positives; exact ties keep candidate index order."""
    order = torch.argsort(similarities, dim=1, descending=True, stable=True)
    positives = torch.arange(similarities.shape[0], device=similarities.device)
    ranks = (order == positives[:, None]).to(torch.int64).argmax(dim=1) + 1
    return {
        "ranks": ranks,
        "R@1": (ranks <= 1).float().mean().item(),
        "R@5": (ranks <= 5).float().mean().item(),
    }


# CPU keeps this tiny example reproducible without a model download.
device = torch.device("cpu")
torch.manual_seed(7)
image_features = torch.randn(6, 8, device=device).detach()
text_features = torch.randn(6, 4, device=device).detach()
projector = nn.Linear(8, 4, bias=False).to(device)
optimizer = torch.optim.SGD(projector.parameters(), lr=0.05)
image_before = image_features.clone()
text_before = text_features.clone()
weight_before = projector.weight.detach().clone()

optimizer.zero_grad()
projected = projector(image_features)
loss, logits, labels = contrastive_loss(projected, text_features)
loss_before = loss.item()
loss.backward()
gradient_norm = projector.weight.grad.norm().item()
optimizer.step()

with torch.no_grad():
    updated = projector(image_features)
    after, _, _ = contrastive_loss(updated, text_features)
    loss_after = after.item()
    cosine = F.normalize(updated, dim=-1) @ F.normalize(text_features, dim=-1).T
    image_to_text = retrieval_metrics(cosine)
    text_to_image = retrieval_metrics(cosine.T)

assert gradient_norm > 0
assert not torch.equal(projector.weight.detach(), weight_before)
assert torch.equal(image_features, image_before)
assert torch.equal(text_features, text_before)
print(f"Loss {loss_before:.3f} -> {loss_after:.3f}; gradient norm {gradient_norm:.3f}")
print("Image-to-text ranks:", image_to_text["ranks"].tolist())
print("Text-to-image ranks:", text_to_image["ranks"].tolist())

The update proves that a trainable bridge can receive a gradient through frozen features. A lower loss on this batch says nothing about retrieval on unseen images. It also does not prescribe which parts of a real model to freeze. LiT, for example, keeps a pretrained image model locked while training the text side. CLIP jointly trains its dual encoders. Those are different training choices for a shared contrastive goal.

Measure the Task, Not an Ideal Cosine

For image-to-text retrieval, rank the annotated positive among all candidate captions. Recall at K (R@K) is the fraction of queries whose positive appears in the first K results under the stated labeling convention. Compute text-to-image retrieval separately: its queries and candidate pool run in the opposite direction.

The code sorts by full score and breaks exact ties by candidate index, matching the explorer. If all three candidates tie for three queries, their diagonal positive ranks are 1, 2 and 3, so R@1 is one third. Counting only strictly greater scores would optimistically assign rank 1 to every tied positive even though the displayed first result can only be one caption.

MeasurementWhat to report with it
Cosine similarityNormalization, positive and negative score distributions, and the candidate pool
Retrieval R@KDirection, K, dataset split, pool size, valid positives and tie policy
Semantic consistencyThe specific attribute or relation tested and how correctness is labeled
Zero-shot transferDataset, prompts, class set and the downstream evaluation protocol

There is no universal “good” cosine of 0.7 or retrieval accuracy of 90%. Adding harder candidates can lower recall without changing the encoder. With six candidates, R@5 is already a lenient measure, and R@10 includes the whole pool. Report held-out results and compare models under the same protocol.

What a Shared Space Still Misses

Global pair matching is useful, but it does not guarantee that every object, attribute or relationship is aligned. The dog caption can retrieve a park scene while confusing which animal is playing. The same issue appears when a car image emphasizes color and wheels but its caption emphasizes motion, or a food image shows an arrangement that its caption describes as a cuisine.

A granularity mismatch arises when a single caption summarizes a scene and the task needs a particular region, object or token. Image features are not always pixel-level, and captions are not always coarse. Region-text supervision, richer captions or token-level interaction can address different gaps; multi-scale features alone do not establish semantic correspondence.

Training data also determines which languages, cultures and visual contexts are represented. Audit caption quality and duplicates, include the intended deployment domains, and evaluate language and subgroup coverage explicitly. A strong average retrieval score can conceal systematic failures in underrepresented groups. CLIP's paper discusses dataset bias and limitations alongside its transfer results.

Match the Mechanism to the Objective

A linear projection and contrastive learning are not competing categories: the projection is a trainable mechanism, and the contrastive loss is an objective that can train it. The same distinction clarifies two other methods from the broader alignment literature.

Cross-modal attention lets one sequence query another sequence's keys and values:

\operatorname{Attention}(Q_v,K_t,V_t) = \operatorname{softmax}\left(\frac{Q_vK_t^T}{\sqrt{d_k}}\right)V_t

This fuses token-level information conditioned on both inputs. It does not automatically create independently searchable global image and text embeddings. CLIP's image-caption score uses a dual encoder; it is not a cross-attention computation. BLIP illustrates how contrastive matching, image-text matching and language modeling can serve different purposes within one framework.

Adversarial alignment tries to make feature distributions less distinguishable by modality. A discriminator learns to predict the modality, while the feature-producing side receives an opposing objective, such as a reversed gradient. This opposing-gradient pattern is formalized in domain-adversarial learning. Minimizing the discriminator's classification loss alone improves modality discrimination; it does not train the features to hide it. Even successful distribution matching does not guarantee that the dog image matches the dog caption, so paired semantic supervision remains a separate concern.

The architecture decides whether to train encoders, projections, query tokens, cross-attention layers or language-model adapters. There is no universal freeze-both, train-adapters, unfreeze-everything recipe. For the concrete CLIP, BLIP-2, LLaVA and Flamingo components and training stages, continue to vision-language alignment compared.

References

If you found this explanation helpful, consider sharing it with others.

Mastodon