Cross-Attention: Bridging Different Modalities

Trace a target query through source keys, attention weights, and a weighted value mix. Understand cross-attention origins, shapes, masks, and applications.

Cross-Attention: Connecting Different Information Sources

A translation decoder has a state for the target position it is working on, but it also needs information from the source sentence. Cross-attention lets that target state query a separate source. The target supplies the query; the source supplies both keys and values.

The sequences do not have to be different modalities. English and French states, object queries and image features, or image features and text embeddings can all play these roles. What makes the operation cross-attention is where Q, K, and V come from.

Interactive Cross-Attention Visualization

Keep the target chat selected and step through its calculation. Inspect a source position to see its dot product, softmax weight, and value contribution. Then change the target to see a different query use the same source keys and values.

The numbers are hand-chosen vectors for one head, not a trained translation model. No mask deliberately includes a padded source slot so you can see why padding needs to be excluded.

What is Cross-Attention?

Let X_t be the target hidden states and X_s the source hidden states. Learned projections create the head's inputs:

Q = X_t W_Q,\qquad K = X_s W_K,\qquad V = X_s W_V

The rest is the same scaled dot-product attention used in self-attention:

O = \operatorname{softmax}_{\text{source}}\!\left(\frac{QK^T}{\sqrt{d_k}} + M\right)V

Here, M is an additive mask: zero for allowed connections and -\infty for excluded connections. Softmax runs across the source positions, separately for each target query.

For one head, with batch size B, target length n_t and source length n_s:

Quantity	Shape	Origin
Q	B \times n_t \times d_k	Target states
K	B \times n_s \times d_k	Source states
V	B \times n_s \times d_v	Source states
Scores and weights	B \times n_t \times n_s	Target-to-source connections
Output O	B \times n_t \times d_v	One source-value mix per target

The score matrix can be rectangular: the target and source lengths need not match. Q and K must share d_k for their dot product; V can have a different value dimension d_v. Source and target hidden widths can also differ if their projections map them to compatible head dimensions.

In self-attention, all three projections use the same sequence. In cross-attention, changing the target state changes Q while a fixed source keeps K and V unchanged.

Why Cross-Attention?

The target and source often serve different purposes. A decoder builds the sequence it will output; an encoder represents the information it must consult. Cross-attention connects them without requiring a one-to-one pairing of their positions.

For translation, several source tokens can contribute to one target state. For captioning or visual question answering, a text state can read image features. For an explicit memory-attention layer, the current state can read a separate memory bank. These are possible architectures for the tasks, rather than requirements for every model that performs them.

How Cross-Attention Works

1. Project the two sources of information

In an autoregressive encoder-decoder model, the encoder produces contextualized source states. The decoder's preceding self-attention produces target states using the allowed target context. Each cross-attention head projects the target into Q and the encoder output into K and V.

The labels “query,” “key,” and “value” describe their computational roles. They are learned vectors, rather than literal questions or dictionary entries.

2. Score and normalize one target row

For a selected target position i and source position j:

s_{ij} = \frac{q_i \cdot k_j}{\sqrt{d_k}},\qquad a_{ij} = \frac{\exp(s_{ij}+M_{ij})}{\sum_{\ell=1}^{n_s}\exp(s_{i\ell}+M_{i\ell})}

Scaling by \sqrt{d_k} keeps the dot-product variance from growing with the head dimension under the usual unit-variance assumptions. It helps avoid overly saturated softmax distributions; it does not guarantee useful gradients for every input.

In the figure, chat has q=[0.5,1] and the source cat has k=[0,2]. Their dot product is 2 and the scaled score is 2/\sqrt{2}. That score competes with the other allowed source scores in the same row.

A large weight means that the head gives that source value a large coefficient in this calculation. It does not by itself prove a semantic match or explain the model's final prediction.

3. Mix the source values

o_i = \sum_{j=1}^{n_s} a_{ij}v_j

The output is a weighted sum of V, not of K and not of the target states. With at least one allowed source position and finite allowed scores, the weights sum to one. A masked padded position contributes exactly zero. A completely masked row has no defined softmax distribution and needs explicit handling in an implementation.

This context vector is one head's output. A full multi-head block combines head outputs, applies an output projection, and uses a residual connection with the target stream. The context vector is not itself a generated word.

Cross-Attention in Transformers

Encoder-Decoder Architecture

In the original Transformer, an autoregressive decoder layer applies:

Masked target self-attention: Q, K, and V all come from target states. Each position can use itself and earlier input positions; shifted decoder inputs prevent access to the next token being predicted.
Encoder-decoder cross-attention: Q comes from the updated target stream. K and V come from the encoder output. The head reads the source information needed by that target state.
Feed-forward network: Processes each target position's representation after the attention sublayer.

Each sublayer also has a residual connection and normalization. Their ordering and normalization placement can vary in later architectures.

code

Source input → Encoder → Source states → K and V
                                            ↓
Target input → Masked self-attention → Q → Cross-attention
                                            ↓
                                  Target representation → FFN

For a fixed source during inference, its encoder states can be computed once. Each decoder layer's projected source K/V can also be cached and reused as target decoding advances. The target Q changes with each new target state. This source cache is separate from the growing target self-attention KV cache.

Source masks and target causality are separate

A complete source sentence is already available in ordinary translation, so a target query can attend to every non-padded source position, including a source position whose index is larger than the target index. There is no automatic triangular target-to-source mask.

Padding still needs a source mask. Streaming speech, restricted memory access, and other tasks may also impose an availability mask. These constraints describe which source positions exist or may be read. The decoder's causal self-attention mask separately controls which target positions can communicate. Neither mask follows simply from calling an operation “cross-attention.”

Types of Cross-Attention

Encoder-decoder and multimodal attention

Translation and summarization can use target text states to read an encoded source document. Captioning and VQA architectures can let text states read image regions. Speech encoder-decoder architectures can let text states read audio representations. The data modalities change, but the projection, source softmax, and value mix remain the same.

Multimodal training alone does not imply cross-attention. Original CLIP encodes images and text independently and compares their embeddings; it does not fuse the two streams with image-to-text cross-attention. Likewise, wav2vec 2.0 uses a speech representation encoder and CTC fine-tuning for recognition, rather than a text decoder querying audio with cross-attention.

Object queries and memory attention

In DETR, decoder states associated with learned object queries attend to encoded image features. The outputs predict a set of objects in parallel, so the target need not be a causally generated text sequence.

A dedicated memory-attention layer can similarly take Q from the current state and K/V from a memory bank. Retrieval augmentation does not always use this design: a model that inserts retrieved text into its ordinary input may process that text with self-attention instead.

Cross-attention in diffusion

In latent diffusion models, intermediate image features supply Q while conditioning text representations supply K and V. This lets the denoising network incorporate text information. Here the “target” is an image-feature stream, not the words being generated by a language decoder.

References

Attention Is All You Need: sections 3.1–3.2 explain decoder cross-attention, projections, and source/target roles.
End-to-End Object Detection with Transformers: section 3.2 explains DETR's object queries and encoder-decoder attention.
High-Resolution Image Synthesis with Latent Diffusion Models: section 3.3 explains cross-attention conditioning.
Learning Transferable Visual Models From Natural Language Supervision: section 2.3 explains CLIP's separate image and text encoders.
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations: sections 2–3 explain the speech encoder and CTC fine-tuning.

For the separate target-side causal restriction, see masked attention.

Transformers & LLMs

ALiBi: Attention with Linear Biases

Follow a query-key distance through ALiBi head slopes, attention logits and normalized weights, then understand the evidence and limits of length extrapolation.

Transformers & LLMs

Flash Attention vs MHA vs GQA vs MQA: Comparing Attention Mechanisms

Compare head sharing with explicit projection dimensions and KV-cache bytes. Separate exact memory ratios from measured quality and latency, and see why FlashAttention is an independent choice.

Transformers & LLMs

Attention Sinks: Stable Streaming LLMs

See how preserving initial attention sinks stabilizes a rolling KV cache, why it costs recent slots, and why stable streaming does not mean infinite recall.

Transformers & LLMs

Grouped-Query Attention: Head Sharing and KV Cache

Trace how grouped-query attention keeps independent query heads while sharing fewer key/value heads, projections, and compact KV-cache rows during LLM decoding.

Transformers & LLMs

Linear Attention Approximations

Understand normalized kernel attention, low-rank sequence projections and their trade-offs, and compare them with sparse attention and exact tiled softmax.

Transformers & LLMs

Masked and Causal Attention

Trace causal attention from shifted next-token labels through the lower-triangular mask, pre-softmax blocking, parallel training, and incremental decoding.