Skip to main content

Multi-Query Attention (MQA)

Trace how multi-query attention keeps independent query heads while sharing one key/value head, with exact cache bytes and a shape-correct decoding example.

Multi-query attention (MQA) keeps all the query heads of multi-head attention, but gives them one shared key head and one shared value head. Every query still computes its own attention scores, weights, and output. The smaller K/V projections and KV cache are the source of its storage savings.

MQA is the Hkv = 1 endpoint of grouped-query attention. Moving along this axis changes how many K/V heads exist, while the query-head count stays fixed. It changes the learned architecture, so an MHA checkpoint cannot become MQA just by changing an inference setting.

Trace independent queries through shared K/V

The figure starts with eight queries and one K/V head. Select a query, then follow Score cached K and Mix shared V: the value rows stay shared, but the selected query's weights and output change. Change K/V heads to 2, 4, or 8 to compare GQA and MHA using the same dimensions.

Sharing K/V preserves query-head work

For query head h, the shared tensors are K0 and V0:

O_h = \operatorname{softmax}\!\left(\frac{Q_h K_0^\top}{\sqrt{d}} + M\right)V_0

Here M is the causal or other attention mask. Two queries can read the same keys and values while producing different softmax rows, because their query vectors differ. Their outputs are concatenated and passed through the output projection:

O = \operatorname{Concat}(O_0, \ldots, O_{H_q-1})W_O

At one dense decode step with T available positions, there are still Hq × T query-key score pairs. In the figure, all four K/V-head choices require 8 × 4,096 = 32,768 score pairs per layer. Sharing K/V reduces projection widths and cache storage; it does not divide dense attention score or value-mixing work by the head count.

Projection and tensor shapes

For hidden states X: [B, T, D], MQA uses these learned projection shapes, excluding biases:

ProjectionMHA weightsMQA weightsMQA result after reshaping
Query[D, Hq × d][D, Hq × d][B, Hq, T, d]
Key[D, Hq × d][D, d][B, 1, T, d]
Value[D, Hq × d][D, d][B, 1, T, d]
Output[Hq × d, D][Hq × d, D][B, T, D]

The Q/K/V parameter counts are 3 × D × Hq × d for MHA and D × (Hq + 2) × d for MQA. The output projection is unchanged. For the eight-head figure, Q/K/V weights fall from 3,145,728 to 1,310,720; including the output projection, attention weights fall from 4,194,304 to 2,359,296. This does not imply the same reduction for the full transformer, whose feed-forward and other parameters remain.

Count cache bytes with the dtype included

For batch size B, L layers, cached length T, Hkv K/V heads, head dimension d, and b bytes per stored element:

\text{cache bytes} = 2 B L T H_{kv} d b

The leading 2 counts both K and V. Relative to MHA with the same dimensions:

\frac{\text{cache bytes}}{\text{MHA cache bytes}} = \frac{H_{kv}}{H_q}

Keeping the figure's eight query heads and other assumptions fixed gives:

ArchitectureK/V headsRaw cacheFraction of MHAScore pairs per decode step, per layer
MHA8512 MiB100%32,768
GQA-44256 MiB50%32,768
GQA-22128 MiB25%32,768
MQA164 MiB12.5%32,768

A separate 32-query-head example

For B = 1, Hq = 32, L = 40, T = 2,048, d = 128, and b = 2 (fp16 or bf16), the raw cache calculation is:

MethodBytes per token across 40 layersBytes for 2,048 tokensRaw cache
MHA (Hkv = 32)655,3601,342,177,2801,280 MiB
MQA (Hkv = 1)20,48041,943,04040 MiB

MQA uses 1/32 = 3.125% of that MHA cache, a 96.875% saving. This is a different configuration from the eight-head figure. MiB means 2^20 bytes; the byte counts describe the tensors, excluding allocator padding, page tables, cache blocks, quantization metadata, and runtime bookkeeping.

A smaller cache can make room for more concurrent sequences or longer histories. Actual capacity also depends on model weights and workspace, and throughput depends on scheduling, bandwidth, kernels, and hardware. A 32× cache reduction is not a promise of 32× throughput or any fixed latency improvement.

A shape-correct PyTorch example

This standalone example projects eight independent queries and compact one-head K/V tensors. It uses PyTorch 2.12's scaled dot-product attention API with enable_gqa=True. The small random weights illustrate shapes; they are not a trained model. Positional transforms and the rest of the transformer are omitted.

python
# examples/multi_query_attention.py
import torch
import torch.nn.functional as F


def project_mqa(x, w_q, w_k, w_v, query_heads, head_dim):
    """Project hidden states to independent Q and one-head K/V tensors."""
    batch, tokens, _ = x.shape
    q = (x @ w_q).reshape(batch, tokens, query_heads, head_dim)
    k = (x @ w_k).reshape(batch, tokens, 1, head_dim)
    v = (x @ w_v).reshape(batch, tokens, 1, head_dim)
    return q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2)


torch.manual_seed(0)
batch, tokens, model_width, query_heads, head_dim = 1, 6, 1024, 8, 128
w_q = torch.randn(model_width, query_heads * head_dim) / model_width**0.5
w_k = torch.randn(model_width, head_dim) / model_width**0.5
w_v = torch.randn(model_width, head_dim) / model_width**0.5
w_o = torch.randn(query_heads * head_dim, model_width) / model_width**0.5

# Prefill: queries and keys start at the same position, so use a causal mask.
x = torch.randn(batch, tokens, model_width)
q, cached_k, cached_v = project_mqa(
    x, w_q, w_k, w_v, query_heads, head_dim
)
prefill = F.scaled_dot_product_attention(
    q, cached_k, cached_v, dropout_p=0.0, is_causal=True, enable_gqa=True
)

# Decode: append only the new K/V row, keeping the cache's head axis at one.
next_x = torch.randn(batch, 1, model_width)
next_q, next_k, next_v = project_mqa(
    next_x, w_q, w_k, w_v, query_heads, head_dim
)
cached_k = torch.cat((cached_k, next_k), dim=2)
cached_v = torch.cat((cached_v, next_v), dim=2)

# All seven cached positions are valid past/current keys for this one query.
decode = F.scaled_dot_product_attention(
    next_q, cached_k, cached_v,
    dropout_p=0.0, is_causal=False, enable_gqa=True,
)
output = decode.transpose(1, 2).reshape(batch, 1, query_heads * head_dim) @ w_o

assert prefill.shape == (batch, query_heads, tokens, head_dim)
assert cached_k.shape == cached_v.shape == (batch, 1, tokens + 1, head_dim)
assert output.shape == (batch, 1, model_width)

The single decode query is the newest token. Because its cache contains only valid past and current positions, all keys are causally allowed, so is_causal=False is correct here. PyTorch's is_causal=True uses an upper-left mask for non-square query/key lengths: with one query and seven keys, it would expose only the first key. Chunked decoding, padded batches, or caches containing unused future slots need an explicit position-aware mask.

The persisted cache stays [B, 1, T, d]. A reference implementation may repeat K/V temporarily to make the math visible, and some attention backends may materialize that repeat internally. Do not persist an expanded [B, Hq, T, d] cache. Kernel support and temporary memory vary, so enable_gqa=True alone is not proof of a fused, zero-copy implementation. The example's torch.cat append is for clarity; production caches commonly write into reserved storage.

Quality, adoption, and serving choices

Shazeer's original MQA paper motivates sharing through the bandwidth cost of repeatedly loading K/V during incremental decoding. It reports quality and decoding results for its evaluated models and tasks. Those experiments do not establish one speedup or quality-loss percentage for every architecture and workload.

Model names also need configuration-specific interpretation:

Sharing reduces the independently learned K/V subspaces, so quality must be evaluated on the intended tasks. The GQA paper describes converting MHA checkpoints by pooling K/V heads and continuing training. Such a conversion changes weight shapes and needs training and evaluation; changing a head-count setting alone does not preserve an existing MHA model's function.

MQA is useful to evaluate when compact decode state matters, such as long cached histories, many concurrent requests, or constrained device memory. Compare MHA, GQA, and MQA using task quality, measured memory, and measured latency under the same workload. FlashAttention is a separate kernel choice that can complement these architectures.

If you found this explanation helpful, consider sharing it with others.

Mastodon