Multi-query attention (MQA) keeps all the query heads of multi-head attention, but gives them one shared key head and one shared value head. Every query still computes its own attention scores, weights, and output. The smaller K/V projections and KV cache are the source of its storage savings.
MQA is the Hkv = 1 endpoint of grouped-query attention. Moving along this axis changes how many K/V heads exist, while the query-head count stays fixed. It changes the learned architecture, so an MHA checkpoint cannot become MQA just by changing an inference setting.
Trace independent queries through shared K/V
The figure starts with eight queries and one K/V head. Select a query, then follow Score cached K and Mix shared V: the value rows stay shared, but the selected query's weights and output change. Change K/V heads to 2, 4, or 8 to compare GQA and MHA using the same dimensions.
Sharing K/V preserves query-head work
For query head h, the shared tensors are K0 and V0:
Here M is the causal or other attention mask. Two queries can read the same keys and values while producing different softmax rows, because their query vectors differ. Their outputs are concatenated and passed through the output projection:
At one dense decode step with T available positions, there are still Hq × T query-key score pairs. In the figure, all four K/V-head choices require 8 × 4,096 = 32,768 score pairs per layer. Sharing K/V reduces projection widths and cache storage; it does not divide dense attention score or value-mixing work by the head count.
Projection and tensor shapes
For hidden states X: [B, T, D], MQA uses these learned projection shapes, excluding biases:
| Projection | MHA weights | MQA weights | MQA result after reshaping |
|---|---|---|---|
| Query | [D, Hq × d] | [D, Hq × d] | [B, Hq, T, d] |
| Key | [D, Hq × d] | [D, d] | [B, 1, T, d] |
| Value | [D, Hq × d] | [D, d] | [B, 1, T, d] |
| Output | [Hq × d, D] | [Hq × d, D] | [B, T, D] |
The Q/K/V parameter counts are 3 × D × Hq × d for MHA and D × (Hq + 2) × d for MQA. The output projection is unchanged. For the eight-head figure, Q/K/V weights fall from 3,145,728 to 1,310,720; including the output projection, attention weights fall from 4,194,304 to 2,359,296. This does not imply the same reduction for the full transformer, whose feed-forward and other parameters remain.
Count cache bytes with the dtype included
For batch size B, L layers, cached length T, Hkv K/V heads, head dimension d, and b bytes per stored element:
The leading 2 counts both K and V. Relative to MHA with the same dimensions:
Keeping the figure's eight query heads and other assumptions fixed gives:
| Architecture | K/V heads | Raw cache | Fraction of MHA | Score pairs per decode step, per layer |
|---|---|---|---|---|
| MHA | 8 | 512 MiB | 100% | 32,768 |
| GQA-4 | 4 | 256 MiB | 50% | 32,768 |
| GQA-2 | 2 | 128 MiB | 25% | 32,768 |
| MQA | 1 | 64 MiB | 12.5% | 32,768 |
A separate 32-query-head example
For B = 1, Hq = 32, L = 40, T = 2,048, d = 128, and b = 2 (fp16 or bf16), the raw cache calculation is:
| Method | Bytes per token across 40 layers | Bytes for 2,048 tokens | Raw cache |
|---|---|---|---|
MHA (Hkv = 32) | 655,360 | 1,342,177,280 | 1,280 MiB |
MQA (Hkv = 1) | 20,480 | 41,943,040 | 40 MiB |
MQA uses 1/32 = 3.125% of that MHA cache, a 96.875% saving. This is a different configuration from the eight-head figure. MiB means 2^20 bytes; the byte counts describe the tensors, excluding allocator padding, page tables, cache blocks, quantization metadata, and runtime bookkeeping.
A smaller cache can make room for more concurrent sequences or longer histories. Actual capacity also depends on model weights and workspace, and throughput depends on scheduling, bandwidth, kernels, and hardware. A 32× cache reduction is not a promise of 32× throughput or any fixed latency improvement.
A shape-correct PyTorch example
This standalone example projects eight independent queries and compact one-head K/V tensors. It uses PyTorch 2.12's scaled dot-product attention API with enable_gqa=True. The small random weights illustrate shapes; they are not a trained model. Positional transforms and the rest of the transformer are omitted.
# examples/multi_query_attention.py
import torch
import torch.nn.functional as F
def project_mqa(x, w_q, w_k, w_v, query_heads, head_dim):
"""Project hidden states to independent Q and one-head K/V tensors."""
batch, tokens, _ = x.shape
q = (x @ w_q).reshape(batch, tokens, query_heads, head_dim)
k = (x @ w_k).reshape(batch, tokens, 1, head_dim)
v = (x @ w_v).reshape(batch, tokens, 1, head_dim)
return q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2)
torch.manual_seed(0)
batch, tokens, model_width, query_heads, head_dim = 1, 6, 1024, 8, 128
w_q = torch.randn(model_width, query_heads * head_dim) / model_width**0.5
w_k = torch.randn(model_width, head_dim) / model_width**0.5
w_v = torch.randn(model_width, head_dim) / model_width**0.5
w_o = torch.randn(query_heads * head_dim, model_width) / model_width**0.5
# Prefill: queries and keys start at the same position, so use a causal mask.
x = torch.randn(batch, tokens, model_width)
q, cached_k, cached_v = project_mqa(
x, w_q, w_k, w_v, query_heads, head_dim
)
prefill = F.scaled_dot_product_attention(
q, cached_k, cached_v, dropout_p=0.0, is_causal=True, enable_gqa=True
)
# Decode: append only the new K/V row, keeping the cache's head axis at one.
next_x = torch.randn(batch, 1, model_width)
next_q, next_k, next_v = project_mqa(
next_x, w_q, w_k, w_v, query_heads, head_dim
)
cached_k = torch.cat((cached_k, next_k), dim=2)
cached_v = torch.cat((cached_v, next_v), dim=2)
# All seven cached positions are valid past/current keys for this one query.
decode = F.scaled_dot_product_attention(
next_q, cached_k, cached_v,
dropout_p=0.0, is_causal=False, enable_gqa=True,
)
output = decode.transpose(1, 2).reshape(batch, 1, query_heads * head_dim) @ w_o
assert prefill.shape == (batch, query_heads, tokens, head_dim)
assert cached_k.shape == cached_v.shape == (batch, 1, tokens + 1, head_dim)
assert output.shape == (batch, 1, model_width)
The single decode query is the newest token. Because its cache contains only valid past and current positions, all keys are causally allowed, so is_causal=False is correct here. PyTorch's is_causal=True uses an upper-left mask for non-square query/key lengths: with one query and seven keys, it would expose only the first key. Chunked decoding, padded batches, or caches containing unused future slots need an explicit position-aware mask.
The persisted cache stays [B, 1, T, d]. A reference implementation may repeat K/V temporarily to make the math visible, and some attention backends may materialize that repeat internally. Do not persist an expanded [B, Hq, T, d] cache. Kernel support and temporary memory vary, so enable_gqa=True alone is not proof of a fused, zero-copy implementation. The example's torch.cat append is for clarity; production caches commonly write into reserved storage.
Quality, adoption, and serving choices
Shazeer's original MQA paper motivates sharing through the bandwidth cost of repeatedly loading K/V during incremental decoding. It reports quality and decoding results for its evaluated models and tasks. Those experiments do not establish one speedup or quality-loss percentage for every architecture and workload.
Model names also need configuration-specific interpretation:
- PaLM uses MQA: its architecture description shares K/V projections while retaining all query heads.
- Falcon-7B uses MQA: the official configuration sets
multi_query = truewith71query heads. - Falcon-40B uses GQA: the official configuration has
128query heads and8K/V heads. It is not the one-K/V-head MQA endpoint.
Sharing reduces the independently learned K/V subspaces, so quality must be evaluated on the intended tasks. The GQA paper describes converting MHA checkpoints by pooling K/V heads and continuing training. Such a conversion changes weight shapes and needs training and evaluation; changing a head-count setting alone does not preserve an existing MHA model's function.
MQA is useful to evaluate when compact decode state matters, such as long cached histories, many concurrent requests, or constrained device memory. Compare MHA, GQA, and MQA using task quality, measured memory, and measured latency under the same workload. FlashAttention is a separate kernel choice that can complement these architectures.
Related concepts
Compare head sharing with explicit projection dimensions and KV-cache bytes. Separate exact memory ratios from measured quality and latency, and see why FlashAttention is an independent choice.
Trace how grouped-query attention keeps independent query heads while sharing fewer key/value heads, projections, and compact KV-cache rows during LLM decoding.
Understand normalized kernel attention, low-rank sequence projections and their trade-offs, and compare them with sparse attention and exact tiled softmax.
Explore causal and symmetric local attention with exact window counts, then distinguish direct keys, multi-layer input reach, and inference KV storage.
Compose sparse attention patterns, follow one query through masking and softmax, and separate fewer allowed pairs from actual sparse-kernel savings.
Follow a query-key distance through ALiBi head slopes, attention logits and normalized weights, then understand the evidence and limits of length extrapolation.
