A model's context capacity is its supported sequence-length budget, measured in tokens. The instructions, conversation, documents, and generated continuation consume positions in that sequence. A larger budget admits more information, but admission alone does not ensure that a model will use the right fact.
Three separate choices determine what is available and what it costs:
- Capacity and input policy: how many token positions are supported, and what the application does when the input exceeds that budget. It might reject the input, truncate it, summarize it, or retrieve selected passages.
- Attention connectivity: which keys a particular query may attend to at a layer. Full causal attention can connect to the retained prefix; local attention connects to a smaller neighborhood.
- Execution and storage: how attention is computed and which previous key/value states are cached. An efficient kernel or cache does not, by itself, enlarge the supported context.
Follow a fact through the context
The final query asks for an access code. Move the planted fact, then change capacity and attention separately. This example uses an explicit input policy: keep the most recent tokens, including the query. That truncation policy is a choice for this illustration, not a consequence of every model's capacity limit.
Try moving the fact to position 0 with capacity 16, then increase capacity to 24. The fact changes from discarded to retained. Switch to Local 4: the same fact remains in the input, but this query cannot attend to it directly at this layer. Move it to position 22 to make it directly eligible again.
The displayed lookup returns the code whenever its slot is eligible. It is a mechanical rule, not a model prediction or a measured retrieval success rate. A real model still has to assign useful attention weights, preserve the content, and generate the answer.
Capacity is not the attention window
In the example, the query is at position 23. With capacity 24, full causal attention permits keys 0 through 23. A local window of four positions permits keys 20 through 23, including the query itself:
Here s is the start of the retained input, q is the query position, and w counts the eligible positions, including the query. Local window conventions differ across implementations, so the definition matters.
This is direct connectivity at one layer. In a stacked transformer, an intermediate hidden state may already contain information from earlier positions. The Mistral 7B paper describes how this expands the receptive field across layers. A missing direct edge therefore does not prove that a fact has no influence on the final answer.
Why a larger context costs more
Dense prefill: pairwise work and score storage
Prefill processes the input sequence. For one head with sequence length n and head dimension d_h, computing all query-key scores requires work proportional to n^2 d_h. Multiplying the weights by values has the same order of work. A causal implementation can skip future pairs, but the growth remains quadratic.
If an implementation materializes a dense score matrix in float32, its storage for one head, one batch item, one layer is:
At 100,000 tokens, that is 10 billion elements: 40 GB in decimal units, about 37.3 GiB, for that single matrix. This is not total model memory. A separately stored probability matrix would add another matrix of the same size; weights, Q/K/V tensors, other heads, and training activations have their own costs. A dense causal mask does not automatically eliminate storage for the masked cells.
FlashAttention computes exact attention in tiles without writing the full attention matrix to GPU memory. Its attention storage scales linearly with sequence length for fixed dimensions, while full-attention pairwise arithmetic still scales quadratically. See the FlashAttention concept for the calculation.
Decode: a growing KV cache
During cached decoding, one new query attends to previous keys instead of rebuilding every old query. Full attention then scores roughly n keys per head for that step. The KV cache stores past keys and values, with payload:
Here B is batch size, L is layer count, H_{\mathrm{KV}} is KV head count, and b is bytes per element. This cache grows linearly in retained token count. It is a different memory object from prefill's score matrix. A local rolling cache can bound retained KV states, but evicting a state also removes that direct key/value access.
Using a long context effectively
Capacity, access, and answer quality are separate measurements. In Lost in the Middle, the authors moved relevant information within multi-document question-answering and key-value retrieval inputs. Several evaluated models performed worse when that information was in the middle than near the beginning or end. This is a result for those models and tasks, not a universal attention pattern or proof that every middle token is ignored.
A planted-fact probe tests this distinction by holding the question and answer fixed, varying fact position and input length, and measuring actual generated answers. To evaluate a model, run it: report the model version, prompt format, token counts, positions, and success criterion. Single-fact retrieval also does not establish performance on summarization or reasoning across several documents. The figure above only explains eligibility.
Extending or working around the budget
These techniques address different constraints:
- Reduce attention cost: sliding-window attention and sparse attention change connectivity. A fixed local width reduces attention work toward O(n w d_h) per head. FlashAttention preserves full connectivity while changing execution and storage.
- Support longer positions: RoPE, position scaling, and ALiBi address positional representation. A supported longer context still requires model-specific configuration, training choices, and evaluation; changing a position formula does not guarantee reliable use of arbitrary lengths.
- Keep a bounded stream stable: attention sinks retain initial and recent KV states. The StreamingLLM paper explicitly separates stable streaming from extending the attention window or improving long-history recall. Evicted content is not an unlimited memory.
- Select information before inference: retrieval-augmented generation (RAG) can fetch relevant passages from an external store and place them inside the input budget. The model can then use the retrieved text, while retrieval quality determines what arrives.
The relevant question is whether the needed information fits, remains accessible under the attention pattern, and is used successfully on the task. A context-length number answers only the first part.
Related concepts
Deep dive into how different prompt components influence model behavior across transformer layers, from surface patterns to abstract reasoning.
Follow a query-key distance through ALiBi head slopes, attention logits and normalized weights, then understand the evidence and limits of length extrapolation.
Compare head sharing with explicit projection dimensions and KV-cache bytes. Separate exact memory ratios from measured quality and latency, and see why FlashAttention is an independent choice.
See how preserving initial attention sinks stabilizes a rolling KV cache, why it costs recent slots, and why stable streaming does not mean infinite recall.
Trace a target query through source keys, attention weights, and a weighted value mix. Understand cross-attention origins, shapes, masks, and applications.
Follow the same Q, K and V through dense and tiled attention. See online softmax preserve the output while avoiding global score and probability matrices.