Skip to main content

Attention Sinks: Stable Streaming LLMs

See how preserving initial attention sinks stabilizes a rolling KV cache, why it costs recent slots, and why stable streaming does not mean infinite recall.

What attention sinks are

An attention sink is a token that receives substantial attention even when its content is not useful for the current prediction. The StreamingLLM researchers observed this behavior at the first few positions in several pretrained language models. Keeping those initial tokens' keys and values helped stabilize inference with a rolling cache. The amount of attention and the number of useful initial tokens depend on the model, layer and head. StreamingLLM, Section 3.1

That solves a specific problem: continuing to process a stream with bounded cache memory. It does not make every past token available to attention or let the model recall arbitrary discarded facts.

This page assumes you know self-attention, the KV cache, and sliding-window attention.

Follow the first eviction

A naive sliding cache keeps only the most recent tokens. Once full, it starts evicting the initial tokens that the model may have learned to attend to. StreamingLLM instead reserves a few entries for initial tokens and rolls the recent portion of the cache.

Compare two policies on the same 20-token stream and eight-entry budget:

  • Naive: retain the eight most recent tokens.
  • Preserve sinks: retain initial IDs 0–3 and the four most recent tokens once the cache is full.

Before eviction, both retain all processed tokens. After eight tokens, processing ID 8 forces a choice: the naive policy evicts ID 0, while the sink policy evicts ID 4. Move the timeline to see retained and discarded IDs at the same point in the stream.

The percentages come from explicitly chosen toy logits, and the PPL curve is a schematic illustration. Neither is a trained model output or a reproduction of the paper's measurements. In real evaluation, perplexity is computed from next-token probabilities for a specified model and dataset.

Why removing initial keys can matter

Attention applies softmax over the retained, causally visible keys. With logits $s_j$ and visible cache entries $C$:

a_j = \frac{e^{s_j}}{\sum_{k \in C} e^{s_k}}, \qquad \sum_{j \in C} a_j = 1

If an initial key previously received substantial attention, removing it changes the normalization denominator. The surviving keys then receive a different distribution of weights, even if their logits stay fixed. That can disrupt behavior learned during training.

Normalization helps explain this redistribution, but softmax alone does not force the first tokens to become sinks. The paper proposes that initial tokens are convenient destinations because they are visible to later queries throughout a causal sequence. Its experiments replacing the first four tokens with line breaks suggest that their position can matter more than their original semantic content. This is an explanation supported by those experiments, not a rule that every model or every query must follow. StreamingLLM, Section 3.1

Keep initial tokens, spend fewer slots on recent content

StreamingLLM retains a few initial KV entries alongside recent entries. Four initial tokens worked well across the paper's tested models; four is a practical choice from those experiments, not a universal constant.

text
Eight-entry teaching cache after processing IDs 0–19:

Naive:          [12, 13, 14, 15, 16, 17, 18, 19]
4 initial + 4:  [ 0,  1,  2,  3, 16, 17, 18, 19]

Both store eight entries. The sink policy has discarded IDs 4–15, including recent IDs 12–15 that the naive cache still retains. That is the tradeoff: preserving four initial entries reduces the recent window from eight to four under the same total budget. At fixed model dimensions, the KV payload stays proportional to this bounded cache size rather than to all tokens processed. StreamingLLM, Section 3.2

The paper reports that preserving initial tokens can restore stable streaming perplexity in its evaluated settings. Removing them is not a universal guarantee of collapse, and the experiments do not justify calling any degradation permanent.

Position handling is part of the implementation

Retained original token IDs and the positions used for attention are different things. StreamingLLM uses positions within its rolling cache when adding positional information. For the eight retained IDs in the example above, the correspondence is:

text
Original token IDs:   [0, 1, 2, 3, 16, 17, 18, 19]
Cache-local positions:[0, 1, 2, 3,  4,  5,  6,  7]

There are eight IDs and eight positions. The mapping changes which distances attention uses; it does not recreate the missing tokens between IDs 3 and 16.

For RoPE, the paper's implementation caches keys before their rotary transformation and applies the rolling cache positions to those keys at each decoding step. You cannot implement that behavior by merely changing an index attached to a key that has already been rotated. Other cache representations require corresponding position handling. For ALiBi, the paper applies a contiguous cache-local distance bias instead of a bias with gaps from the original stream. StreamingLLM, Section 3.2

Stable streaming has bounded recall

A long stream and a long attendable context are separate capabilities. The model can keep processing new tokens while attending only to its retained initial and recent entries. An evicted token's KV is unavailable for direct attention. Later retained states may carry some information forward, but they do not guarantee retrieval of an arbitrary earlier fact.

StreamingLLM explicitly targets stable use of recent content, rather than extending the context window or improving long-context retrieval. If a task needs a discarded fact, that information must be supplied again, retrieved externally, or retained through another suitable mechanism. Preserving attention sinks alone does not provide it. StreamingLLM, Section 2

Further reading

For the distinction between keeping a stream running and retaining a particular fact, see context windows.

If you found this explanation helpful, consider sharing it with others.

Mastodon