Sliding Window Attention

Explore causal and symmetric local attention with exact window counts, then distinguish direct keys, multi-layer input reach, and inference KV storage.

Give each query a local set of keys

In full self-attention, a query can compare with every key allowed by the model's causal or bidirectional mask. Sliding window attention adds a distance restriction: the query can only attend directly to nearby positions. It uses the same dot products, softmax, and weighted value mixture within that smaller set.

Three questions keep the mechanism clear: which keys are allowed in one layer, which input positions can influence the query through several layers, and which K/V entries are stored during inference? They describe different things. Start with the first two in a fixed 12-position example.

Move the query, keep the convention fixed

Here, w is the maximum total number of keys, including the query's own position. The default causal window has width 5: query 7 can use keys 3 through 7. After two layers it has paths back to input 0, but those earlier inputs do not become extra direct keys in the second layer. Move the query to an edge, then switch to symmetric mode to see the clipping and direction change.

These are connectivity sets for schematic token slots. No attention weights or retrieval accuracy are being simulated.

Causal and symmetric windows

Use zero-based positions 0,\ldots,n-1. With the page's self-inclusive width convention, the causal set is:

\mathcal{W}_{\mathrm{causal}}(i)=\{j:\max(0,i-w+1)\leq j\leq i\}

It contains at most w-1 earlier keys plus the current key. Future positions are excluded, so this pattern can be used for autoregressive decoding.

For the symmetric example, choose an odd width and define r=(w-1)/2. Its key set is:

\mathcal{W}_{\mathrm{sym}}(i)=\{j:\max(0,i-r)\leq j\leq\min(n-1,i+r)\}

Away from the edges, it contains r keys on either side plus self, still w keys in total. Future keys are allowed; this suits bidirectional processing rather than next-token generation. Both modes have fewer keys at sequence boundaries.

For either key set, the attention output is:

o_i=\operatorname{softmax}\!\left(\frac{q_iK_{\mathcal{W}(i)}^\top}{\sqrt{d_k}}\right)V_{\mathcal{W}(i)}

A mask expresses the same restriction by adding zero to allowed logits and negative infinity to blocked logits before softmax. This makes blocked weights zero. Computing a dense matrix and masking it has the right semantics, but still computes the dense matrix.

Depth creates paths, not extra direct keys

At layer 2, the query reads hidden states from its local keys at layer 1. Those states have already mixed information from their own local neighborhoods. Repeat this composition to find which original inputs have a path to the query.

For L layers with the same contiguous window at every layer, the causal reachable input set is exactly:

\mathcal{R}_{\mathrm{causal}}^{(L)}(i)=\{j:\max(0,i-L(w-1))\leq j\leq i\}

The maximum backward distance before edge clipping is L(w-1), and the unclipped set contains 1+L(w-1) positions including self. For symmetric windows, the maximum distance in either direction is Lr:

\mathcal{R}_{\mathrm{sym}}^{(L)}(i)=\{j:\max(0,i-Lr)\leq j\leq\min(n-1,i+Lr)\}

These sets assume the illustrated stack of local attention layers, with position-wise feed-forward blocks and residual connections. Other connectivity patterns can change them. Even when a path exists, useful information must survive several hidden-state updates. Reachability does not establish reliable long-range recall, and it does not imply that every reachable input's original K/V remains in the cache.

Count pairs before claiming savings

For a single head, evaluating and mixing the allowed pairs takes attention arithmetic of order O(nwd) for key/value widths of order d, versus O(n^2d) for full attention. The bound is linear in n when width and head dimensions stay fixed. Q/K/V projections are separate work.

Boundary clipping makes the exact pair count smaller than nw. For the window widths on this page, all of which satisfy w\leq n:

P_{\mathrm{causal}}=\sum_{i=0}^{n-1}\min(w,i+1)=nw-\frac{w(w-1)}{2}

P_{\mathrm{sym}}=nw-r(r+1),\qquad r=\frac{w-1}{2}

Compare like-for-like baselines: full causal attention has n(n+1)/2 allowed pairs, while full bidirectional attention has n^2. At n=12,w=5, the figure counts 50 versus 78 causal pairs, or 54 versus 144 symmetric pairs. Changing the depth control does not change these one-layer counts.

Turning the smaller pair set into runtime savings requires an implementation that skips blocked work, such as a suitable local sparse kernel. A dense masked implementation can still take quadratic arithmetic and score storage. An implementation that materializes only allowed scores uses O(nw) score storage; a tiled implementation may avoid storing even that matrix. FlashAttention changes attention's memory access and temporary storage, while the local mask changes which pairs are evaluated. Neither pair ratios nor asymptotic bounds are measured speedups.

A rolling cache is a separate inference choice

For causal decoding, a local-only layer can evict old KV cache entries once its future queries cannot attend directly to them. With a suitable rolling cache, per-layer K/V storage stays bounded by the window. Upper-layer cached states can carry indirect influence from older inputs; this is not storage of every old input or a promise of recovery after eviction. Prompt prefill also needs a chunking and kernel strategy, rather than simply discarding the prompt before its local states are computed.

The original Mistral 7B paper reports 32 layers, a window setting of 4096, and a context length of 8192 in Table 1. Its prose writes the local interval as i-W through i, using a backward-distance convention, whereas this illustration defines w as the total keys including self. Its approximate 131K theoretical span describes stacked connectivity; it is not the model's stated training context or evidence of 131K-token recall. The paper also describes a rolling buffer and chunked prefill. Performance depends on the kernel, hardware, and workload tested.

Local windows can be part of a wider pattern

Global tokens: Longformer combines local windows with selected tokens that attend to, and are attended by, all positions. BigBird also combines local, global, and random connections. Global connections change the graph beyond the local-only bounds above.
Dilated windows: selecting spaced positions can widen a layer's span without adding as many keys, as in dilated convolutions. Longformer explores dilation as well as its local and global patterns; the reachable sets must be recomputed for that graph.
Attention sinks: retaining a few initial tokens alongside a recent window can stabilize streaming generation in the studied setting. Those retained positions are not a global retrieval mechanism for all discarded content.

These choices belong to the broader family of sparse attention patterns. The context window page separates input capacity from the connectivity used within it.

Decide which information needs a direct path

Local attention is useful when nearby information matters and a model can learn to carry broader context through its layers. A task that needs an arbitrary distant fact in a single attention hop may need a different pattern, global connections, or a separate retrieval mechanism. More depth creates more possible paths, but choosing a window is still a modeling and implementation trade-off that should be evaluated on the intended task.