ALiBi: Attention with Linear Biases

Attention with Linear Biases (ALiBi) adds a head-specific distance penalty to attention logits before softmax. It supplies position information without a learned position embedding table. The query, key and value projections remain learned.

The penalty favors nearby keys when content scores are equal. A distant key with a strong content score can still receive more attention than a nearby one. Start with one query and keep that distinction visible through the calculation.

Follow one query-key pair

The query starts at position 5, “mat”, and the selected key is position 0, “The”. Their distance is five tokens. Head 0 of a four-head example has slope 0.25, so the distance bias is −1.25. Adding that to the illustrative content logit of 2 gives a biased logit of 0.75.

Change the selected key to compare distances, or change the head to compare slopes. Change the query to see the causal mask exclude future keys. The content logits stay fixed so you can isolate the effect of the bias and the set of allowed keys; these are teaching values, not outputs from a trained model.

The selected pair alone does not determine its attention weight. Softmax also needs every other allowed key's biased logit. In the opening example, key 0 receives about 20.0% with ALiBi, compared with 35.9% without the distance bias. Those percentages use the same six allowed keys in both columns.

From distance to a normalized weight

For query position i, key position j and head h, first compute the scaled content score:

ALiBi then subtracts a linear distance penalty. In causal self-attention, the allowed keys satisfy j \le i, so their distance is i-j:

Future keys get s_{ij}^{(h)} = -\infty from the causal mask, regardless of their content score or distance bias. Finally, normalize over all allowed keys:

The weights sum to one for each query and head. They mix the value vectors exactly as in ordinary attention; ALiBi changes the logits used to form those weights.

For two allowed keys, the softmax denominator cancels in their ratio:

With equal content scores, each extra token of distance multiplies the relative weight by e^{-m_h}. With unequal scores, content and distance compete. That is why the explorer's weights do not necessarily decrease in order of distance.

Different heads, different distance penalties

The slopes are fixed at initialization. For a power-of-two head count H, the reference schedule is, with heads indexed from zero:

The ratio between consecutive slopes is 2^{-8/H}. It is one half for eight heads, giving 0.5, 0.25, …, 0.00390625. For four heads the ratio is one quarter, giving 0.25, 0.0625, 0.015625 and 0.00390625. Larger slopes impose a stronger preference for recent keys; smaller slopes leave distant keys less penalized. Neither slope creates a hard attention cutoff.

For non-power-of-two head counts, the official slope generator preserves the schedule for the largest smaller power of two, then appends alternating entries from the schedule with twice that many heads. Six heads therefore use:

head:   0       1        2          3           4      5
slope:  0.25    0.0625   0.015625   0.00390625  0.5    0.125

Select six heads in the explorer to inspect that ordering. It is not a new geometric progression over six entries, and head 4 has a steeper slope than head 0.

The mask and the bias have different jobs

The causal mask prevents access to future tokens. ALiBi expresses a soft preference among the remaining tokens. Its self-distance bias is zero, but the current token's final weight still depends on all other allowed logits.

The unmasked absolute-distance matrix is symmetric. A causally masked attention matrix is not: future entries are excluded. You cannot reuse its lower triangle as an equivalent upper triangle.

There is a useful implementation simplification for causal attention. Since -m_h(i-j)=m_hj-m_hi, the term -m_hi is constant across one query row. Softmax is unchanged by adding or removing a row-wide constant. The reference implementation uses this property to construct a key-position bias and combine it with the causal mask. This is row-shift invariance, not symmetry of causal attention.

The following full-sequence example makes both jobs explicit. It assumes unpadded Q, K and V tensors of shape [batch, heads, tokens, head_dim], with the head slopes already chosen:

# src/pages/concepts/transformers/alibi/index.mdx
import torch


def causal_alibi_attention(q, k, v, slopes):
    """Compute full-sequence causal ALiBi attention for unpadded Q/K/V."""
    scores = (q @ k.transpose(-2, -1)) / q.size(-1) ** 0.5
    positions = torch.arange(q.size(-2), device=q.device)
    distance = positions[:, None] - positions[None, :]
    bias = -slopes[None, :, None, None] * distance
    logits = (scores + bias).masked_fill(distance < 0, float("-inf"))
    weights = torch.softmax(logits, dim=-1)
    return weights @ v

This materializes the logits to show the math. A production attention kernel may generate biases inside its tiles instead. With a KV cache, query and key positions must refer to their actual sequence positions, rather than treating the newest query as position zero. Padding masks, dropout and kernel support also need to follow the model's attention implementation.

What length extrapolation means

The distance rule remains defined when inference uses positions beyond the training sequence length. For example, a slope of 0.00390625 gives a bias of −3.99609375 at distance 1023 and −7.99609375 at distance 2047. No new position-table entries are needed to compute either value.

That arithmetic property makes length extrapolation possible; it does not establish model quality at every longer length. Longer contexts change the content, the number of competing keys and the distances seen by the model. Dense attention still has quadratic computation in sequence length, and a KV cache still grows with the stored tokens.

The ALiBi paper, Train Short, Test Long, reports a specific result: a 1.3-billion-parameter language model trained on sequences of length 1024 and evaluated at 2048 matched the perplexity of a sinusoidal-position baseline trained at length 2048. In that comparison, the shorter ALiBi training used 11% less memory and ran 11% faster.

Those are measured results for the paper's experiment. They are not a universal 11% kernel speedup, a guarantee at 16× the training length, or a ranking of ALiBi against every RoPE-based model. The authors' repository provides experiment instructions and checkpoints, including WikiText-103 evaluations with explicit context-window settings.

For an existing checkpoint, keep the position method and slope schedule it was trained with unless you have a tested adaptation procedure. For a new model, evaluate the desired lengths and tasks using the same inference and attention implementation you intend to deploy. ALiBi adds no learned position parameters, while its learned attention projections and the practical costs of longer sequences remain.

Related concepts