Rotary Position Embeddings (RoPE)

Rotary position embeddings (RoPE) rotate pairs of query and key features according to each token's position. The rotation preserves each pair's length. When a query and key are compared, their position-dependent rotations combine into a relative offset.

RoPE supplies a position signal without learned position parameters. It does not make attention depend on distance alone: the query and key still carry content, and reliable use beyond the training context requires evaluation.

Rotate one feature pair

Press Next once with pair 0 selected: position 1 rotates by one radian, about 57.3°. Switch to a higher pair index to see a slower rotation at the same position. The figure uses a unit vector and an eight-dimensional head to isolate the arithmetic.

Why attention needs a position signal

Without a position signal or an order-dependent mask, self-attention is permutation-equivariant. Reordering input tokens reorders its outputs, without supplying a separate representation of their positions. A causal mask already provides an order-dependent visibility constraint, but a position encoding tells the model about offsets within the visible sequence.

An absolute encoding adds a position vector to the token representation. RoPE instead acts on projected Q/K feature pairs before their dot products are computed. Standard RoPE does not rotate V.

Position times angular frequency

For pair i in a head of even dimension d, with frequency base b:

The angle is already in radians. There is no extra 2π factor. With d = 8 and b = 10,000, the four frequencies are 1, 0.1, 0.01, and 0.001 radians per token.

Pair 0 rotates fastest. Increasing the pair index lowers its frequency, so higher pairs vary more slowly over a long span. Raising the base slows pairs with i > 0; it leaves pair 0's frequency at one radian per token.

For features (x₂ᵢ, x₂ᵢ₊₁) at position m:

Rotation changes direction and preserves magnitude. Equivalently, represent the pair as a complex number and multiply it by e^{im\theta_i}. Both forms implement the same geometry when they use the same pairing convention.

The same gap gives the same position contribution

For fixed content vectors q and k in one pair:

The absolute positions cancel because rotations compose. Shifting both positions by the same amount preserves this score; changing their gap generally changes it.

Move either position below, then compare the three shared shifts. The demonstration uses q = k = [1, 0], so its dot product simplifies to \cos((m-n)\theta_i).

Real queries and keys are learned, context-dependent vectors. Moving a sentence into a different surrounding context can change them. Relative-position geometry alone does not guarantee identical attention patterns for that sentence.

What the frequency mix says about distance

Fast and slow pairs provide different position scales. A position-only average across pairs can show a declining, oscillating envelope, but an individual score can rise, fall or change sign with distance. Content-dependent coefficients and softmax further shape the final attention weights.

Change the base below to inspect an equal-weight average of 64 unit-vector pairs. This is a teaching calculation, not measured model attention or an extrapolation benchmark.

A shape-faithful PyTorch implementation

This example pairs adjacent channels and accepts explicit positions. x has shape [batch, tokens, heads, head_dim]; positions has one entry per token. Explicit positions also keep a decode token at its original offset after prompt prefill.

import torch


def apply_rope(x, positions, base=10000.0):
    """Rotate adjacent feature pairs using their absolute token positions."""
    head_dim = x.shape[-1]
    assert head_dim % 2 == 0
    assert positions.numel() == x.shape[1]

    inv_freq = base ** (
        -torch.arange(0, head_dim, 2, device=x.device).float() / head_dim
    )
    angles = positions.to(device=x.device, dtype=torch.float32)[:, None] * inv_freq
    cos = angles.cos()[None, :, None, :].to(dtype=x.dtype)
    sin = angles.sin()[None, :, None, :].to(dtype=x.dtype)
    pairs = x.reshape(*x.shape[:-1], head_dim // 2, 2)
    first, second = pairs[..., 0], pairs[..., 1]
    return torch.stack(
        (first * cos - second * sin, first * sin + second * cos), dim=-1
    ).flatten(-2)

Apply this function to Q and K before computing attention, with the appropriate position vector for each. For one newly appended decode token at cache offset T, use position [T], not [0]. Store its rotated K in the cache, then reuse earlier rotated keys at their original positions.

Some checkpoints use a split-half channel layout instead of adjacent pairs. Both layouts express two-dimensional rotations, but they are not interchangeable without consistently permuting weights and channels. Match the checkpoint's convention. Production kernels can fuse the operation and cache trigonometric tables.

The RoFormer paper develops the relative-position identity. Meta's reference implementation computes position × inverse frequency directly and applies the complex rotation.

Defined beyond training does not mean trained beyond training

The formula accepts arbitrarily large position indices. A model's training distribution, numerical precision and position-frequency interactions still limit useful extrapolation. RoPE alone is not a guarantee of stable attention, retrieval or generation at an extended context length.

Common extension approaches change different parts of the calculation:

• Position interpolation compresses position indices into a smaller range, usually with adaptation and task evaluation.
• Frequency/base scaling, including NTK-inspired approaches, changes the per-pair schedule. It need not affect every pair equally.
• YaRN and related methods adjust frequency bands and attention scaling using a prescribed scheme; configuration support depends on the implementation and version.

Use the scaling scheme and position convention supported by the specific checkpoint. Evaluate quality at the intended lengths and tasks rather than treating a larger rope_theta or one config field as an automatic context upgrade.

Compare where the position signal enters

RoPE composes with KV caching and FlashAttention. These solve different problems: position representation, reuse of prior tensors, and efficient attention execution.

Compare RoPE with ALiBi and ViT positional embeddings. The surrounding calculation is self-attention; restricting its visible positions is the separate topic of sliding window attention.

Related concepts