ALiBi: Attention with Linear Biases
Follow a query-key distance through ALiBi head slopes, attention logits and normalized weights, then understand the evidence and limits of length extrapolation.
7 min readConcept
Explore machine learning concepts related to Position Encoding. Clear explanations and practical insights.
Follow a query-key distance through ALiBi head slopes, attention logits and normalized weights, then understand the evidence and limits of length extrapolation.
Trace RoPE in radians, see how shared shifts preserve relative-position scores, and distinguish its frequency schedule from long-context guarantees.