ALiBi: Attention with Linear Biases
Learn ALiBi, the position encoding method that adds linear biases to attention scores for exceptional length extrapolation in transformers.
19 min readConcept
Tagged with
position-encoding
Explore machine learning concepts related to position-encoding. Clear explanations and practical insights.
2
Concepts Found
Concepts Related to position-encoding
Rotary Position Embeddings (RoPE)
Learn Rotary Position Embeddings (RoPE), the elegant position encoding using rotation matrices, powering LLaMA, Mistral, and modern LLMs.
8 min readConcept