Masked and Causal Attention
Learn how masked attention enables autoregressive generation and prevents information leakage in transformers and language models.
No direct links0 refs
ML Concepts - Page 23
Clear explanations of core machine learning concepts, from foundational ideas to advanced techniques. Understand attention mechanisms, transformers, skip connections, and more.
190
Total Concepts
358
Topics
Page 23
of 32
Concepts - Page 23
Multi-Query Attention (MQA)
Learn Multi-Query Attention (MQA), the optimization that shares keys and values across attention heads for massive memory savings.
No direct links0 refs
Rotary Position Embeddings (RoPE)
Learn Rotary Position Embeddings (RoPE), the elegant position encoding using rotation matrices, powering LLaMA, Mistral, and modern LLMs.
No direct links0 refs
Scaled Dot-Product Attention
Master scaled dot-product attention, the fundamental transformer building block. Learn why scaling is crucial for stable training.
No direct links0 refs
Sliding Window Attention
Sliding Window Attention for long sequences: local context windows enable O(n) complexity, used in Mistral and Longformer models.
No direct links0 refs
Sparse Attention Patterns
Explore sparse attention mechanisms that reduce quadratic complexity to linear or sub-quadratic, enabling efficient processing of long sequences.
No direct links0 refs
Showing 133-138 of 190 articles