Grouped-Query Attention (GQA)
Learn how Grouped-Query Attention balances the quality of Multi-Head Attention with the efficiency of Multi-Query Attention, enabling faster inference in large language models.
No direct links0 refs
ML Concepts - Page 16
Clear explanations of core machine learning concepts, from foundational ideas to advanced techniques. Understand attention mechanisms, transformers, skip connections, and more.
167
Total Concepts
274
Topics
Page 16
of 28
Concepts - Page 16
Linear Attention Approximations
Explore linear complexity attention mechanisms including Performer, Linformer, and other efficient transformers that scale to very long sequences.
No direct links0 refs
Masked and Causal Attention
Learn how masked attention enables autoregressive generation and prevents information leakage in transformers, essential for language models and sequential generation.
No direct links0 refs
Multi-Query Attention (MQA)
Understand Multi-Query Attention, the radical efficiency optimization that shares keys and values across all attention heads, enabling massive memory savings for inference.
No direct links0 refs
Rotary Position Embeddings (RoPE)
Understand Rotary Position Embeddings, the elegant position encoding method that encodes relative positions through rotation matrices, used in LLaMA, GPT-NeoX, and most modern LLMs.
No direct links0 refs
Scaled Dot-Product Attention
Master the fundamental building block of transformers - scaled dot-product attention. Learn why scaling is crucial and how the mechanism enables parallel computation.
No direct links0 refs
Showing 91-96 of 167 articles