Grouped-Query Attention (GQA)
Learn how Grouped-Query Attention (GQA) balances Multi-Head quality with Multi-Query efficiency for faster LLM inference.
No direct links0 refs
ML Concepts - Page 18
Clear explanations of core machine learning concepts, from foundational ideas to advanced techniques. Understand attention mechanisms, transformers, skip connections, and more.
187
Total Concepts
356
Topics
Page 18
of 32
Concepts - Page 18
Linear Attention Approximations
Explore linear complexity attention mechanisms including Performer, Linformer, and other efficient transformers that scale to very long sequences.
No direct links0 refs
Masked and Causal Attention
Learn how masked attention enables autoregressive generation and prevents information leakage in transformers and language models.
No direct links0 refs
Multi-Query Attention (MQA)
Learn Multi-Query Attention (MQA), the optimization that shares keys and values across attention heads for massive memory savings.
No direct links0 refs
Rotary Position Embeddings (RoPE)
Learn Rotary Position Embeddings (RoPE), the elegant position encoding using rotation matrices, powering LLaMA, Mistral, and modern LLMs.
No direct links0 refs
Scaled Dot-Product Attention
Master scaled dot-product attention, the fundamental transformer building block. Learn why scaling is crucial for stable training.
No direct links0 refs
Showing 103-108 of 187 articles