Multi-Query Attention (MQA)
Learn Multi-Query Attention (MQA), the optimization that shares keys and values across attention heads for massive memory savings.
No direct links0 refs
ML Concepts - Page 23
Clear explanations of core machine learning concepts, from foundational ideas to advanced techniques. Understand attention mechanisms, transformers, skip connections, and more.
189
Total Concepts
357
Topics
Page 23
of 32
Concepts - Page 23
Rotary Position Embeddings (RoPE)
Learn Rotary Position Embeddings (RoPE), the elegant position encoding using rotation matrices, powering LLaMA, Mistral, and modern LLMs.
No direct links0 refs
Scaled Dot-Product Attention
Master scaled dot-product attention, the fundamental transformer building block. Learn why scaling is crucial for stable training.
No direct links0 refs
Sliding Window Attention
Sliding Window Attention for long sequences: local context windows enable O(n) complexity, used in Mistral and Longformer models.
No direct links0 refs
Sparse Attention Patterns
Explore sparse attention mechanisms that reduce quadratic complexity to linear or sub-quadratic, enabling efficient processing of long sequences.
No direct links0 refs
Understanding NVIDIA Persistence Daemon
Eliminating GPU initialization latency through nvidia-persistenced - a userspace daemon that maintains GPU driver state for optimal startup performance.
No direct links0 refs
Showing 133-138 of 189 articles