CLS Token in Vision Transformers
Learn how the CLS token acts as a global information aggregator in Vision Transformers, enabling whole-image classification through attention mechanisms.
No direct links0 refs
🧠
Attention Mechanisms
Understanding attention mechanisms in neural networks, from basic concepts to advanced implementations.
17
Concepts
All Attention Mechanisms Concepts
Hierarchical Attention in Vision Transformers
Explore how hierarchical attention enables Vision Transformers (ViT) to process sequential data by encoding relative positions.
No direct links0 refs
Multi-Head Attention in Vision Transformers
Explore how multi-head attention enables Vision Transformers (ViT) to process sequential data by encoding relative positions.
No direct links0 refs
Positional Embeddings in Vision Transformers
Explore how positional embeddings enable Vision Transformers (ViT) to process sequential data by encoding relative positions.
No direct links0 refs
Interactive Look: Self-Attention in Vision Transformers
Explore how self-attention enables Vision Transformers (ViT) to understand images by capturing global context, with CNN comparison.
No direct links0 refs
ALiBi: Attention with Linear Biases
Learn ALiBi, the position encoding method that adds linear biases to attention scores for exceptional length extrapolation in transformers.
No direct links0 refs
MHA vs GQA vs MQA: Choosing the Right Attention
Compare Multi-Head, Grouped-Query, and Multi-Query Attention mechanisms to understand their trade-offs and choose the optimal approach for your use case.
No direct links0 refs
Attention Sinks: Stable Streaming LLMs
Learn about attention sinks, where LLMs concentrate attention on initial tokens, and how preserving them enables streaming inference.
No direct links0 refs
Cross-Attention: Bridging Different Modalities
Understand cross-attention, the mechanism that enables transformers to align and fuse information from different sources, sequences, or modalities.
No direct links0 refs
Grouped-Query Attention (GQA)
Learn how Grouped-Query Attention (GQA) balances Multi-Head quality with Multi-Query efficiency for faster LLM inference.
No direct links0 refs
Linear Attention Approximations
Explore linear complexity attention mechanisms including Performer, Linformer, and other efficient transformers that scale to very long sequences.
No direct links0 refs
Masked and Causal Attention
Learn how masked attention enables autoregressive generation and prevents information leakage in transformers and language models.
No direct links0 refs
Multi-Query Attention (MQA)
Learn Multi-Query Attention (MQA), the optimization that shares keys and values across attention heads for massive memory savings.
No direct links0 refs
Rotary Position Embeddings (RoPE)
Learn Rotary Position Embeddings (RoPE), the elegant position encoding using rotation matrices, powering LLaMA, Mistral, and modern LLMs.
No direct links0 refs
Scaled Dot-Product Attention
Master scaled dot-product attention, the fundamental transformer building block. Learn why scaling is crucial for stable training.
No direct links0 refs
Sliding Window Attention
Sliding Window Attention for long sequences: local context windows enable O(n) complexity, used in Mistral and Longformer models.
No direct links0 refs
Sparse Attention Patterns
Explore sparse attention mechanisms that reduce quadratic complexity to linear or sub-quadratic, enabling efficient processing of long sequences.
No direct links0 refs