CLS Token in Vision Transformers
Learn how the CLS token acts as a global information aggregator in Vision Transformers, enabling whole-image classification through attention mechanisms.
No direct links0 refs
🤖
Transformers & LLMs
Attention mechanisms, large language models, and multimodal architectures: the building blocks of modern AI.
26
Concepts
All Transformers & LLMs Concepts
Hierarchical Attention in Vision Transformers
Explore how hierarchical attention enables Vision Transformers (ViT) to process sequential data by encoding relative positions.
No direct links0 refs
Multi-Head Attention in Vision Transformers
Explore how multi-head attention enables Vision Transformers (ViT) to process sequential data by encoding relative positions.
No direct links0 refs
Positional Embeddings in Vision Transformers
Explore how positional embeddings enable Vision Transformers (ViT) to process sequential data by encoding relative positions.
No direct links0 refs
Interactive Look: Self-Attention in Vision Transformers
Explore how self-attention enables Vision Transformers (ViT) to understand images by capturing global context, with CNN comparison.
No direct links0 refs
ALiBi: Attention with Linear Biases
Learn ALiBi, the position encoding method that adds linear biases to attention scores for exceptional length extrapolation in transformers.
No direct links0 refs
MHA vs GQA vs MQA: Choosing the Right Attention
Compare Multi-Head, Grouped-Query, and Multi-Query Attention mechanisms to understand their trade-offs and choose the optimal approach for your use case.
No direct links0 refs
Attention Sinks: Stable Streaming LLMs
Learn about attention sinks, where LLMs concentrate attention on initial tokens, and how preserving them enables streaming inference.
No direct links0 refs
Cross-Attention: Bridging Different Modalities
Understand cross-attention, the mechanism that enables transformers to align and fuse information from different sources, sequences, or modalities.
No direct links0 refs
Grouped-Query Attention (GQA)
Learn how Grouped-Query Attention (GQA) balances Multi-Head quality with Multi-Query efficiency for faster LLM inference.
No direct links0 refs
Linear Attention Approximations
Explore linear complexity attention mechanisms including Performer, Linformer, and other efficient transformers that scale to very long sequences.
No direct links0 refs
Masked and Causal Attention
Learn how masked attention enables autoregressive generation and prevents information leakage in transformers and language models.
No direct links0 refs
Multi-Query Attention (MQA)
Learn Multi-Query Attention (MQA), the optimization that shares keys and values across attention heads for massive memory savings.
No direct links0 refs
Rotary Position Embeddings (RoPE)
Learn Rotary Position Embeddings (RoPE), the elegant position encoding using rotation matrices, powering LLaMA, Mistral, and modern LLMs.
No direct links0 refs
Scaled Dot-Product Attention
Master scaled dot-product attention, the fundamental transformer building block. Learn why scaling is crucial for stable training.
No direct links0 refs
Sliding Window Attention
Sliding Window Attention for long sequences: local context windows enable O(n) complexity, used in Mistral and Longformer models.
No direct links0 refs
Sparse Attention Patterns
Explore sparse attention mechanisms that reduce quadratic complexity to linear or sub-quadratic, enabling efficient processing of long sequences.
No direct links0 refs
The Vision-Language Alignment Problem
How vision-language models align visual and text representations using contrastive learning, cross-modal attention, and CLIP-style training.
No direct links0 refs
Context Windows: The Memory Limits of LLMs
Interactive visualization of LLM context windows - sliding windows, expanding contexts, and attention patterns that define model memory limits.
No direct links0 refs
Flash Attention: IO-Aware Exact Attention
Interactive Flash Attention visualization - the IO-aware algorithm achieving memory-efficient exact attention through tiling and kernel fusion.
No direct links0 refs
KV Cache: The Secret to Fast LLM Inference
Interactive KV cache visualization - how key-value caching in LLM transformers enables fast text generation without quadratic recomputation.
No direct links0 refs
The Modality Gap in Multimodal AI
The modality gap in CLIP and vision-language models: why image and text embeddings occupy separate regions despite contrastive training.
No direct links0 refs
Multimodal Scaling Laws
Discover how multimodal vision-language models like CLIP, ALIGN, and LLaVA scale with data, parameters, and compute following Chinchilla-style power laws.
No direct links0 refs
Tokenization: Converting Text to Numbers
Interactive exploration of tokenization methods in LLMs - BPE, SentencePiece, and WordPiece. Understand how text becomes tokens that models can process.
No direct links0 refs
Vision-Language Adapters: Efficient Fine-tuning
Master LoRA, bottleneck adapters, and prefix tuning for parameter-efficient fine-tuning of vision-language models like LLaVA with minimal compute and memory.
No direct links0 refs
Mixture of Experts (MoE)
Understanding sparse mixture of experts models - architecture, routing mechanisms, load balancing, and efficient scaling strategies for large language models
No direct links0 refs