CLS Token in Vision Transformers
Trace how a learned CLS row joins image patches, gathers evidence through self-attention, and becomes the image-level classification readout.
7 min readConcept
Tagged with
Attention
Explore machine learning concepts related to Attention. Clear explanations and practical insights.
21
Concepts Found
Concepts Related to Attention
Hierarchical Attention in Vision Transformers
Trace how local windows, shifted cross-window exchange, and patch merging turn one high-resolution token grid into a multi-scale vision hierarchy.
10 min readConcept
Multi-Head Attention
How multi-head attention runs scaled dot-product attention in parallel across several representation subspaces to build context-aware token embeddings.
7 min readConcept
Positional Embeddings in Vision Transformers
Explore how positional embeddings enable Vision Transformers (ViT) to process sequential data by encoding relative positions.
6 min readConcept
Self-Attention in Vision Transformers
Follow one image patch through Q/K/V projection, scaled scores, row-wise softmax, value mixing, and the residual update inside a Vision Transformer.
10 min readConcept
ALiBi: Attention with Linear Biases
Learn ALiBi, the position encoding method that adds linear biases to attention scores for exceptional length extrapolation in transformers.
12 min readConcept
Flash Attention vs MHA vs GQA vs MQA: Comparing Attention Mechanisms
How Flash Attention, Multi-Head Attention (MHA), Grouped-Query Attention (GQA), and Multi-Query Attention (MQA) compare — algorithm vs architecture, KV-cache memory, quality trade-offs, and how to choose for production transformer inference.
5 min readConcept
Attention Sinks: Stable Streaming LLMs
Learn about attention sinks, where LLMs concentrate attention on initial tokens, and how preserving them enables streaming inference.
7 min readConcept
Cross-Attention: Bridging Different Modalities
Understand cross-attention, the mechanism that enables transformers to align and fuse information from different sources, sequences, or modalities.
7 min readConcept
Grouped-Query Attention: Head Sharing and KV Cache
Trace how grouped-query attention keeps independent query heads while sharing fewer key/value heads, projections, and compact KV-cache rows during LLM decoding.
11 min readConcept
Linear Attention Approximations
Explore linear complexity attention mechanisms including Performer, Linformer, and other efficient transformers that scale to very long sequences.
7 min readConcept
Masked and Causal Attention
Trace causal attention from shifted next-token labels through the lower-triangular mask, pre-softmax blocking, parallel training, and incremental decoding.
10 min readConcept
Multi-Query Attention (MQA)
Learn Multi-Query Attention (MQA), the optimization that shares keys and values across attention heads for massive memory savings.
6 min readConcept
Rotary Position Embeddings (RoPE)
Learn Rotary Position Embeddings (RoPE), the elegant position encoding using rotation matrices, powering LLaMA, Mistral, and modern LLMs.
9 min readConcept
Scaled Dot-Product Attention
Master scaled dot-product attention, the fundamental transformer building block. Learn why scaling is crucial for stable training.
6 min readConcept
Sliding Window Attention
Sliding Window Attention for long sequences: local context windows enable O(n) complexity, used in Mistral and Longformer models.
7 min readConcept
Sparse Attention Patterns
Explore sparse attention mechanisms that reduce quadratic complexity to linear or sub-quadratic, enabling efficient processing of long sequences.
6 min readConcept
Adaptive Tiling: Efficient Visual Token Generation
Learn adaptive tiling in vision transformers: dynamically partition images based on visual complexity to reduce token counts while preserving detail.
7 min readConcept
Context Windows: The Memory Limits of LLMs
Interactive visualization of LLM context windows - sliding windows, expanding contexts, and attention patterns that define model memory limits.
6 min readConcept
Flash Attention: IO-Aware Exact Attention
Interactive Flash Attention visualization - the IO-aware algorithm achieving memory-efficient exact attention through tiling and kernel fusion.
8 min readConcept
Prompt Influence Flow Through Transformer Layers
Deep dive into how different prompt components influence model behavior across transformer layers, from surface patterns to abstract reasoning.
6 min readConcept