CLS Token in Vision Transformers
Trace how a learned CLS row joins image patches, gathers evidence through self-attention, and becomes the image-level classification readout.
Related
7 min read · beginner
Transformers & LLMs
Attention mechanisms, large language models, and multimodal architectures: the building blocks of modern AI.
- LLM
- 28 concepts
All Transformers & LLMs Concepts
Hierarchical Attention in Vision Transformers
Trace how local windows, shifted cross-window exchange, and patch merging turn one high-resolution token grid into a multi-scale vision hierarchy.
Related
10 min read · advanced
Multi-Head Attention
How multi-head attention runs scaled dot-product attention in parallel across several representation subspaces to build context-aware token embeddings.
Related
7 min read · intermediate
Positional Embeddings in Vision Transformers
Explore how positional embeddings enable Vision Transformers (ViT) to process sequential data by encoding relative positions.
Related
6 min read · intermediate
Self-Attention in Vision Transformers
Follow one image patch through Q/K/V projection, scaled scores, row-wise softmax, value mixing, and the residual update inside a Vision Transformer.
Related
10 min read · beginner
ALiBi: Attention with Linear Biases
Learn ALiBi, the position encoding method that adds linear biases to attention scores for exceptional length extrapolation in transformers.
Related
12 min read · intermediate
Flash Attention vs MHA vs GQA vs MQA: Comparing Attention Mechanisms
How Flash Attention, Multi-Head Attention (MHA), Grouped-Query Attention (GQA), and Multi-Query Attention (MQA) compare — algorithm vs architecture, KV-cache memory, quality trade-offs, and how to choose for production transformer inference.
Related
5 min read · intermediate
Attention Sinks: Stable Streaming LLMs
Learn about attention sinks, where LLMs concentrate attention on initial tokens, and how preserving them enables streaming inference.
Related
7 min read · advanced
Cross-Attention: Bridging Different Modalities
Understand cross-attention, the mechanism that enables transformers to align and fuse information from different sources, sequences, or modalities.
Related
7 min read · intermediate
Grouped-Query Attention: Head Sharing and KV Cache
Trace how grouped-query attention keeps independent query heads while sharing fewer key/value heads, projections, and compact KV-cache rows during LLM decoding.
Related
11 min read · intermediate
Linear Attention Approximations
Explore linear complexity attention mechanisms including Performer, Linformer, and other efficient transformers that scale to very long sequences.
Related
7 min read · advanced
Masked and Causal Attention
Trace causal attention from shifted next-token labels through the lower-triangular mask, pre-softmax blocking, parallel training, and incremental decoding.
Related
10 min read · intermediate
Multi-Query Attention (MQA)
Learn Multi-Query Attention (MQA), the optimization that shares keys and values across attention heads for massive memory savings.
Related
6 min read · intermediate
Rotary Position Embeddings (RoPE)
Learn Rotary Position Embeddings (RoPE), the elegant position encoding using rotation matrices, powering LLaMA, Mistral, and modern LLMs.
Related
9 min read · intermediate
Scaled Dot-Product Attention
Master scaled dot-product attention, the fundamental transformer building block. Learn why scaling is crucial for stable training.
Related
6 min read · intermediate
Sliding Window Attention
Sliding Window Attention for long sequences: local context windows enable O(n) complexity, used in Mistral and Longformer models.
Related
7 min read · intermediate
Sparse Attention Patterns
Explore sparse attention mechanisms that reduce quadratic complexity to linear or sub-quadratic, enabling efficient processing of long sequences.
Related
6 min read · advanced
The Vision-Language Alignment Problem
How vision-language models align visual and text representations using contrastive learning, cross-modal attention, and CLIP-style training.
Related
7 min read
Context Windows: The Memory Limits of LLMs
Interactive visualization of LLM context windows - sliding windows, expanding contexts, and attention patterns that define model memory limits.
Related
6 min read · beginner
Flash Attention: IO-Aware Exact Attention
Interactive Flash Attention visualization - the IO-aware algorithm achieving memory-efficient exact attention through tiling and kernel fusion.
Related
8 min read · advanced
KV Cache: The Secret to Fast LLM Inference
Interactive KV cache visualization - how key-value caching in LLM transformers enables fast text generation without quadratic recomputation.
Related
6 min read · intermediate
The Modality Gap in Multimodal AI
The modality gap in CLIP and vision-language models: why image and text embeddings occupy separate regions despite contrastive training.
Related
6 min read
Multimodal Scaling Laws
Discover how multimodal vision-language models like CLIP, ALIGN, and LLaVA scale with data, parameters, and compute following Chinchilla-style power laws.
Related
7 min read · advanced
Tokenization: Converting Text to Numbers
Interactive exploration of tokenization methods in LLMs - BPE, SentencePiece, and WordPiece. Understand how text becomes tokens that models can process.
Related
6 min read · beginner
Vision-Language Adapters: Efficient Fine-tuning
Master LoRA, bottleneck adapters, and prefix tuning for parameter-efficient fine-tuning of vision-language models like LLaVA with minimal compute and memory.
Related
10 min read
Mixture of Experts (MoE)
Understanding sparse mixture of experts models - architecture, routing mechanisms, load balancing, and efficient scaling strategies for large language models
Related
12 min read · advanced
Speculative Decoding: Draft, Then Verify
A small draft model proposes tokens, the target model checks them all in one pass, and a rejection rule keeps the output distribution exactly the target's.
Related
10 min read · intermediate
Vision-Language Alignment: CLIP, BLIP-2, LLaVA, Flamingo
CLIP, BLIP-2, LLaVA and Flamingo align images with text by different routes. Compare their objectives, bridges, data, and which parts train or stay frozen.
Related
10 min read · intermediate