Tokenization: Converting Text to Numbers
Trace ranked BPE merges from text to pieces and integer IDs. Compare BPE, WordPiece and Unigram, and separate normalization from the SentencePiece framework.
Related
7 min read · beginner
Attention mechanisms, large language models, and multimodal architectures: the building blocks of modern AI.
Start with tokens, follow an attention calculation, then decide where queries can read.
Starting knowledge: Vectors, matrix multiplication and probability distributions
Select a concept. Explore its connections, or open the full explanation.
Previewing Tokenization.
Explore how text becomes token IDs before following the computations over their embeddings.
Turn text into model inputs
Trace BPE merges and compare tokenization methods, normalization and the token budget.
Read conceptUsed in: connections 1 to 2 of 2.
Build the scaled dot-product calculation, run it across heads, then block future keys for causal prediction.
Use the same attention calculation with queries from one sequence and keys and values from another.
Lines show suggested reading within a branch. Connections offer optional ways to explore; they are not a checklist of prerequisites.
Trace ranked BPE merges from text to pieces and integer IDs. Compare BPE, WordPiece and Unigram, and separate normalization from the SentencePiece framework.
7 min read · beginner
Compare scaled and unscaled attention on the same dot products, then derive the square-root scaling factor and its limits.
6 min read · intermediate
How multi-head attention runs scaled dot-product attention in parallel across several representation subspaces to build context-aware token embeddings.
7 min read · intermediate
Trace causal attention from shifted next-token labels through the lower-triangular mask, pre-softmax blocking, parallel training, and incremental decoding.
10 min read · intermediate
Trace a target query through source keys, attention weights, and a weighted value mix. Understand cross-attention origins, shapes, masks, and applications.
7 min read · intermediate
Follow one image patch through Q/K/V projection, scaled scores, row-wise softmax, value mixing, and the residual update inside a Vision Transformer.
10 min read · beginner
Move the same image patches with and without positional addition to see how learned absolute positions change a Vision Transformer’s attention inputs.
6 min read · intermediate
Trace how a learned CLS row joins image patches, gathers evidence through self-attention, and becomes the image-level classification readout.
7 min read · beginner
Trace how local windows, shifted cross-window exchange, and patch merging turn one high-resolution token grid into a multi-scale vision hierarchy.
10 min read · advanced
See how context capacity, attention masks, and KV caches differ. Move a planted fact through a token sequence to inspect what one query can access.
6 min read · beginner
Trace RoPE in radians, see how shared shifts preserve relative-position scores, and distinguish its frequency schedule from long-context guarantees.
9 min read · intermediate
Follow a query-key distance through ALiBi head slopes, attention logits and normalized weights, then understand the evidence and limits of length extrapolation.
7 min read · intermediate
Explore causal and symmetric local attention with exact window counts, then distinguish direct keys, multi-layer input reach, and inference KV storage.
8 min read · intermediate
Compose sparse attention patterns, follow one query through masking and softmax, and separate fewer allowed pairs from actual sparse-kernel savings.
6 min read · advanced
Follow the same Q, K and V through dense and tiled attention. See online softmax preserve the output while avoiding global score and probability matrices.
10 min read · advanced
Understand normalized kernel attention, low-rank sequence projections and their trade-offs, and compare them with sparse attention and exact tiled softmax.
7 min read · advanced
Trace a cached decode step, separate saved projections from growing attention work, and calculate KV-cache memory with explicit dimensions.
6 min read · intermediate
A small draft model proposes tokens, the target model checks them all in one pass, and a rejection rule keeps the output distribution exactly the target's.
10 min read · intermediate
Trace how multi-query attention keeps independent query heads while sharing one key/value head, with exact cache bytes and a shape-correct decoding example.
9 min read · intermediate
Trace how grouped-query attention keeps independent query heads while sharing fewer key/value heads, projections, and compact KV-cache rows during LLM decoding.
11 min read · intermediate
See how preserving initial attention sinks stabilizes a rolling KV cache, why it costs recent slots, and why stable streaming does not mean infinite recall.
7 min read · advanced
Compare head sharing with explicit projection dimensions and KV-cache bytes. Separate exact memory ratios from measured quality and latency, and see why FlashAttention is an independent choice.
7 min read · intermediate
Trace one image-caption pair through cosine scores, contrastive positives and negatives, a trainable projection, and retrieval evaluation.
8 min read
Measure the centroid gap between normalized image and text embeddings, then check why a smaller gap alone does not guarantee correct retrieval.
6 min read
CLIP, BLIP-2, LLaVA and Flamingo align images with text by different routes. Compare their objectives, bridges, data, and which parts train or stay frozen.
10 min read · intermediate
Trace frozen weights and trainable LoRA matrices, calculate adapter parameter counts, and compare bottleneck adapters, prefix tuning and BitFit for vision-language models.
12 min read
Trace top-k expert routing, normalized output weights and capacity overflow, then compare stored parameters and expert work with explicit compute baselines.
12 min read · advanced
Explore a fixed compute budget with Chinchilla’s text-only loss fit, then see why multimodal scaling needs its own data, objectives, and evaluation metrics.
7 min read · advanced