Hierarchical Attention in Vision Transformers
Trace how local windows, shifted cross-window exchange, and patch merging turn one high-resolution token grid into a multi-scale vision hierarchy.
Explore machine learning concepts related to Optimization. Clear explanations and practical insights.
Trace how local windows, shifted cross-window exchange, and patch merging turn one high-resolution token grid into a multi-scale vision hierarchy.
Master virtual memory and TLB address translation with interactive demos. Learn page tables, page faults, and memory management optimization.
Understand cross-entropy loss for classification: interactive demos of binary and multi-class CE, the -log(p) curve, softmax gradients, and focal loss.
Compare head sharing with explicit projection dimensions and KV-cache bytes. Separate exact memory ratios from measured quality and latency, and see why FlashAttention is an independent choice.
Understand dilated (atrous) convolutions: how dilation rates expand receptive fields without extra parameters and how to avoid gridding artifacts.
Why DataLoader num_workers matters: processes hide load latency behind GPU work, how to find the sweet spot, and the memory/GIL pitfalls that come with the pool.
Master sequential vs strided memory access patterns. Learn how cache efficiency and hardware prefetching affect application performance.
Trace how grouped-query attention keeps independent query heads while sharing fewer key/value heads, projections, and compact KV-cache rows during LLM decoding.
Understand normalized kernel attention, low-rank sequence projections and their trade-offs, and compare them with sparse attention and exact tiled softmax.
Embedding quantization simulator: explore memory-accuracy trade-offs from float32 to int8 and binary representations for retrieval.
Master vector compression techniques from scalar to product quantization. Learn how to reduce memory usage by 10-100× while preserving search quality.
Learn when Python __slots__ reduces memory, how slot storage differs from __dict__, and the caveats for dataclasses and inheritance.
Trace how multi-query attention keeps independent query heads while sharing one key/value head, with exact cache bytes and a shape-correct decoding example.
Trace data, structural and control hazards, understand forwarding and load-use stalls, and connect register dependencies to scheduling.
Replay branch outcomes to compare predictors, then connect local and global history to speculative execution, recovery and profiling.
Interactive guide to MSE vs MAE for regression: explore outlier sensitivity, gradient behavior, and Huber loss with visualizations.
Explore causal and symmetric local attention with exact window counts, then distinguish direct keys, multi-layer input reach, and inference KV storage.
Complete C++ thread safety guide — race conditions with step-through simulation, mutexes, atomics, condition variables, deadlock detection, memory ordering, and Thread Sanitizer walkthrough.
Compose sparse attention patterns, follow one query through masking and softmax, and separate fewer allowed pairs from actual sparse-kernel savings.
Explore an illustrative region-tiling policy: resize tiles, calculate visual tokens, and distinguish analytical savings from measured model quality.
Why ordinary multi-process CUDA time slicing can leave SMs idle, how Volta+ MPS co-schedules separate client contexts, execution-resource caps, and when to choose MPS or MIG.
Amdahl and Gustafson ceilings, strong vs weak scaling, roofline bounds, and hiding all-reduce behind compute — the levers that decide whether more GPUs actually buy science.
Master prompt engineering for large language models: from basic composition to Chain-of-Thought, few-shot, and advanced techniques.
Follow the same Q, K and V through dense and tiled attention. See online softmax preserve the output while avoiding global score and probability matrices.
Trace a cached decode step, separate saved projections from growing attention work, and calculate KV-cache memory with explicit dimensions.
Explore neural scaling laws in deep learning: power law relationships between model size, data, and compute that predict AI performance.
Learn visual complexity analysis in deep learning - how neural networks measure entropy, edges, and saliency for adaptive image processing.
Explore a fixed compute budget with Chinchilla’s text-only loss fit, then see why multimodal scaling needs its own data, objectives, and evaluation metrics.
Learn how gradients propagate through deep neural networks during backpropagation. Understand vanishing and exploding gradient problems.
Understand the NAdam optimizer that fuses Adam adaptive learning rates with Nesterov look-ahead momentum with a reproducible constant-momentum update and interactive optimization examples.
C++ compiler optimization lab notebook — compare optimization levels, inspect compiler rewrites, diagnose auto-vectorization, and build production verification commands.
Understand internal covariate shift: why layer input distributions change during training, how it slows convergence, and how batch norm fixes it.
Structure of Arrays vs Array of Structures as instruments: cache-line fill, SIMD gather vs contiguous load, GPU coalescing, and AoSoA hybrids — when layout is a 10× decision.
A small draft model proposes tokens, the target model checks them all in one pass, and a rejection rule keeps the output distribution exactly the target's.
Learn how Transparent Huge Pages (THP) reduces TLB misses by promoting 4KB to 2MB pages. Understand performance benefits and memory bloat tradeoffs.
See how tail calls reuse stack frames, when compilers can eliminate them, and how CPython 3.14 applies tail calls to C opcode dispatch.