Feature Pyramid Networks (FPN) Explained
Learn how Feature Pyramid Networks build multi-scale feature representations through top-down pathways and lateral connections for robust object detection.
10 min readConcept
Tagged with
Deep Learning
Explore machine learning concepts related to Deep Learning. Clear explanations and practical insights.
49
Concepts Found
Concepts Related to Deep Learning
CLS Token in Vision Transformers
Trace how a learned CLS row joins image patches, gathers evidence through self-attention, and becomes the image-level classification readout.
7 min readConcept
Tensor Cores: Mixed Precision & Matrix Acceleration
How NVIDIA Tensor Cores accelerate GEMM: warp-level MMA, precision dials (TF32/FP16/BF16/INT8/FP8), shape alignment cliffs, AMP vs model.half(), and when the speedup actually shows up.
16 min readConcept
PyTorch DataLoader Pipeline
PyTorch DataLoader deep dive — Dataset, Sampler, Workers, Collate internals, num_workers throughput profiling, memory analysis, serialization costs, production patterns (LMDB, WebDataset), and bottleneck diagnosis.
22 min readConcept
Hierarchical Attention in Vision Transformers
Trace how local windows, shifted cross-window exchange, and patch merging turn one high-resolution token grid into a multi-scale vision hierarchy.
10 min readConcept
Multi-Head Attention
How multi-head attention runs scaled dot-product attention in parallel across several representation subspaces to build context-aware token embeddings.
7 min readConcept
Anchor-Based vs Anchor-Free Object Detection
Compare anchor-based vs anchor-free object detection: Faster R-CNN and RetinaNet anchors vs FCOS and CenterNet point-based methods.
12 min readConcept
Representation Collapse in Self-Supervised Learning
Understanding complete, dimensional, and cluster collapse — the failure modes that every self-supervised method must prevent. Learn why collapse happens and how contrastive, asymmetric, regularization, and masking approaches solve it.
10 min readConcept
Positional Embeddings in Vision Transformers
Move the same image patches with and without positional addition to see how learned absolute positions change a Vision Transformer’s attention inputs.
6 min readConcept
Batch Norm vs Layer Norm: When to Use Which
BatchNorm normalizes over the batch and spatial axes; LayerNorm normalizes over the channel and spatial axes for each sample. The choice changes whether your model trains stably with batch=1, depends on batch composition at inference, and behaves consistently across train and eval.
8 min readConcept
Self-Attention in Vision Transformers
Follow one image patch through Q/K/V projection, scaled scores, row-wise softmax, value mixing, and the residual update inside a Vision Transformer.
10 min readConcept
DETR Explained: Object Detection with Transformers
Understanding end-to-end object detection with transformers, from DETR's object queries to bipartite matching and attention-based localization
15 min readConcept
Convolution Operation: The Foundation of CNNs
Interactive guide to convolution in CNNs: visualize sliding windows, kernels, stride, padding, and feature detection with step-by-step demos.
10 min readConcept
ALiBi: Attention with Linear Biases
Follow a query-key distance through ALiBi head slopes, attention logits and normalized weights, then understand the evidence and limits of length extrapolation.
7 min readConcept
Cross-Entropy Loss for Classification
Understand cross-entropy loss for classification: interactive demos of binary and multi-class CE, the -log(p) curve, softmax gradients, and focal loss.
15 min readConcept
Flash Attention vs MHA vs GQA vs MQA: Comparing Attention Mechanisms
Compare head sharing with explicit projection dimensions and KV-cache bytes. Separate exact memory ratios from measured quality and latency, and see why FlashAttention is an independent choice.
7 min readConcept
Dilated Convolutions: Expanding Receptive Fields Efficiently
Understand dilated (atrous) convolutions: how dilation rates expand receptive fields exponentially without extra parameters and how to avoid gridding artifacts.
10 min readConcept
Attention Sinks: Stable Streaming LLMs
See how preserving initial attention sinks stabilizes a rolling KV cache, why it costs recent slots, and why stable streaming does not mean infinite recall.
7 min readConcept
Receptive Field in CNNs
Understand receptive fields in CNNs: how convolutional layers expand their field of view and the gap between theoretical and effective receptive fields.
7 min readConcept
Cross-Attention: Bridging Different Modalities
Trace a target query through source keys, attention weights, and a weighted value mix. Understand cross-attention origins, shapes, masks, and applications.
7 min readConcept
VAE Latent Space: Understanding Variational Autoencoders
Explore VAE latent space in deep learning. Learn variational autoencoder encoding, decoding, interpolation, and the reparameterization trick.
4 min readConcept
Grouped-Query Attention: Head Sharing and KV Cache
Trace how grouped-query attention keeps independent query heads while sharing fewer key/value heads, projections, and compact KV-cache rows during LLM decoding.
11 min readConcept
Contrastive Loss for Representation Learning
Understand contrastive loss for representation learning: interactive demos of InfoNCE, triplet loss, and embedding space clustering with temperature tuning.
8 min readConcept
Linear Attention Approximations
Understand normalized kernel attention, low-rank sequence projections and their trade-offs, and compare them with sparse attention and exact tiled softmax.
7 min readConcept
Dropout Regularization
Understand dropout regularization: how randomly silencing neurons prevents overfitting, the inverted dropout trick, and when to use each dropout variant.
10 min readConcept
Masked and Causal Attention
Trace causal attention from shifted next-token labels through the lower-triangular mask, pre-softmax blocking, parallel training, and incremental decoding.
10 min readConcept
Focal Loss: Focusing on Hard Examples
Learn focal loss for deep learning: down-weight easy examples, focus on hard ones. Interactive demos of gamma, alpha balancing, and RetinaNet.
9 min readConcept
Multi-Query Attention (MQA)
Trace how multi-query attention keeps independent query heads while sharing one key/value head, with exact cache bytes and a shape-correct decoding example.
9 min readConcept
He/Kaiming Initialization
Learn He (Kaiming) initialization for ReLU networks: why ReLU needs special weight initialization, variance flow, and dead neurons explained.
7 min readConcept
Rotary Position Embeddings (RoPE)
Trace RoPE in radians, see how shared shifts preserve relative-position scores, and distinguish its frequency schedule from long-context guarantees.
9 min readConcept
KL Divergence in Machine Learning
Learn KL divergence for machine learning: measure distribution differences in VAEs, knowledge distillation, and variational inference.
7 min readConcept
Scaled Dot-Product Attention
Compare scaled and unscaled attention on the same dot products, then derive the square-root scaling factor and its limits.
6 min readConcept
MSE and MAE Loss Functions
Interactive guide to MSE vs MAE for regression: explore outlier sensitivity, gradient behavior, and Huber loss with visualizations.
8 min readConcept
Sliding Window Attention
Explore causal and symmetric local attention with exact window counts, then distinguish direct keys, multi-layer input reach, and inference KV storage.
8 min readConcept
Xavier/Glorot Initialization
Learn Xavier (Glorot) initialization: how it balances forward signals and backward gradients to enable stable deep network training with tanh and sigmoid.
8 min readConcept
Sparse Attention Patterns
Compose sparse attention patterns, follow one query through masking and softmax, and separate fewer allowed pairs from actual sparse-kernel savings.
6 min readConcept
Adaptive Tiling: Efficient Visual Token Generation
Learn adaptive tiling in vision transformers: dynamically partition images based on visual complexity to reduce token counts while preserving detail.
7 min readConcept
Emergent Abilities in Large Language Models
Explore emergent abilities in large language models: sudden capabilities at scale thresholds, phase transitions, and the mirage debate.
8 min readConcept
Prompt Engineering for LLMs
Master prompt engineering for large language models: from basic composition to Chain-of-Thought, few-shot, and advanced techniques.
7 min readConcept
Prompt Influence Flow Through Transformer Layers
Deep dive into how different prompt components influence model behavior across transformer layers, from surface patterns to abstract reasoning.
6 min readConcept
NVIDIA vs AMD for Deep Learning: CUDA vs ROCm and the Datacenter Accelerators
NVIDIA vs AMD for deep learning compared at both layers: the CUDA vs ROCm software moat, the microarchitecture (warp vs wavefront, SM vs CU, Tensor vs Matrix Cores), and the datacenter accelerators (H100/H200/B200 vs MI300X/MI325X).
10 min readConcept
Neural Scaling Laws Explained
Explore neural scaling laws in deep learning: power law relationships between model size, data, and compute that predict AI performance.
8 min readConcept
Visual Complexity Analysis: Smart Image Processing
Learn visual complexity analysis in deep learning - how neural networks measure entropy, edges, and saliency for adaptive image processing.
8 min readConcept
Gradient Flow in Deep Networks
Learn how gradients propagate through deep neural networks during backpropagation. Understand vanishing and exploding gradient problems.
9 min readConcept
NAdam: Nesterov-Accelerated Adam
Understand the NAdam optimizer that fuses Adam adaptive learning rates with Nesterov look-ahead momentum for faster, smoother convergence in deep learning.
6 min readConcept
Layer Normalization for Transformers
Learn layer normalization for transformers and sequence models: how normalizing across features enables batch-independent training.
7 min readConcept
Internal Covariate Shift
Understand internal covariate shift: why layer input distributions change during training, how it slows convergence, and how batch norm fixes it.
8 min readConcept
Batch Normalization in Deep Learning
Learn batch normalization in deep learning: how normalizing layer inputs accelerates training, improves gradient flow, and acts as regularization.
7 min readConcept
Skip Connections in Neural Networks
Learn how skip connections and residual learning enable training of very deep neural networks. Understand the ResNet revolution with interactive visualizations.
9 min readConcept