Gradient Flow in Deep Networks
Learn how gradients propagate through deep neural networks during backpropagation. Understand vanishing and exploding gradient problems.
Related
10 min read
Neural network fundamentals: normalization, convolutions, losses, graph networks, and training dynamics.
Follow gradients, choose initial weights, and keep training stable.
Starting knowledge: Neural-network layers, derivatives and a basic training loop
Select a concept. Explore its connections, or open the full explanation.
Previewing Gradient flow.
Follow the chain rule, then compare how Xavier and He choose an initial scale for different activations.
Signal and initialization
Explain how repeated derivatives can shrink or amplify gradients through a network.
Read conceptBackground: connections 1 to 1 of 1.
Inspect batch and feature statistics, compare their uses, then optionally examine the debate about internal covariate shift.
Use the gradient-flow connection to understand the identity path. The illustrated ResNet block also uses convolutions and BatchNorm.
Use gradients to update parameters, then separate momentum, look-ahead and adaptive step sizes.
Inspect random masking and train/eval behavior as a way to address overfitting.
Choose which pretrained parameters a new task can update, and separate that choice from gradients and normalization state. This branch applies to supervised and self-supervised encoders.
Lines show suggested reading within a branch. Connections offer optional ways to explore; they are not a checklist of prerequisites.
Learn how gradients propagate through deep neural networks during backpropagation. Understand vanishing and exploding gradient problems.
10 min read
Learn Xavier (Glorot) initialization: how it balances forward signals and backward gradients to enable stable deep network training with tanh and sigmoid.
8 min read
Learn He (Kaiming) initialization for ReLU networks: why ReLU needs special weight initialization, variance flow, and dead neurons explained.
9 min read
Learn batch normalization in deep learning: how normalizing layer inputs accelerates training, improves gradient flow, and acts as regularization.
8 min read
Learn layer normalization for transformers and sequence models: how normalizing across features enables batch-independent training.
9 min read
BatchNorm normalizes over the batch and spatial axes; LayerNorm normalizes over the channel and spatial axes for each sample. The choice changes whether your model trains stably with batch=1, depends on batch composition at inference, and behaves consistently across train and eval.
7 min read · intermediate
Understand internal covariate shift: why layer input distributions change during training, how it slows convergence, and how batch norm fixes it.
6 min read
Learn how skip connections and residual learning enable training of very deep neural networks. Understand the ResNet revolution with interactive visualizations.
9 min read
Understand the NAdam optimizer that fuses Adam adaptive learning rates with Nesterov look-ahead momentum with a reproducible constant-momentum update and interactive optimization examples.
6 min read
Understand dropout regularization: how randomly silencing neurons prevents overfitting, the inverted dropout trick, and when to use each dropout variant.
11 min read
Move a pretrained encoder to a downstream task: trace frozen features, head training, partial and full fine-tuning, learning rates, and BatchNorm state.
9 min read · intermediate
Interactive guide to MSE vs MAE for regression: explore outlier sensitivity, gradient behavior, and Huber loss with visualizations.
9 min read
Understand cross-entropy loss for classification: interactive demos of binary and multi-class CE, the -log(p) curve, softmax gradients, and focal loss.
15 min read
Learn focal loss for deep learning: down-weight easy examples, focus on hard ones. Interactive demos of gamma, alpha balancing, and RetinaNet.
12 min read
Learn KL divergence for machine learning: measure distribution differences in VAEs, knowledge distillation, and variational inference.
7 min read
Explore VAE latent space in deep learning. Learn variational autoencoder encoding, decoding, interpolation, and the reparameterization trick.
5 min read
Understand where self-supervised targets come from, how masked prediction creates a learning task, and why predicting pixels differs from predicting learned features.
7 min read · beginner
Follow how a learned target encoder supplies feature targets, how stop-gradient separates backward paths, and how exponential moving averages update its parameters.
6 min read · beginner
Understand contrastive loss for representation learning: interactive demos of InfoNCE, triplet loss, and embedding space clustering with temperature tuning.
8 min read
Understanding complete, dimensional, and cluster collapse — the failure modes that every self-supervised method must prevent. Learn why collapse happens and how contrastive, asymmetric, regularization, and masking approaches solve it.
10 min read · advanced
Interactive guide to convolution in CNNs: visualize sliding windows, kernels, stride, padding, and feature detection with step-by-step demos.
10 min read
Understand receptive fields in CNNs: how convolutional layers expand their field of view and the gap between theoretical and effective receptive fields.
7 min read
Understand dilated (atrous) convolutions: how dilation rates expand receptive fields without extra parameters and how to avoid gridding artifacts.
12 min read
Learn visual complexity analysis in deep learning - how neural networks measure entropy, edges, and saliency for adaptive image processing.
9 min read
Explore an illustrative region-tiling policy: resize tiles, calculate visual tokens, and distinguish analytical savings from measured model quality.
6 min read
Understanding node importance through centrality measures, shortest paths, hop distances, clustering coefficients, and fundamental graph metrics
15 min read
Learning low-dimensional vector representations of graphs through random walks, DeepWalk, Node2Vec, and skip-gram models
12 min read
Learn Graph Convolutional Networks (GCN) with spectral theory, message passing, and node classification for geometric deep learning.
15 min read
Adaptive attention-based aggregation for graph neural networks - multi-head attention, learned weights, and interpretable graph learning
10 min read
Hierarchical graph coarsening techniques - TopK, SAGPool, DiffPool, and readout operations for graph-level representations
12 min read
How the silhouette score measures clustering quality for every individual point — comparing intra-cluster cohesion to nearest-cluster separation, with per-point diagnostics and explicit distance-metric assumptions.
8 min read · intermediate
How the Calinski-Harabasz index evaluates clustering quality by measuring the ratio of between-cluster to within-cluster variance — fast, intuitive, and ideal for k-selection with convex clusters.
8 min read · intermediate
How the Davies-Bouldin index evaluates clustering quality by finding each cluster's most similar neighbor — a pessimistic, worst-case metric that catches overlapping cluster pairs.
8 min read · intermediate
Master prompt engineering for large language models: from basic composition to Chain-of-Thought, few-shot, and advanced techniques.
5 min read
Explore how prompt token positions connect through causal self-attention, and why connectivity and attention weights do not establish causal influence.
5 min read
Explore neural scaling laws in deep learning: power law relationships between model size, data, and compute that predict AI performance.
6 min read
Explore emergent abilities in large language models: sudden capabilities at scale thresholds, phase transitions, and the mirage debate.
5 min read