He/Kaiming Initialization: Optimizing for ReLU Networks
Before a neural network learns anything, its weights must be set to some initial values. This choice matters enormously — the wrong initialization can kill a network before training begins. He initialization (also called Kaiming initialization) solves a specific problem: how to set weights when your network uses ReLU activations, which behave fundamentally differently from older activations like tanh.
The core insight is simple. ReLU zeroes negative inputs. For a symmetric zero-mean input, this halves its second moment (mean square), which is different from centered variance. If you don't compensate for this halving, signals vanish exponentially as they travel deeper. He initialization doubles the weight variance to cancel out ReLU's halving — balancing the expected signal scale at initialization under independence assumptions.
The Signal Amplifier Analogy
Think of a deep network as a chain of amplifiers connected in series. Each amplifier boosts the signal, then passes it through a filter (ReLU) that removes the bottom half of the waveform. If you set each amplifier's gain for a normal full-waveform signal (Xavier), the chain bleeds power at every stage. He initialization sets the gain to compensate for the filter — doubling the power to offset the 50% that ReLU removes.
After 6 stages with gain ×1 and a half-power filter, this scalar mean-square signal shrinks to 1/64. It illustrates forward attenuation; gradients require a separate chain-rule calculation.
With gain set too high (×4), the signal doubles at every stage. By the end it has exploded — this is the exploding gradient problem. This shows growth in the chosen scalar model. It does not simulate numerical overflow, noise, or training divergence.
Input ready.
The ReLU Problem
Why ReLU Breaks Standard Initialization
The base Xavier derivation uses a linear, zero-centered approximation. Tanh can approximate it near zero, whereas sigmoid is not zero-centered. The base weight variance is:
But ReLU is not symmetric. It passes all positive values unchanged and zeroes all negative values:
For a symmetric zero-mean pre-activation z, ReLU halves the second moment:
It does not halve the centered variance. For z\sim\mathcal{N}(0,\sigma^2), the output has positive mean \sigma/\sqrt{2\pi}, giving:
He's Solution: Preserve the Next Pre-activation Scale
With independent zero-mean weights, zero biases, and the usual independence approximation for incoming activations, z_l=W_la_{l-1} satisfies:
Choosing \operatorname{Var}(W_l)=2/n_{in} makes this recurrence's multiplier one. Xavier with equal fan-in and fan-out instead gives a multiplier of one half for successive ReLU layers. This is an initialization-scale calculation, not a guarantee of stable gradients, successful optimization, or constant post-ReLU variance in every finite network. He et al., Section 2.2 make the second-moment distinction explicitly.
Variance Flow Explorer
Compare pre-activation variance, post-activation centered variance, and mean square. Step through the layers to see which quantity He preserves in this Gaussian model; choose tanh to see bounded outputs even when the input scale grows.
Follow signal moments through depth
Choose a layer and distinguish the centered variance from the mean square that drives the next layer.
He · layer 0
- Pre variance
- 1.000
- Post variance
- 1.000
- Post mean
- 0.000
- Post E[a²]
- 1.000
ReLU has a positive mean. Its mean square is half the symmetric input variance; its centered variance is smaller. He compensates the mean-square factor, which stabilizes the next pre-activation variance in this model.
LeCun initialization uses Var(W) = 1/n_in, designed for self-normalizing networks with SELU activation. For standard ReLU networks, it underestimates the needed variance.
The tiny setting uses standard deviation 0.01 (variance 0.0001). It produces very small activations in this plain stack; small nonzero values are not the same as dead neurons.
He Initialization Variants
He Normal
Draw weights from a normal distribution centered at zero (the second parameter below is variance):
Apply it explicitly, for example nn.init.kaiming_normal_(layer.weight, mode='fan_in', nonlinearity='relu'). Appending ReLU does not change a module's constructor initialization. PyTorch documents nn.Linear with a uniform distribution on [-1/sqrt(fan_in), 1/sqrt(fan_in)], which has a different variance.
He Uniform
Draw weights from a uniform distribution with matched variance:
The bounds are derived from matching the variance of a uniform distribution to the He normal variance. Some practitioners prefer uniform for its bounded range.
Generalized Gain
Different ReLU variants preserve different fractions of the input second moment. A negative slope α retains (1+α²)/2 of a symmetric input second moment; its compensating gain accounts for that fraction. He initialization uses a gain factor:
| Activation | Gain | Reasoning |
|---|---|---|
| ReLU | \sqrt{2} | Halves a symmetric input second moment |
| Leaky ReLU (\alpha) | \sqrt{\frac{2}{1 + \alpha^2}} | Negative slope retains more second moment |
| ELU | 1.0 | Smooth negative region preserves more signal |
| SELU | 0.75 | Self-normalizing property reduces needed gain |
The Dead Neuron Problem
A ReLU is inactive for an input when its pre-activation is nonpositive. That is ordinary gating, not proof that a unit is permanently dead. A unit inactive on every example in a finite input set receives no local ReLU gradient on those examples, but another input or changes to earlier layers may activate it.
The grid below propagates 32 deterministic input vectors through independent dense weight matrices. Each neuron receives the full preceding activation vector. Red means zero output on all 32 examples, amber means nonzero but small RMS output, and teal means larger RMS output. The 0.1 cutoff is only a display threshold. Changing a positive initialization scale changes magnitudes without necessarily changing which units are zero.
All-zero weights prevent hidden units from learning distinct representations through ordinary symmetric updates. The zero setting makes this extreme case visible; it should not be confused with a temporary zero for one input.
Deterministic sampled illustration: 32 input vectors pass through shared independent 200×200 weight matrices with zero biases. Color uses each unit’s RMS output across these inputs. A red unit is inactive on this finite sample, not proven permanently dead; other inputs or upstream parameter updates can activate it. Tiny positive outputs are distinct from exact zeros.
He vs Xavier: Head-to-Head
The difference between He and Xavier becomes dramatic in deep ReLU networks. Watch both methods propagate a signal through 20 layers — He preserves pre-activation scale in the mean-field recurrence while Xavier shrinks it.
With tanh activation, the gap narrows significantly. Both output variances remain bounded by 1. Xavier is a compromise derived near the linear regime; neither choice exactly preserves tanh variance at every layer.
He and Xavier: track the moments
Choose a layer and distinguish the centered variance from the mean square that drives the next layer.
He · layer 0
- Pre variance
- 1.000
- Post variance
- 1.000
- Post mean
- 0.000
- Post E[a²]
- 1.000
Xavier · layer 0
- Pre variance
- 1.000
- Post variance
- 1.000
- Post mean
- 0.000
- Post E[a²]
- 1.000
ReLU has a positive mean. Its mean square is half the symmetric input variance; its centered variance is smaller. He compensates the mean-square factor, which stabilizes the next pre-activation variance in this model.
Choosing Your Initialization
Each method is designed for specific activation functions and network architectures. Compare their formulas, activation pairing, and depth handling.
He Normal
N(0, √(2/n_in))
- Best for
- ReLU, Leaky ReLU
- Assumes symmetric?
- No
- Depth handling
- excellent
- Conv-friendly?
- Yes
Default for modern CNNs and MLPs with ReLU activations
He Uniform
U[-√(6/n_in), √(6/n_in)]
- Best for
- ReLU, ELU
- Assumes symmetric?
- No
- Depth handling
- excellent
- Conv-friendly?
- Yes
Alternative to He Normal with bounded weights
Xavier Normal
N(0, √(2/(n_in+n_out)))
- Best for
- Tanh, Sigmoid
- Assumes symmetric?
- Yes
- Depth handling
- good
- Conv-friendly?
- Yes
Networks with symmetric activations like tanh
Xavier Uniform
U[-√(6/(n_in+n_out)), √(6/(n_in+n_out))]
- Best for
- Tanh, Sigmoid
- Assumes symmetric?
- Yes
- Depth handling
- good
- Conv-friendly?
- Yes
Bounded variant of Xavier for symmetric activations
LeCun Normal
N(0, √(1/n_in))
- Best for
- SELU
- Assumes symmetric?
- No
- Depth handling
- moderate
- Conv-friendly?
- No
Self-normalizing networks with SELU activation
Orthogonal
Q from QR decomposition
- Best for
- RNNs, LSTMs
- Assumes symmetric?
- No
- Depth handling
- excellent
- Conv-friendly?
- No
Recurrent networks where gradient preservation is critical
Use He initialization when:
- Your network uses ReLU, Leaky ReLU, ELU, or PReLU
- You are building deep CNNs or MLPs
- You need stable gradients through many layers
Use Xavier instead when:
- Your network uses tanh or sigmoid activations
- You are using transformer architectures with GELU
- The activation is symmetric around zero
Fan-in vs Fan-out
He initialization has two modes that control different aspects of signal flow:
Fan-in mode preserves the variance of activations during the forward pass. This is the default and the right choice for most networks — it ensures signals maintain their strength as they flow from input to output.
Fan-out mode preserves the variance of gradients during the backward pass. This matters for convolutional layers with batch normalization, where gradient flow during backpropagation is the bottleneck. ResNets and similar architectures use mode='fan_out' for this reason.
In practice, the difference between fan-in and fan-out is small for layers where n_{in} \approx n_{out}. It only matters significantly when the layer changes dimensionality dramatically.
Common Pitfalls
1. Wrong Activation Pairing
He initialization is designed for ReLU-family activations. Using it with tanh or sigmoid overestimates the needed variance, causing initial activations to saturate. Xavier is the correct choice for symmetric activations.
2. Ignoring Batch Normalization
Batch normalization re-normalizes activations at every layer, reducing sensitivity to initialization. However, initialization still affects the first few training steps and gradient magnitudes. For networks with BN, use He with mode='fan_out' to prioritize gradient flow.
3. Residual Network Scaling
In residual networks, the skip connection adds the identity to the transformed signal. Without adjustment, variance doubles at every residual block. The fix is to zero-initialize the last batch normalization layer in each residual branch, so the block initially acts as an identity function.
4. Very Deep Networks Without Normalization
For networks deeper than 50 layers without batch normalization, even He initialization may not suffice. Techniques like FIXUP initialization scale weights by d^{-1/4} (where d is depth) to account for the accumulation effect across many layers.
Related concepts
Learn Xavier (Glorot) initialization: how it balances forward signals and backward gradients to enable stable deep network training with tanh and sigmoid.
Learn batch normalization in deep learning: how normalizing layer inputs accelerates training, improves gradient flow, and acts as regularization.
Learn how skip connections and residual learning enable training of very deep neural networks. Understand the ResNet revolution with interactive visualizations.
BatchNorm normalizes over the batch and spatial axes; LayerNorm normalizes over the channel and spatial axes for each sample. The choice changes whether your model trains stably with batch=1, depends on batch composition at inference, and behaves consistently across train and eval.
Understand dropout regularization: how randomly silencing neurons prevents overfitting, the inverted dropout trick, and when to use each dropout variant.
Learn how gradients propagate through deep neural networks during backpropagation. Understand vanishing and exploding gradient problems.
