Skip to main content

He/Kaiming Initialization

Learn He (Kaiming) initialization for ReLU networks: why ReLU needs special weight initialization, variance flow, and dead neurons explained.

He/Kaiming Initialization: Optimizing for ReLU Networks

Before a neural network learns anything, its weights must be set to some initial values. This choice matters enormously — the wrong initialization can kill a network before training begins. He initialization (also called Kaiming initialization) solves a specific problem: how to set weights when your network uses ReLU activations, which behave fundamentally differently from older activations like tanh.

The core insight is simple. ReLU zeroes negative inputs. For a symmetric zero-mean input, this halves its second moment (mean square), which is different from centered variance. If you don't compensate for this halving, signals vanish exponentially as they travel deeper. He initialization doubles the weight variance to cancel out ReLU's halving — balancing the expected signal scale at initialization under independence assumptions.

The Signal Amplifier Analogy

Think of a deep network as a chain of amplifiers connected in series. Each amplifier boosts the signal, then passes it through a filter (ReLU) that removes the bottom half of the waveform. If you set each amplifier's gain for a normal full-waveform signal (Xavier), the chain bleeds power at every stage. He initialization sets the gain to compensate for the filter — doubling the power to offset the 50% that ReLU removes.

After 6 stages with gain ×1 and a half-power filter, this scalar mean-square signal shrinks to 1/64. It illustrates forward attenuation; gradients require a separate chain-rule calculation.

With gain set too high (×4), the signal doubles at every stage. By the end it has exploded — this is the exploding gradient problem. This shows growth in the chosen scalar model. It does not simulate numerical overflow, noise, or training divergence.

Signal amplifier chain
Gain
Step 1 / 8
Input
1.00
× gain → ½
Stage 1
—
× gain → ½
Stage 2
—
× gain → ½
Stage 3
—
× gain → ½
Stage 4
—
× gain → ½
Stage 5
—
× gain → ½
Stage 6
—

Input ready.

The ReLU Problem

Why ReLU Breaks Standard Initialization

The base Xavier derivation uses a linear, zero-centered approximation. Tanh can approximate it near zero, whereas sigmoid is not zero-centered. The base weight variance is:

\text{Var}(W) = \frac{2}{n_{in} + n_{out}}

But ReLU is not symmetric. It passes all positive values unchanged and zeroes all negative values:

f(x) = \max(0, x)

For a symmetric zero-mean pre-activation z, ReLU halves the second moment:

\mathbb{E}[\operatorname{ReLU}(z)^2] = \frac{1}{2}\mathbb{E}[z^2] = \frac{1}{2}\operatorname{Var}(z)

It does not halve the centered variance. For z\sim\mathcal{N}(0,\sigma^2), the output has positive mean \sigma/\sqrt{2\pi}, giving:

\operatorname{Var}(\operatorname{ReLU}(z)) = \sigma^2\left(\frac12-\frac{1}{2\pi}\right)

He's Solution: Preserve the Next Pre-activation Scale

With independent zero-mean weights, zero biases, and the usual independence approximation for incoming activations, z_l=W_la_{l-1} satisfies:

\operatorname{Var}(z_l)=n_{in}\operatorname{Var}(W_l)\mathbb{E}[a_{l-1}^2]=\frac{n_{in}\operatorname{Var}(W_l)}{2}\operatorname{Var}(z_{l-1})

Choosing \operatorname{Var}(W_l)=2/n_{in} makes this recurrence's multiplier one. Xavier with equal fan-in and fan-out instead gives a multiplier of one half for successive ReLU layers. This is an initialization-scale calculation, not a guarantee of stable gradients, successful optimization, or constant post-ReLU variance in every finite network. He et al., Section 2.2 make the second-moment distinction explicitly.

Variance Flow Explorer

Compare pre-activation variance, post-activation centered variance, and mean square. Step through the layers to see which quantity He preserves in this Gaussian model; choose tanh to see bounded outputs even when the input scale grows.

Follow signal moments through depth

Choose a layer and distinguish the centered variance from the mean square that drives the next layer.

20
Step 1 / 21
0
variance through layer 0; exact values follow

He · layer 0

Pre variance
1.000
Post variance
1.000
Post mean
0.000
Post E[a²]
1.000

ReLU has a positive mean. Its mean square is half the symmetric input variance; its centered variance is smaller. He compensates the mean-square factor, which stabilizes the next pre-activation variance in this model.

Analytically calculated Gaussian mean-field illustration: equal-width layers (256 units), independent zero-mean weights, zero biases. ReLU moments are exact under these assumptions; tanh moments use deterministic numerical integration. These are initialization calculations, not measured training results. Finite networks need not follow them exactly.

LeCun initialization uses Var(W) = 1/n_in, designed for self-normalizing networks with SELU activation. For standard ReLU networks, it underestimates the needed variance.

The tiny setting uses standard deviation 0.01 (variance 0.0001). It produces very small activations in this plain stack; small nonzero values are not the same as dead neurons.

He Initialization Variants

He Normal

Draw weights from a normal distribution centered at zero (the second parameter below is variance):

W \sim \mathcal{N}\left(0, \frac{2}{n_{in}}\right)

Apply it explicitly, for example nn.init.kaiming_normal_(layer.weight, mode='fan_in', nonlinearity='relu'). Appending ReLU does not change a module's constructor initialization. PyTorch documents nn.Linear with a uniform distribution on [-1/sqrt(fan_in), 1/sqrt(fan_in)], which has a different variance.

He Uniform

Draw weights from a uniform distribution with matched variance:

W \sim U\left[-\sqrt{\frac{6}{n_{in}}}, \sqrt{\frac{6}{n_{in}}}\right]

The bounds are derived from matching the variance of a uniform distribution to the He normal variance. Some practitioners prefer uniform for its bounded range.

Generalized Gain

Different ReLU variants preserve different fractions of the input second moment. A negative slope α retains (1+α²)/2 of a symmetric input second moment; its compensating gain accounts for that fraction. He initialization uses a gain factor:

\text{Var}(W) = \frac{\text{gain}^2}{n_{in}}
ActivationGainReasoning
ReLU\sqrt{2}Halves a symmetric input second moment
Leaky ReLU (\alpha)\sqrt{\frac{2}{1 + \alpha^2}}Negative slope retains more second moment
ELU1.0Smooth negative region preserves more signal
SELU0.75Self-normalizing property reduces needed gain

The Dead Neuron Problem

A ReLU is inactive for an input when its pre-activation is nonpositive. That is ordinary gating, not proof that a unit is permanently dead. A unit inactive on every example in a finite input set receives no local ReLU gradient on those examples, but another input or changes to earlier layers may activate it.

The grid below propagates 32 deterministic input vectors through independent dense weight matrices. Each neuron receives the full preceding activation vector. Red means zero output on all 32 examples, amber means nonzero but small RMS output, and teal means larger RMS output. The 0.1 cutoff is only a display threshold. Changing a positive initialization scale changes magnitudes without necessarily changing which units are zero.

All-zero weights prevent hidden units from learning distinct representations through ordinary symmetric updates. The zero setting makes this extreme case visible; it should not be confused with a temporary zero for one input.

Neuron activity over 32 inputs
Initialization Method
5 layers
200 neurons in a 20 by 10 grid
Inactive on all 32 inputsSmall RMS output (<0.1)RMS output ≥0.1

Deterministic sampled illustration: 32 input vectors pass through shared independent 200×200 weight matrices with zero biases. Color uses each unit’s RMS output across these inputs. A red unit is inactive on this finite sample, not proven permanently dead; other inputs or upstream parameter updates can activate it. Tiny positive outputs are distinct from exact zeros.

He vs Xavier: Head-to-Head

The difference between He and Xavier becomes dramatic in deep ReLU networks. Watch both methods propagate a signal through 20 layers — He preserves pre-activation scale in the mean-field recurrence while Xavier shrinks it.

With tanh activation, the gap narrows significantly. Both output variances remain bounded by 1. Xavier is a compromise derived near the linear regime; neither choice exactly preserves tanh variance at every layer.

He and Xavier: track the moments

Choose a layer and distinguish the centered variance from the mean square that drives the next layer.

20
Step 1 / 21
0
variance through layer 0; exact values follow

He · layer 0

Pre variance
1.000
Post variance
1.000
Post mean
0.000
Post E[a²]
1.000

Xavier · layer 0

Pre variance
1.000
Post variance
1.000
Post mean
0.000
Post E[a²]
1.000

ReLU has a positive mean. Its mean square is half the symmetric input variance; its centered variance is smaller. He compensates the mean-square factor, which stabilizes the next pre-activation variance in this model.

Analytically calculated Gaussian mean-field illustration: equal-width layers (256 units), independent zero-mean weights, zero biases. ReLU moments are exact under these assumptions; tanh moments use deterministic numerical integration. These are initialization calculations, not measured training results. Finite networks need not follow them exactly.

Choosing Your Initialization

Each method is designed for specific activation functions and network architectures. Compare their formulas, activation pairing, and depth handling.

Initialization methods

He Normal

N(0, √(2/n_in))

Best for
ReLU, Leaky ReLU
Assumes symmetric?
No
Depth handling
excellent
Conv-friendly?
Yes

Default for modern CNNs and MLPs with ReLU activations

He Uniform

U[-√(6/n_in), √(6/n_in)]

Best for
ReLU, ELU
Assumes symmetric?
No
Depth handling
excellent
Conv-friendly?
Yes

Alternative to He Normal with bounded weights

Xavier Normal

N(0, √(2/(n_in+n_out)))

Best for
Tanh, Sigmoid
Assumes symmetric?
Yes
Depth handling
good
Conv-friendly?
Yes

Networks with symmetric activations like tanh

Xavier Uniform

U[-√(6/(n_in+n_out)), √(6/(n_in+n_out))]

Best for
Tanh, Sigmoid
Assumes symmetric?
Yes
Depth handling
good
Conv-friendly?
Yes

Bounded variant of Xavier for symmetric activations

LeCun Normal

N(0, √(1/n_in))

Best for
SELU
Assumes symmetric?
No
Depth handling
moderate
Conv-friendly?
No

Self-normalizing networks with SELU activation

Orthogonal

Q from QR decomposition

Best for
RNNs, LSTMs
Assumes symmetric?
No
Depth handling
excellent
Conv-friendly?
No

Recurrent networks where gradient preservation is critical

Use He initialization when:

  • Your network uses ReLU, Leaky ReLU, ELU, or PReLU
  • You are building deep CNNs or MLPs
  • You need stable gradients through many layers

Use Xavier instead when:

  • Your network uses tanh or sigmoid activations
  • You are using transformer architectures with GELU
  • The activation is symmetric around zero

Fan-in vs Fan-out

He initialization has two modes that control different aspects of signal flow:

Fan-in mode preserves the variance of activations during the forward pass. This is the default and the right choice for most networks — it ensures signals maintain their strength as they flow from input to output.

Fan-out mode preserves the variance of gradients during the backward pass. This matters for convolutional layers with batch normalization, where gradient flow during backpropagation is the bottleneck. ResNets and similar architectures use mode='fan_out' for this reason.

In practice, the difference between fan-in and fan-out is small for layers where n_{in} \approx n_{out}. It only matters significantly when the layer changes dimensionality dramatically.

Common Pitfalls

1. Wrong Activation Pairing

He initialization is designed for ReLU-family activations. Using it with tanh or sigmoid overestimates the needed variance, causing initial activations to saturate. Xavier is the correct choice for symmetric activations.

2. Ignoring Batch Normalization

Batch normalization re-normalizes activations at every layer, reducing sensitivity to initialization. However, initialization still affects the first few training steps and gradient magnitudes. For networks with BN, use He with mode='fan_out' to prioritize gradient flow.

3. Residual Network Scaling

In residual networks, the skip connection adds the identity to the transformed signal. Without adjustment, variance doubles at every residual block. The fix is to zero-initialize the last batch normalization layer in each residual branch, so the block initially acts as an identity function.

4. Very Deep Networks Without Normalization

For networks deeper than 50 layers without batch normalization, even He initialization may not suffice. Techniques like FIXUP initialization scale weights by d^{-1/4} (where d is depth) to account for the accumulation effect across many layers.

If you found this explanation helpful, consider sharing it with others.

Mastodon