Skip to main content

Xavier/Glorot Initialization

Learn Xavier (Glorot) initialization: how it balances forward signals and backward gradients to enable stable deep network training with tanh and sigmoid.

Xavier/Glorot Initialization: Balancing Signals and Gradients

Xavier initialization, introduced by Glorot and Bengio in 2010, chooses weight scale using both fan-in and fan-out. Its derivation balances forward activation and backward gradient variance for linear or approximately linear, zero-centered activations. Tanh is compatible near its linear region; sigmoid is not zero-centered and can saturate. Glorot and Bengio’s original analysis discusses these limitations.

The core idea is to treat forward and backward propagation as two constraints. Equal fan counts can satisfy both in the linear approximation. Unequal fan counts require a compromise; the formula does not guarantee constant activation or gradient variance at arbitrary depth.

The Balancing Scale Analogy

Think of each layer in a neural network as a balancing scale with two trays. One tray holds the forward-flowing signal (activations), the other holds the backward-flowing gradient. If you set weights using only the number of inputs (fan-in), the forward signal stays strong but the gradient may weaken. If you use only the number of outputs (fan-out), gradients stay strong but signals may drift. Xavier balances the two constraints rather than satisfying both exactly for unequal fan counts.

The Balancing Scale

Xavier initialization balances two competing needs: controlling forward activation scale and backward gradient scale. Exact preservation of both is possible only under compatible fan counts and activation assumptions.

Authored scale analogy: the displayed factors are chosen examples, not a simulated tanh network or exact Xavier moments. The later explorer computes the fan-count compromise.

Xavier averages fan_in and fan_out, compromising between the forward and backward scale constraints; unequal fan counts cannot satisfy both exactly.

Forward Signal (activations)
1.00
Input
0.98
Layer 1
0.96
Layer 2
0.95
Layer 3
0.93
Layer 4
Backward Gradient (loss signal)
1.00
Output
0.98
Layer 3
0.96
Layer 2
0.95
Layer 1
0.93
Layer 0
Step 1 / 5
—
Forward signal
—
Backward gradient
—
Scale equilibrium

The Mathematical Foundation

Forward Pass Variance

Consider a single linear layer with n_{in} inputs. Each output is a weighted sum of inputs:

y = \sum_{i=1}^{n_{in}} w_i \cdot x_i

Assuming weights and inputs are independent with zero mean, the variance of the output is:

\text{Var}(y) = n_{in} \cdot \text{Var}(W) \cdot \text{Var}(x)

To preserve variance across this layer (so the output has the same scale as the input), you need:

\text{Var}(W) = \frac{1}{n_{in}}

Backward Pass Variance

During backpropagation, gradients flow in the opposite direction. The gradient with respect to each input involves a sum over n_{out} output connections:

\text{Var}\left(\frac{\partial L}{\partial x}\right) = n_{out} \cdot \text{Var}(W) \cdot \text{Var}\left(\frac{\partial L}{\partial y}\right)

To preserve gradient variance, you need a different constraint:

\text{Var}(W) = \frac{1}{n_{out}}

The Xavier Compromise

Under the linear, independent initialization approximation, these two constraints conflict whenever n_{in} \neq n_{out}. Xavier uses a compromise between the fan counts; it does not satisfy both exact constraints when they differ:

\text{Var}(W) = \frac{2}{n_{in} + n_{out}}

This is the harmonic mean of the two requested weight variances. For unequal fan counts, the forward multiplier is 2n_in/(n_in+n_out) and the backward multiplier is 2n_out/(n_in+n_out). One can be far from one; “balanced” means a compromise, not exact preservation.

Variance Preservation Through Layers

Propagation through many layers also depends on the activation. Tanh compresses large inputs, so even equal-width Xavier layers can show decreasing output variance. Large weights can saturate tanh while its output variance remains bounded. The next calculation uses gain 1 for the base Xavier formula.

The following Gaussian mean-field calculation separates the pre-activation variance from the bounded tanh output variance. It is an analytical teaching model, not a sampled training run. Inspect a layer to see the transformed distribution; tanh outputs cannot have variance above 1.

Inspect the activation distribution

Choose a layer and distinguish the centered variance from the mean square that drives the next layer.

Step 1 / 11
0
variance through layer 0; exact values follow

Xavier · layer 0

Pre variance
1.000
Post variance
1.000
Post mean
0.000
Post E[a²]
1.000

Layer 0 distribution: each bar is probability mass in one bin; heights use the largest bin as a reference. Tanh outputs stay in [−1, 1]. Large input scales accumulate mass near the endpoints.

Calculated activation histogram for layer 0

Tanh is bounded: post-activation variance cannot exceed 1. Larger pre-activation spread can increase saturation even when the output spread remains bounded. Xavier is a forward/backward compromise near the linear regime, not exact tanh variance preservation.

Analytically calculated Gaussian mean-field illustration: equal-width layers (256 units), independent zero-mean weights, zero biases. ReLU moments are exact under these assumptions; tanh moments use deterministic numerical integration. These are initialization calculations, not measured training results. Finite networks need not follow them exactly.

Xavier Initialization Variants

Xavier Normal

Draw weights from a Gaussian distribution centered at zero (second parameter: variance):

W \sim \mathcal{N}\left(0, \frac{2}{n_{in} + n_{out}}\right)

The standard deviation is the square root of the Xavier variance. This is the most common variant and produces a bell-shaped distribution of weights concentrated around zero.

Xavier Uniform

Draw weights from a uniform distribution with matched variance:

W \sim U\left[-\sqrt{\frac{6}{n_{in} + n_{out}}}, \sqrt{\frac{6}{n_{in} + n_{out}}}\right]

The bounds come from the variance of a uniform distribution: for U[-a, a], the variance is a^2 / 3. Setting this equal to the Xavier variance and solving for a gives the square root of six divided by the sum of fan-in and fan-out. Some practitioners prefer uniform for its bounded range, which can be helpful for debugging.

Gain Factors for Different Activations

The base Xavier formula assumes linear activation. For non-linear activations, a gain factor scales the variance to compensate for the activation's effect on signal magnitude.

ActivationGainEffect
Linear / Identity1.0No scaling needed
Sigmoid1.0Approximately linear near zero
Tanh5/3Compensates for tanh's compression
ReLUUse He insteadXavier is not designed for ReLU

With gain applied, the variance becomes \text{Var}(W) = \text{gain}^2 \cdot \frac{2}{n_{in} + n_{out}}.

Fan-in and Fan-out Explorer

The fan-in is the number of input connections to a neuron, and the fan-out is the number of output connections. For a fully connected layer mapping 256 inputs to 128 outputs, fan-in is 256 and fan-out is 128. For convolutional layers, these numbers include the kernel size: a 3x3 convolution with 64 input channels has a fan-in of 3 x 3 x 64 = 576.

Fan-in / Fan-out Explorer

Adjust the number of input and output connections to see how Xavier computes the weight distribution. The wider the layer, the smaller each weight must be to preserve signal strength.

162565121024
162565121024
Weight distribution: N(0, 0.0625)
-0.25000.250
0.00391
Xavier variance
0.0625
Std deviation
1.0:1
fan_in : fan_out
MethodFormulaVarianceProtects
Fan-in only1/n_in0.00391Forward pass
Fan-out only1/n_out0.00391Backward pass
Xavier (avg)2/(n_in+n_out)0.00391Both (compromise)

When fan_in equals fan_out (256), Xavier, LeCun (fan_in only), and fan_out-only all compute the same variance. The compromise is exact — no trade-off needed. This is the ideal case where forward and backward passes are equally protected.

Xavier vs He: The Activation Mismatch Problem

For symmetric input, ReLU halves the second moment, not the centered variance. Equal-width Xavier scaling then shrinks the mean square through a plain ReLU stack. He compensates this factor. With tanh, larger pre-activation variance can instead increase saturation; bounded output variance alone does not establish good gradient flow.

Match the initialization gain and assumptions to the activation and architecture, then inspect both activations and gradients.

He and Xavier: track the moments

Choose a layer and distinguish the centered variance from the mean square that drives the next layer.

15
Step 1 / 16
0
variance through layer 0; exact values follow

He · layer 0

Pre variance
1.000
Post variance
1.000
Post mean
0.000
Post E[a²]
1.000

Xavier · layer 0

Pre variance
1.000
Post variance
1.000
Post mean
0.000
Post E[a²]
1.000

Tanh is bounded: post-activation variance cannot exceed 1. Larger pre-activation spread can increase saturation even when the output spread remains bounded. Xavier is a forward/backward compromise near the linear regime, not exact tanh variance preservation.

Analytically calculated Gaussian mean-field illustration: equal-width layers (256 units), independent zero-mean weights, zero biases. ReLU moments are exact under these assumptions; tanh moments use deterministic numerical integration. These are initialization calculations, not measured training results. Finite networks need not follow them exactly.

Choosing Your Initialization

Initialization Decision Guide

Given your activation function, which initialization should you use? This guide maps each method to its intended use case and rates its depth handling.

Xavier Normal
N(0, √(2/(n_in+n_out)))
Best for:
TanhSigmoidSoftmax
Depth handling:
good
Base linear fan-count compromise; often used with a gain for tanh. Sigmoid saturation and its nonzero mean need care.
Xavier Uniform
U[-√(6/(n_in+n_out)), √(6/(n_in+n_out))]
Best for:
TanhSigmoid
Depth handling:
good
Bounded variant of Xavier. Preferred when you want weights constrained to a known range.
He Normal
N(0, √(2/n_in))
Best for:
ReLULeaky ReLUPReLU
Depth handling:
excellent
Modern default for CNNs and MLPs with ReLU-family activations. Compensates for the symmetric-input ReLU second-moment half-factor.
He Uniform
U[-√(6/n_in), √(6/n_in)]
Best for:
ReLUELU
Depth handling:
excellent
Bounded alternative to He Normal. Good for debugging since weights have known bounds.
LeCun Normal
N(0, √(1/n_in))
Best for:
SELU
Depth handling:
moderate
Designed for self-normalizing networks (SNNs) with SELU activation and alpha-dropout.
Orthogonal
Q from QR decomposition
Best for:
Any (RNNs)
Depth handling:
excellent
Recurrent networks where gradient preservation over many timesteps is critical. Preserves norm exactly.
Use Xavier when:
  • Your layer is linear or approximately linear and zero-centered
  • Building transformers with GELU or GeLU-like activations
  • The network is moderately deep (under 50 layers)
  • Using standard feedforward layers without normalization tricks
Use He instead when:
  • Your activation is ReLU, Leaky ReLU, PReLU, or ELU
  • Building deep CNNs (ResNets, VGG-style)
  • The activation zeroes negative inputs (asymmetric)
  • You need stable gradients through many layers with ReLU

Common Pitfalls

1. Using Xavier with ReLU Networks

In a plain equal-width ReLU stack, base Xavier scaling gives a factor of one half in the mean-square recurrence. He is designed to compensate that factor. Normalization, residual paths, finite width, and training updates change this simple picture.

2. Forgetting Bias Initialization

Biases should almost always be initialized to zero. Non-zero biases break the symmetry assumptions that Xavier depends on and can push activations into saturated regions of tanh or sigmoid. The one exception is LSTM forget gates, which benefit from a bias of 1.0 to encourage information flow early in training.

3. Ignoring Network Depth

There is no universal depth cutoff for Xavier. Width, nonlinearities, normalization, and residual structure all matter. Inspect layer statistics at initialization and use the architecture’s tested initialization recipe.

4. Applying Xavier to Non-Standard Architectures

Residual connections, attention mechanisms, and mixture-of-experts layers change the signal flow in ways that violate Xavier's assumptions. In transformers, for instance, the attention output is a weighted sum whose variance depends on the sequence length and attention distribution — not just fan-in and fan-out. Many modern frameworks apply architecture-specific initialization on top of Xavier to account for these effects.

If you found this explanation helpful, consider sharing it with others.

Mastodon