TL;DR
- BatchNorm normalizes each feature using training mini-batch statistics, then applies learned scale and shift; convolutional statistics also span spatial positions within each channel.
- Training outputs and gradients depend on companion examples, so backpropagation must include the mean and variance computations.
- Frozen statistics make inference a deterministic affine map; normalization axes, epsilon placement, and variance conventions define the exact operator.
- The original Inception results distinguish BN alone from a larger recipe change, and training-step savings from latency measurements.
- Reducing internal covariate shift is the paper's motivating explanation, not a settled universal account of BatchNorm's optimization benefits.
Why normalize inside a model
Normalizing the data before training is useful, but the inputs to an internal layer are produced by other trainable layers. Their scale and distribution can change every time the network updates. Batch Normalization puts a differentiable normalization transform inside the model, estimates its statistics from the current mini-batch during training, and adds a learned scale and shift so that standardization does not force a fixed output representation.
Ioffe and Szegedy's ICML 2015 paper is influential because it connects an explicit operator with a complete training-and-inference procedure and a substantial Inception study. Reading only “subtract mean, divide by standard deviation” misses the cross-example gradient paths, the variance convention, and the need to replace batch-dependent behavior at inference. Original proceedings paper, §3 and Algorithms 1–2
The figures here contain calculated teaching values, not a trained classifier or a reproduction of the ImageNet experiments. The review date above is separate from the original publication year. Original-paper excerpts come from the proceedings PDF and retain their page coordinates and hashes.
What is being normalized?
Consider one feature across m examples: x1 through xm. Algorithm 1 computes the mean of those m values and their mean squared deviation, dividing by m. It then centers each value and divides by the square root of that variance plus epsilon. Repeating this independently for every feature is cheaper than whitening an entire feature vector: it does not estimate and invert a covariance matrix or decorrelate feature dimensions. Paper, §3 and Algorithm 1
This distinction defines the axes. In a fully connected layer, a feature has one coordinate per example, so the batch supplies its normalization set. For a convolutional feature map, the paper shares one gamma and beta per channel and collects statistics across both the mini-batch and spatial positions. A channel with m examples and p×q locations therefore supplies m·p·q values. BatchNorm is not normalizing all channels of each individual pixel together; that would describe a different normalization geometry. Paper, §3.2
Change one sample, move the whole feature
One scalar feature across four training examples. The mean and variance belong to the batch, not to each example independently.
Every axis uses the same −6 to 16 scale. Numbered dots follow the same four examples; amber 4 is the movable sample. Shading shows the spread, and the dashed line follows the mean.
Before · input x
Dashed mean 3.750 · shaded ±1σ (2.681)
Sample values · same identities 1–4
Normalize · x̂ = (x − μ) / √(σ² + ε)
Dashed mean 0.000 · shaded ±1σ (1.000)
Sample values · same identities 1–4
After · y = 1x̂ + 0
Dashed mean 0.000 · shaded ±1σ (1.000)
Sample values · same identities 1–4
μ=3.7500 · σ²=7.1875 · mean(y)=0.00
Move the fourth input while keeping the first three fixed. All normalized values change because all share the mean and denominator. A training example's output is consequently a function of its companions. A batch-dependent transform can supply useful noise and change optimization behavior, but it also makes batch formation and data distribution relevant. A deliberately unrepresentative batch cannot be assumed to estimate the same moments as the training population.
Epsilon is added inside the square root here. It keeps the calculation defined for a constant feature, whose centered values become zero. It also means normalized variance is sigma-squared divided by sigma-squared plus epsilon, rather than exactly one. Saying that the transform “always gives unit variance” is an approximation that fails precisely at small-variance boundaries. For a singleton normalization set, the algebra is defined with positive epsilon but its output carries no varying normalized signal; the original training procedure assumes a useful mini-batch larger than one.
Why learn gamma and beta after normalization?
Standardizing a layer's input can constrain what subsequent computation sees. The paper therefore learns an affine transformation after normalization: y = gamma·x-hat + beta. Gamma restores a controllable scale and beta restores a controllable offset. They are optimized with the network's original weights, not recalculated as batch statistics. The distinction between a learned parameter and a statistic matters for both differentiation and inference.
For a fixed population mean and variance, appropriate gamma and beta can express the identity map. This is an argument about the transform's representational flexibility. During training, the statistics depend on each mini-batch, so one fixed pair of parameters does not invert every arbitrarily different batch normalization simultaneously. The ability to represent an identity under the intended statistics should not be turned into a claim that the training operator is pointwise invertible without knowing those statistics. Paper, §3
The paper commonly inserts normalization before a nonlinearity, as g(BN(Wu)). A bias added immediately before centering cancels, which explains why such a layer may omit that preceding bias. The learned beta supplies a shift after centering. This is a local algebraic observation about the placement of operations; it does not justify removing arbitrary biases from other parts of a network.
Backpropagation must include the batch
The output y1 depends directly on x1, but it also depends on x2 through the mean and variance. Detaching those statistics during training discards real derivative paths and implements a different operation. The original paper explicitly differentiates through both statistics, then gives derivatives for gamma and beta. The compact backward formula in the code below is an equivalent vector-Jacobian product. Paper, §3 derivatives
One output sends gradients to every batch member
Select a scalar loss L=yᵢ. Each number is ∂L/∂xⱼ, including paths through the batch mean and variance.
Upstream ∂L/∂y (output 1)
Full input gradient ∂L/∂x
L = y1 → direct path + shared mean + shared variance
Every input participates in μ and σ². Teal is the direct term; amber terms travel through those shared statistics. Bars share one signed scale: left is negative, center is zero, right is positive.
Input x1 = 1sum 0.1816
Direct+0.3730
Through μ-0.0933
Through σ²-0.0981
Input x2 = 2sum -0.1557
Direct+0.0000
Through μ-0.0933
Through σ²-0.0624
Input x3 = 4sum -0.0843
Direct+0.0000
Through μ-0.0933
Through σ²+0.0089
Input x4 = 8sum 0.0584
Direct+0.0000
Through μ-0.0933
Through σ²+0.1516
Common bar endpoints: −0.3730 and +0.3730.
Σ input gradients = -0.00000000 · dγ=-1.0258 · dβ=1
Select a loss equal to one output. Its gradient generally reaches every input in the normalization set. Some entries can have opposite signs: increasing a companion changes both centering and scale. The sum of input gradients is zero up to numerical error because shifting all inputs equally leaves centered differences unchanged. This gives a useful independent invariant in addition to checking the formula term by term.
A finite-difference check provides a second test. Perturb one input while recomputing the entire forward batch, evaluate the change in a supplied scalar loss, and compare it with the analytic derivative. Reusing cached statistics while perturbing only the selected value would test the detached-statistics operator instead. The canonical example recomputes mean and variance on both sides of each perturbation. It also checks gamma and beta separately so agreement on input gradients cannot hide an affine-parameter mistake.
Inference is a different, deterministic operator
The paper wants an inference output to depend on the input itself rather than whichever other examples happen to arrive with it. Algorithm 2 therefore freezes the network's parameters and replaces training mini-batch moments with population estimates collected from training data. With fixed mean, variance, gamma, beta, and epsilon, BatchNorm becomes a single affine map. It can be combined algebraically with a preceding linear transformation. Paper, §3.1 and Algorithm 2
Freeze the statistics, remove test-batch dependence
The first sample stays x=2. Change only its companion and compare training normalization with a frozen-statistic affine map.
Follow teal sample 1 (x=2) while amber companion 4 moves. Dashed rings mark sample 1 with the initial companion x₄=8; a dashed segment shows its displacement. Both output axes share the same −3 to 9 scale.
Before · supplied batch inputs
Sample values · same identities 1–4
Training: companion → μ=4.250, σ²=5.188 → every output
Inference: stored μ=3, σ²=4 → each output independently
Training · companion changes the shared statistics
Sample values · same identities 1–4
Frozen inference · sample 1 stays at ≈−0.5
Sample values · same identities 1–4
For x=2, y=-0.987877. The other examples affect this value through μ and σ².
The original procedure estimates population mean by averaging batch means and estimates variance using m/(m−1) times the average of batch variances, for equal-size training batches. That correction is separate from Algorithm 1's training normalization, which uses the variance divided by m. Silently using the unbiased m−1 denominator in training changes the operator and its gradient. Conversely, retaining noisy current-batch statistics in an ordinary inference path changes its dependence on other test examples.
Modern libraries often maintain moving statistics during training and expose their own momentum semantics and variance conventions. The scalar reference here follows the paper's explicit equal-batch estimator for illustration and accepts fixed statistics for inference. It does not claim to reproduce every framework's running update. Nor is the original estimator the exact pooled variance of any arbitrary collection of finite batches with differing means: its population-estimation interpretation relies on the intended representative sampling.
Original source, equations, and a small reference
1. Normalize a feature over a training mini-batch
Original paper
Algorithm 1 uses the mini-batch variance with denominator m, then learned scale and shift.
Math and tensor shapes
The reference represents one scalar feature across m examples. A dense layer repeats this across features; convolutional BN pools each channel over batch and spatial positions. Normalized variance is σ²/(σ²+ε), so it is not exactly one when epsilon matters.
Python implementation
def batch_norm(x, gamma=1.0, beta=0.0, epsilon=1e-5):
"""Algorithm 1: biased mini-batch variance (divide by m), then affine."""
if not x or not all(math.isfinite(v) for v in [*x, gamma, beta, epsilon]):
raise ValueError("Nonempty finite values required")
if epsilon <= 0:
raise ValueError("Positive epsilon required")
mean = sum(x) / len(x)
variance = sum((v - mean) ** 2 for v in x) / len(x)
normalized = [(v - mean) / math.sqrt(variance + epsilon) for v in x]
return dict(mean=mean, variance=variance, normalized=normalized,
output=[gamma * v + beta for v in normalized])2. Differentiate through both batch statistics
Original paper
The original derivative chain includes paths through mean, variance and the affine parameters.
Math and tensor shapes
Here dᵢ=∂L/∂yᵢ. The compact vector-Jacobian form is algebraically equivalent to the original chain. dγ=Σdᵢx̂ᵢ and dβ=Σdᵢ. The numerical check perturbs every input independently and compares central differences against this derivative, including nonzero epsilon.
Python implementation
def backward(x, dy, gamma=1.0, epsilon=1e-5):
"""Differentiate through mean AND variance; vector-Jacobian product."""
if len(dy) != len(x) or not all(math.isfinite(v) for v in dy):
raise ValueError("Finite gradient of matching shape required")
f = batch_norm(x, gamma=gamma, epsilon=epsilon)
n, xhat = len(x), f['normalized']
dbeta = sum(dy)
dgamma = sum(d * h for d, h in zip(dy, xhat))
dx = [gamma / math.sqrt(f['variance'] + epsilon) / n
* (n * d - dbeta - h * dgamma) for d, h in zip(dy, xhat)]
return dict(dx=dx, dGamma=dgamma, dBeta=dbeta)3. Replace the stochastic batch transform with a fixed affine map
Original paper
Algorithm 2 freezes population estimates and makes inference independent of companion examples.
Math and tensor shapes
For equal-size batches above one sample, the supplied estimator follows the paper’s m/(m−1) correction on average training mini-batch variance. This is the paper’s population estimate, not a claim that pooled finite batches with different means have that exact pooled variance. Modern running-statistic conventions vary by framework.
Python implementation
def population_stats(batches):
"""Algorithm 2 estimate: equal-sized batches, m/(m-1) variance correction."""
if not batches or len(batches[0]) < 2:
raise ValueError("Batches must contain at least two samples")
m = len(batches[0])
if any(len(b) != m for b in batches):
raise ValueError("Equal batch sizes required")
stats = [batch_norm(b) for b in batches]
return dict(mean=sum(s['mean'] for s in stats) / len(stats),
variance=m / (m - 1) * sum(s['variance'] for s in stats) / len(stats))
def inference(x, mean, variance, gamma=1.0, beta=0.0, epsilon=1e-5):
"""Frozen statistics: each result is independent of other test examples."""
if not x or not all(math.isfinite(v) for v in [*x, mean, variance, gamma, beta, epsilon]):
raise ValueError("Nonempty finite values required")
if variance < 0 or epsilon <= 0:
raise ValueError("Nonnegative variance and positive epsilon required")
return [gamma * (v - mean) / math.sqrt(variance + epsilon) + beta for v in x]Original excerpts: Sergey Ioffe, Christian Szegedy; ICML 2015 proceedings PDF. PDF/crop hashes and coordinates are retained in the source manifest.
The canonical Python program uses only the standard library. Run it for its finite-difference checks or pass --json to inspect the deterministic forward, backward, population-estimate, and inference payload. The README and requirements state what is omitted: no Inception training, convolutional implementation, optimizer, dataset, recorded checkpoint, or accelerator measurement.
What happened in the original Inception study?
Section 4.2 evaluates an Inception variant on ILSVRC2012 ImageNet classification, replacing 5×5 convolutions with pairs of 3×3 convolutions. The network has 13.6 million parameters and uses ReLU convolutional layers. Training uses mini-batches of 32 and distributed asynchronous SGD with momentum; validation curves measure single-crop top-1 accuracy. The setup matters because the paper's headline acceleration is measured in training steps to a target validation accuracy, not a hardware-independent latency ratio. Paper, §4.2
The baseline Inception reaches 72.2% after 31.0 million steps. Adding BN alone, labeled BN-Baseline, reaches that level after 13.3 million steps. BN-x5 reaches it after 2.1 million steps and is the source of the paper's roughly fourteenfold claim. But BN-x5 also increases the initial learning rate fivefold and includes the §4.2.1 recipe changes: removing dropout, changing shuffling, reducing L2 regularization, accelerating learning-rate decay, removing local response normalization, and reducing photometric distortion. The full improvement cannot be attributed to the isolated normalization arithmetic as though those settings were held constant. Paper, Figures 2–3 and §4.2.2
The reported 4.82% top-5 test error is a different result: a six-network BN-Inception ensemble with multicrop inference, evaluated by the ImageNet test server. Figure 4 distinguishes it from single-model, single-crop validation results. The paper reports 7.82% top-5 error for its single-crop single-network entry. These results should not be mixed into one unspecified “BatchNorm accuracy.” Neither the small figures above nor the executable scalar example reproduces any of those classification numbers. Paper, §4.2.3 and Figure 4
The motivation is not a settled causal explanation
The title attributes the benefit to reducing internal covariate shift, the changing distribution of a layer's inputs as preceding parameters update. That is the authors' motivation and interpretation. It should be taught alongside the actual transform without elevating it into a settled explanation of why all BatchNorm networks optimize better.
Santurkar, Tsipras, Ilyas, and Madry's NeurIPS 2018 study challenges distributional stabilization as the central cause and studies smoother optimization behavior and more predictive gradients. Their results show why a good historical review distinguishes the original mechanism, the original causal hypothesis, and later investigations. They do not license replacing one universal slogan with another that ignores architecture, loss, and batch regime. How Does Batch Normalization Help Optimization?
The implementation lessons are concrete even when the broader explanation is debated: choose the correct normalization axes; differentiate through the training statistics; preserve the epsilon and variance conventions; and evaluate with the intended frozen-statistic path. Continue with the BatchNorm concept, internal covariate shift, or batch versus layer normalization to compare the mechanism with nearby ideas.
