TL;DR. BatchNorm normalizes each channel using statistics pooled across the batch and spatial dimensions, so it depends on batch composition. LayerNorm normalizes each sample using statistics pooled across the channel and spatial dimensions, so it depends only on that one sample. CNNs typically use BatchNorm, transformers and any small-batch or sequence-of-ones inference path use LayerNorm. The rest of the choice falls out of those two facts.
What each one actually does
For an activation tensor of shape (N, C, H, W) — N samples, C channels, H×W spatial:
- BatchNorm computes one mean and variance per channel. To normalize the value at position (n, c, h, w), BN uses the mean of every value in the same channel c across all n, h, w. There are C means and C variances per layer.
- LayerNorm computes one mean and variance per sample. To normalize the value at (n, c, h, w), LN uses the mean of every value in the same sample n across all c, h, w. There are N means and N variances per batch.
Both then apply the same shape: subtract mean, divide by sqrt(variance + ε), then scale and shift by learned per-channel parameters γ and β.
The visualization below shows which cells get pooled together for each variant in a tensor of four samples, six channels and eight spatial positions per block. Select a block to change the focus position; the highlighted region is "every cell that gets averaged together with this one."
Normalize per channel, across the batch + spatial axes
For each channel c, BN computes mean and variance over every (n, h, w) position. During training, stats depend on batch composition — that's the strength (regularization) and the weakness (small-batch instability, train/test divergence).
Side-by-side
| Aspect | Batch Norm | Layer Norm |
|---|---|---|
| Normalization axis | per-channel; pools over (N, H, W) | per-sample; pools over (C, H, W) |
| Number of mean/var pairs | C (one per channel) | N (one per sample) |
| Depends on batch composition? | Yes — mean/var change as batch changes | No — each sample is normalized independently |
| Batch size 1 | Eval with running statistics works. Training needs more than one value per channel across its reduction axes. | Works when the normalized feature set has useful variation. |
| Train vs eval behavior | Different. Train uses batch stats; eval uses running stats | Identical. Same math both modes |
| Sensitive to batch composition order? | Yes — bad batches produce bad stats | Independent of other samples for the selected axes |
| Implicit regularization | Yes — noise from changing batches helps generalization | None from normalization itself |
| State with affine parameters | 2C learned values (γ, β) plus 2C running-stat buffers | 2 × product(normalized_shape) learned values; no running-stat buffers |
| Compute cost | Local reduction plus running-stat updates; SyncBN adds communication | Local reduction over the chosen feature axes |
| Distributed training | Local BN works; SyncBatchNorm optionally pools across GPUs | Works with no extra sync — stats are local |
| Sequence models | Awkward — different sequence lengths per sample, batch stats become unstable | Native — designed for variable-length sequences |
| CNNs | Default choice for ResNet-style classifiers | Works but loses regularization benefit |
| Transformers | Unusual; reduction and masking need careful design | Common choice; RMSNorm is also widely used |
| Inference latency | Fixed affine transform in eval; often foldable | Mean and variance over normalized axes |
Choosing the axes and training behavior
Use the architecture and the statistics available during training to choose normalization. Transformer LayerNorm usually reduces only the last hidden dimension of [batch, tokens, hidden], independently for each token. For images, LayerNorm can instead reduce [C,H,W], depending on normalized_shape; “per sample” alone does not specify the axes.
BatchNorm2d estimates one mean and variance per channel over N×H×W. Batch size one is not automatically degenerate: [1,1,2,2] containing [1,2,3,4] has biased variance 1.25. The degenerate training case is one value per channel, such as [1,C,1,1]. Correlated spatial samples and small reductions can still make estimates noisy.
Default BatchNorm evaluation uses stored running statistics, so it supports batch-one serving. Call model.eval() for inference. With track_running_stats=False, evaluation also uses current batch statistics, so batch composition and reduction size matter then too. PyTorch BatchNorm2d documentation
- CNN with reliable per-channel training statistics: BatchNorm is a common starting point. Preserve a pretrained architecture unless experiments justify a change.
- Transformer token representations: LayerNorm or an architecture's specified RMSNorm keeps the reduction within each token.
- CNN with noisy small-batch training statistics: Consider GroupNorm, frozen BatchNorm statistics, or synchronized statistics and validate on the task.
- Style transfer: InstanceNorm reduces spatial dimensions independently within each channel. Its statistical axes differ from LayerNorm across all channels and spatial dimensions.
Distribution shift and optimization
Internal covariate shift was BatchNorm's original motivation. Santurkar et al. challenged that causal account and provided evidence for smoother optimization in their experiments. Normalizing moments does not freeze the entire distribution or guarantee successful training. See BatchNorm, LayerNorm, and internal covariate shift.
Common pitfalls
- Serving in training mode: Current-batch statistics can make predictions depend on other requests. Use evaluation mode with the intended running buffers; a change in serving batch size alone does not require replacing BatchNorm.
- Padding in a normalization reduction: If BatchNorm pools sequence positions, padding can contaminate its statistics. Token-wise LayerNorm does not pool tokens, though attention still needs a padding mask.
- Fine-tuning running statistics: Decide explicitly whether to update or freeze pretrained BatchNorm statistics, based on the new data and reduction size. Changing normalization requires training and validation.
- Equating GroupNorm(1) with every LayerNorm: GroupNorm with one group and LayerNorm over
[C,H,W]share statistical axes, but their affine parameter shapes can differ. Token-wise LayerNorm is another reduction entirely. - Confusing gradient accumulation with statistics aggregation: SyncBatchNorm combines statistics across participating devices within a forward pass. Accumulating gradients over microbatches does not aggregate their normalization statistics, and DDP does not change this.
Compute and state
For a hidden width of 1024, token-wise LayerNorm has 2048 affine parameters. BatchNorm over 1024 channels has 2048 affine parameters and 2048 running-stat buffer values. Both perform reductions, but their axes, memory access, and synchronization differ; operation counts alone do not predict latency. Evaluation BatchNorm can often be folded into the preceding affine or convolutional layer. Measure the actual model and serving workload before making a performance claim.
Related concepts
Learn layer normalization for transformers and sequence models: how normalizing across features enables batch-independent training.
Learn batch normalization in deep learning: how normalizing layer inputs accelerates training, improves gradient flow, and acts as regularization.
Understand internal covariate shift: why layer input distributions change during training, how it slows convergence, and how batch norm fixes it.
Learn how skip connections and residual learning enable training of very deep neural networks. Understand the ResNet revolution with interactive visualizations.
Explore VAE latent space in deep learning. Learn variational autoencoder encoding, decoding, interpolation, and the reparameterization trick.
Explore an illustrative region-tiling policy: resize tiles, calculate visual tokens, and distinguish analytical savings from measured model quality.
