Skip to main content

Batch Norm vs Layer Norm: When to Use Which

BatchNorm normalizes over the batch and spatial axes; LayerNorm normalizes over the channel and spatial axes for each sample. The choice changes whether your model trains stably with batch=1, depends on batch composition at inference, and behaves consistently across train and eval.

TL;DR. BatchNorm normalizes each channel using statistics pooled across the batch and spatial dimensions, so it depends on batch composition. LayerNorm normalizes each sample using statistics pooled across the channel and spatial dimensions, so it depends only on that one sample. CNNs typically use BatchNorm, transformers and any small-batch or sequence-of-ones inference path use LayerNorm. The rest of the choice falls out of those two facts.

What each one actually does

For an activation tensor of shape (N, C, H, W) — N samples, C channels, H×W spatial:

  • BatchNorm computes one mean and variance per channel. To normalize the value at position (n, c, h, w), BN uses the mean of every value in the same channel c across all n, h, w. There are C means and C variances per layer.
  • LayerNorm computes one mean and variance per sample. To normalize the value at (n, c, h, w), LN uses the mean of every value in the same sample n across all c, h, w. There are N means and N variances per batch.

Both then apply the same shape: subtract mean, divide by sqrt(variance + ε), then scale and shift by learned per-channel parameters γ and β.

The visualization below shows which cells get pooled together for each variant in a tensor of four samples, six channels and eight spatial positions per block. Select a block to change the focus position; the highlighted region is "every cell that gets averaged together with this one."

Normalization axes
Variant

Normalize per channel, across the batch + spatial axes

c0c1c2c3c4c5n0n1n2n3
highlighted cells share this normalization groupoutlined block: focus cell

For each channel c, BN computes mean and variance over every (n, h, w) position. During training, stats depend on batch composition — that's the strength (regularization) and the weakness (small-batch instability, train/test divergence).

Side-by-side

AspectBatch NormLayer Norm
Normalization axisper-channel; pools over (N, H, W)per-sample; pools over (C, H, W)
Number of mean/var pairsC (one per channel)N (one per sample)
Depends on batch composition?Yes — mean/var change as batch changesNo — each sample is normalized independently
Batch size 1Eval with running statistics works. Training needs more than one value per channel across its reduction axes.Works when the normalized feature set has useful variation.
Train vs eval behaviorDifferent. Train uses batch stats; eval uses running statsIdentical. Same math both modes
Sensitive to batch composition order?Yes — bad batches produce bad statsIndependent of other samples for the selected axes
Implicit regularizationYes — noise from changing batches helps generalizationNone from normalization itself
State with affine parameters2C learned values (γ, β) plus 2C running-stat buffers2 × product(normalized_shape) learned values; no running-stat buffers
Compute costLocal reduction plus running-stat updates; SyncBN adds communicationLocal reduction over the chosen feature axes
Distributed trainingLocal BN works; SyncBatchNorm optionally pools across GPUsWorks with no extra sync — stats are local
Sequence modelsAwkward — different sequence lengths per sample, batch stats become unstableNative — designed for variable-length sequences
CNNsDefault choice for ResNet-style classifiersWorks but loses regularization benefit
TransformersUnusual; reduction and masking need careful designCommon choice; RMSNorm is also widely used
Inference latencyFixed affine transform in eval; often foldableMean and variance over normalized axes

Choosing the axes and training behavior

Use the architecture and the statistics available during training to choose normalization. Transformer LayerNorm usually reduces only the last hidden dimension of [batch, tokens, hidden], independently for each token. For images, LayerNorm can instead reduce [C,H,W], depending on normalized_shape; “per sample” alone does not specify the axes.

BatchNorm2d estimates one mean and variance per channel over N×H×W. Batch size one is not automatically degenerate: [1,1,2,2] containing [1,2,3,4] has biased variance 1.25. The degenerate training case is one value per channel, such as [1,C,1,1]. Correlated spatial samples and small reductions can still make estimates noisy.

Default BatchNorm evaluation uses stored running statistics, so it supports batch-one serving. Call model.eval() for inference. With track_running_stats=False, evaluation also uses current batch statistics, so batch composition and reduction size matter then too. PyTorch BatchNorm2d documentation

  • CNN with reliable per-channel training statistics: BatchNorm is a common starting point. Preserve a pretrained architecture unless experiments justify a change.
  • Transformer token representations: LayerNorm or an architecture's specified RMSNorm keeps the reduction within each token.
  • CNN with noisy small-batch training statistics: Consider GroupNorm, frozen BatchNorm statistics, or synchronized statistics and validate on the task.
  • Style transfer: InstanceNorm reduces spatial dimensions independently within each channel. Its statistical axes differ from LayerNorm across all channels and spatial dimensions.

Distribution shift and optimization

Internal covariate shift was BatchNorm's original motivation. Santurkar et al. challenged that causal account and provided evidence for smoother optimization in their experiments. Normalizing moments does not freeze the entire distribution or guarantee successful training. See BatchNorm, LayerNorm, and internal covariate shift.

Common pitfalls

  • Serving in training mode: Current-batch statistics can make predictions depend on other requests. Use evaluation mode with the intended running buffers; a change in serving batch size alone does not require replacing BatchNorm.
  • Padding in a normalization reduction: If BatchNorm pools sequence positions, padding can contaminate its statistics. Token-wise LayerNorm does not pool tokens, though attention still needs a padding mask.
  • Fine-tuning running statistics: Decide explicitly whether to update or freeze pretrained BatchNorm statistics, based on the new data and reduction size. Changing normalization requires training and validation.
  • Equating GroupNorm(1) with every LayerNorm: GroupNorm with one group and LayerNorm over [C,H,W] share statistical axes, but their affine parameter shapes can differ. Token-wise LayerNorm is another reduction entirely.
  • Confusing gradient accumulation with statistics aggregation: SyncBatchNorm combines statistics across participating devices within a forward pass. Accumulating gradients over microbatches does not aggregate their normalization statistics, and DDP does not change this.

Compute and state

For a hidden width of 1024, token-wise LayerNorm has 2048 affine parameters. BatchNorm over 1024 channels has 2048 affine parameters and 2048 running-stat buffer values. Both perform reductions, but their axes, memory access, and synchronization differ; operation counts alone do not predict latency. Evaluation BatchNorm can often be folded into the preceding affine or convolutional layer. Measure the actual model and serving workload before making a performance claim.

If you found this explanation helpful, consider sharing it with others.

Mastodon