Skip to main content

Layer Normalization for Transformers

Learn layer normalization for transformers and sequence models: how normalizing across features enables batch-independent training.

Layer Normalization

Batch normalization transformed deep learning by stabilizing training, but it carries a fundamental limitation: it depends on batch statistics. Small training reductions can make estimates noisy. Default BatchNorm evaluation instead uses stored statistics and supports independent, batch-one inference.

Layer normalization solves this by normalizing each sample independently across its own features. Instead of asking "how does this feature compare across the batch?", layer norm asks "how does each feature compare to the other features within this single sample?" This shift in perspective is why layer norm became the default normalization in transformers and sequence models.

The Individual Grading Analogy

Consider two approaches to grading an exam. Batch normalization grades on a curve: each student's score is adjusted relative to the entire class average. If the class happens to be unusually strong or weak, every individual grade shifts accordingly. Layer normalization takes a different approach: it evaluates each student against their own performance across all subjects. A student who scores 90 in math, 60 in English, and 75 in science gets normalized based on their personal mean of 75 — independent of how anyone else performed.

This means layer norm never needs to see other students in the batch. Each sample carries enough information to normalize itself, which is exactly why it works for online learning, variable-length sequences, and single-sample inference.

Compare the same scores by subject or by student, then press Play to normalize them. Alice's Science score of 42 is judged against her own average of 71.5 with layer norm, giving the same result for 1 student or 100; batch norm compares her Math score of 85 with Bob's 55 and Charlie's 30, so adding or removing a student changes everyone's normalized score.

Grading axes
Grading
Step 1 / 4
StudentMathScienceEnglishHistory
Alice
85
42
91
68
Bob
55
60
58
63
Charlie
30
95
25
78

Original scores.

Layer Norm: Grade each student relative to their own average across subjects. Statistics computed per student (row-wise).

The Mathematics

Here a feature vector means one token in [batch, tokens, hidden], normalized over hidden. On images, LayerNorm follows the specified normalized_shape, which may include channels and spatial axes. State those axes explicitly.

For a feature vector with H entries, layer normalization first computes the mean across all features:

\mu = \frac{1}{H} \sum_{i=1}^{H} x_i

Then computes the variance across those same features:

\sigma^{2} = \frac{1}{H} \sum_{i=1}^{H} (x_i - \mu)^{2}

Each feature is then centered and scaled to unit variance:

\hat{x}_i = \frac{x_i - \mu}{\sqrt{\sigma^{2} + \epsilon}}

Finally, learnable parameters \gamma (scale) and \beta (shift) restore the network's ability to represent any affine transformation of the normalized values:

y_i = \gamma_i \cdot \hat{x}_i + \beta_i

The critical point is that \mu and \sigma^2 are computed from a single sample's features — there is no dependence on other samples in the batch. The small constant \epsilon (typically 1e-5 or 1e-6) prevents division by zero when all features happen to be identical.

Adjust gamma and beta or generate a new sample to see how layer normalization transforms its features. Notice how the mean shifts to zero and the spread normalizes to unit variance, regardless of the original scale; gamma scales the normalized output and beta shifts it, allowing the network to learn the optimal distribution for each layer.

Single-sample features
1.00
0.11.03.0
0.00
-3.00.03.0
View
Original and normalized features, with mean line and standard deviation band
OriginalNormalizedOriginal mean ± std

μ and σ computed across ALL features of ONE sample

y = γ × (x - μ) / √(σ² + ε) + β
where μ = 0.96, σ = 3.90, γ = 1.00, β = 0.00

Batch Norm vs Layer Norm

The difference between batch and layer normalization comes down to which axis you normalize across. In a tensor with shape [Batch, Features], batch norm computes statistics vertically down each feature column across all samples, while layer norm computes statistics horizontally across all features within each sample.

During training, BatchNorm couples values within each channel’s reduction group. A one-value group is degenerate, but BatchNorm2d with N=1 can still pool multiple spatial values. Default evaluation uses running statistics and supports a single input. LayerNorm computes its chosen within-sample reduction in both modes.

Batch norm also behaves differently during training and inference (using running statistics at test time), which introduces a train-test discrepancy. Layer norm computes the same way in both phases, eliminating this source of subtle bugs.

Press Play or step through the highlighted axes in the same data: down a feature column for batch norm, across a sample row for layer norm. This independence is why layer norm is the default choice in Transformers, RNNs, and any model processing variable-length sequences.

Normalization axes
Normalization
Step 1 / 4
Batch Samples ↓Features →
Samplef1f2f3f4f5f6f7f8
S14.20.43.0-0.43.13.90.92.2
S23.11.7-1.9-1.42.7-1.81.11.0
S32.7-0.62.21.92.70.54.71.7
S40.01.22.3-2.60.0-0.14.01.1

→ Sample S1: selected normalization slice.

Values: [4.2, 0.4, 3.0, -0.4, 3.1, 3.9, 0.9, 2.2]

Selected sliceOther values

Layer Norm in Transformers

Layer normalization is the normalization technique used in virtually all transformer architectures, but exactly where it appears matters significantly.

Post-norm (the original transformer) applies layer norm after the residual connection: \text{LayerNorm}(x + \text{Sublayer}(x)). This was the design in "Attention Is All You Need" but turns out to be harder to train for very deep models because gradients must flow through the normalization at every layer.

Pre-norm applies layer norm before the sublayer: x + \text{Sublayer}(\text{LayerNorm}(x)). This creates a clean residual pathway where gradients flow directly through the addition, bypassing normalization entirely. Pre-norm is now standard in GPT, LLaMA, and most large language models because it helps maintain an identity residual path. Warmup and other stability choices still depend on the training recipe.

Compare both placements and press Play to follow data through the attention, feed-forward, and residual blocks. GPT-2 uses pre-norm blocks and a final normalization after the block stack. Xiong et al. analyze why pre-norm can reduce dependence on warmup in their studied settings; this is not a promise that every pre-norm model can omit warmup.

Transformer placement within one block
Placement
Step 0 / 8

Input ready.

Pre-Norm: x + Attention(LayerNorm(x)) then x + FFN(LayerNorm(x))

This diagram shows one block. A pre-norm model such as GPT-2 also applies a final normalization after the full block stack, before its output head. Placement alone does not determine training speed.

Comparing Normalization Variants

Layer norm is one member of a family of normalization techniques. Each variant normalizes across a different set of dimensions, making it suited to different architectures and tasks.

Compare the variants' computational cost, gradient behavior, and flexibility below.

Normalization variants
Standard Layer NormDefault
y = γ(x-μ)/σ + β
Complexitymoderate
Gradient Flowgood
Memorymoderate
Adoptionexcellent
Best for: General use, BERT, ViT
RMS Norm
y = γ · x / RMS(x)
Complexityexcellent
Gradient Flowexcellent
Memoryexcellent
Adoptionexcellent
Best for: LLaMA, T5, efficient LLMs
Adaptive Layer Norm
y = γ(c)(x-μ)/σ + β(c)
Complexitymoderate
Gradient Flowgood
Memorymoderate
Adoptionmoderate
Best for: Diffusion models, DiT
Pre-Norm
x + F(LN(x))
Complexityexcellent
Gradient Flowexcellent
Memorygood
Adoptionexcellent
Best for: GPT, LLaMA, deep models
Post-Norm
LN(x + F(x))
Complexityexcellent
Gradient Flowmoderate
Memorygood
Adoptiongood
Best for: BERT, original Transformer

Use Layer Norm when:

  • You need batch-size-independent normalization
  • Working with Transformers, RNNs, or sequence models
  • Batch sizes are small or variable
  • Processing variable-length inputs

Consider RMS Norm when:

  • Training large language models where efficiency matters
  • You want to skip the mean-centering step
  • Empirical performance matches standard LN on your task
  • Memory and compute budget is tight (saves ~10-15%)

Advantages

Batch-size independence is the headline benefit. Layer norm works identically whether the batch contains 1024 samples or just 1. This makes it essential for online learning, reinforcement learning with single environment steps, and any setting where batch sizes are unpredictable.

Variable-length sequence handling follows directly. In a batch of sentences with different lengths, batch norm would need to handle padding carefully or compute statistics over misaligned positions. Layer norm sidesteps this entirely by normalizing each token's feature vector independently.

Training-inference consistency eliminates a common source of errors. There are no running mean or variance buffers to maintain, no momentum hyperparameter to tune, and no discrepancy between training and evaluation modes.

Stable gradient flow in deep models, especially when combined with pre-norm placement, can improve optimization, but does not remove the need to validate initialization, learning rates, and warmup.

Common Pitfalls

1. Normalizing Across the Wrong Dimension

The most frequent implementation mistake is applying normalization across the batch or sequence dimension instead of the feature dimension. In a tensor of shape [Batch, SeqLen, Hidden], layer norm should normalize across the last dimension (Hidden). Getting this wrong silently produces a model that trains but performs poorly.

2. Confusing Pre-norm and Post-norm Placement

Switching between pre-norm and post-norm changes the computation. The GPT-2 reference implementation retains ln_f after its pre-norm blocks. Pre-norm commonly adds a final normalization after the residual stack, separate from the normalization before each sublayer, and hyperparameters tuned for one placement may not transfer to the other.

3. Ignoring RMSNorm as an Alternative

For large language models, RMSNorm (which skips the mean-centering step and only divides by the root mean square) is computationally cheaper and performs comparably. Using full layer norm where RMSNorm suffices wastes compute at scale.

4. Epsilon Value Too Small

Setting \epsilon smaller than 1e-6 in float16 or bfloat16 training can cause numerical instability. Mixed-precision training often requires increasing epsilon to 1e-5 to avoid NaN gradients.

If you found this explanation helpful, consider sharing it with others.

Mastodon