Skip to main content

Liger Kernel: Fused RMSNorm

Derive RMSNorm, trace fused rowwise normalization, inspect mixed-precision cast boundaries, and follow the input and shared-weight gradients in Liger Kernel.

TL;DR

  • RMSNorm normalizes each token row using its mean square without subtracting the mean; Liger optimizes this existing normalization formula.
  • Fusing the row reduction, normalization, and learned scale avoids materializing intermediate feature tensors, but actual speedups require workload-specific measurement.
  • Epsilon, weight offset, dtype, and cast boundaries are part of the operation and must match the model when replacing its implementation.
  • Backward reduces across features for input gradients and across token rows for shared-weight gradients, so a fused forward does not make training a single kernel.
  • The figures and runnable Python reference explain the mathematics; they do not establish GPU timings or low-precision kernel agreement.

Why fuse normalization

A normalization layer does relatively little arithmetic: square a row, reduce it to a scale, then multiply the row by that scale and a learned weight. Yet expressing each operation separately can move the same features through device memory several times. RMSNorm is a useful place to study kernel fusion because the statistical dependency fits inside a single token row, while the learned parameters connect rows during training.

This is a focused reading of RMSNorm in Liger Kernel: Efficient Triton Kernels for LLM Training, the same paper covered in the companion fused SwiGLU review. The original preprint appeared in 2024; this page uses arXiv v3, revised January 24, 2025. Liger supplies an efficient implementation of RMSNorm, which was introduced earlier by Biao Zhang and Rico Sennrich in 2019. It does not introduce a new normalization formula. Liger paper, §3.2

The figures below calculate small authored examples. They are neither training results nor measured GPU timings. Implementation details refer to official Liger commit 37607058f981, a separate source snapshot from the paper revision.

Normalize the magnitude without centering

Consider activations with shape B×T×HB \times T \times H: batch, token, and hidden feature. RMSNorm treats the first two dimensions as M=BTM = BT independent rows and normalizes across the last dimension. For one row xx, define

\begin{aligned} s &= \frac{1}{H}\sum_{j=1}^{H}x_j^2 \\ r &= \frac{1}{\sqrt{s+\epsilon}} \\ y_j &= \gamma_j x_j r \end{aligned}

The statistic is the mean of the squares, not the variance around the mean. LayerNorm first subtracts the row mean and uses the centered variance; it can also add a learned bias. RMSNorm does not subtract the mean and uses a learned scale. For a constant positive row, centering gives zero, whereas RMS normalization gives values close to one before the learned weight. These are different transformations, even when both stabilize feature magnitude. Original RMSNorm paper

Shift the positive row in the explorer. LayerNorm’s output stays fixed while RMSNorm’s output changes. Then choose a centered row: when the mean is zero and the weights and epsilon agree, the two computations coincide.

Scaling is different from centering

Shift the same four features, then compare RMSNorm with LayerNorm. Both use the displayed weight and epsilon; LayerNorm has zero bias.

0
1.000

Input x

0
1.000
1
2.000
2
3.000
3
4.000

RMSNorm: scale about zero

0
0.365
1
0.730
2
1.095
3
1.461

LayerNorm: center, then scale

0
-1.342
1
-0.447
2
0.447
3
1.342

All bars share a fixed −12 to +12 axis across every slider setting. Thin outlines retain the input; filled bars show each result. Shift the row and watch the amber output stay in place while the teal output changes.

Mean
2.5000
Mean square
7.5000
Centered variance
1.2500
Reciprocal RMS
0.3651

Changing the shift leaves LayerNorm unchanged but generally changes RMSNorm.

Analytical illustration in JavaScript binary64. No training or mixed-precision kernel runs here. A learned weight can change the output mean and RMS.

The identity mean⁡(x2)=var⁡(x)+mean⁡(x)2\operatorname{mean}(x^2)=\operatorname{var}(x)+\operatorname{mean}(x)^2 explains the difference. RMSNorm includes the squared mean in its denominator. Its output is not required to have zero mean. A nonuniform learned weight can further change both the mean and the root mean square of the output.

Epsilon belongs inside the square root. With an all-zero row, positive epsilon keeps the reciprocal RMS finite and the output remains zero. With a tiny row, epsilon can dominate the mean square, so the normalized output need not have RMS exactly one. Scaling an input by a positive constant is approximately canceled only when epsilon is negligible compared with the row’s mean square. Changing epsilon changes the function, especially close to zero.

Fuse the work around the row reduction

The reduction determines the scope of the forward kernel. Every feature needs the same reciprocal RMS, but no other token row is needed to compute it. A rowwise Triton program can load the features, reduce their squares, normalize, apply the weight, and store the output. Liger’s pinned source also contains a block-row path for some shapes; the figure follows the simpler rowwise dataflow rather than claiming one launch configuration covers every workload. Pinned forward implementation

The useful boundary is the full normalized activation. In an explicitly staged graph, that tensor is written to memory and read again by the weight multiplication. Fusion lets these operations share values inside the program. The compiler chooses registers and any required communication or spilling; “fused” does not guarantee that every value fits in registers.

Step through the forward pass, then switch to the staged graph. The count deliberately names its graph and counts logical feature accesses, so it can be checked independently of a GPU or compiler.

Keep the normalized row inside the kernel

Follow four values through the program. In the staged graph, squared and normalized features detour through memory; the fused path avoids both round trips.

Step 1 / 7

Inputs in memory. A four-feature teaching row and its learned weight are ready. Play follows the values; no work runs on page load.

Inside the program

Device memory

Input row

Load x

Square x

No x² tensor stored

Reduce → r

Normalize x × r

No x̂ tensor stored

Apply γ

Store y

Moving values: [1.000, 2.000, 3.000, 4.000]. Motion shows data dependencies, not GPU timing.

4096

At this step: 0 feature reads and 0 feature writes. The full-pass totals remain beside them for comparison.

The normalized row stays inside the same program as the weight multiplication.

Logical feature elements for the full pass
AccessCount
Input + weight reads262,144
Tensor writes131,072
Reads + writes393,216

Training also saves 32 reciprocal RMS values, one per row, outside the feature counts above.

Animated analytical dataflow, not a hardware trace. The four-feature row illustrates dependencies; logical counts use 32 rows of the selected width. Scalar statistics, casts, cache reuse, padding and spills are excluded.

For the displayed staged graph, squaring reads and writes MHMH elements, reduction reads MHMH, normalization reads and writes MHMH, and weighting reads two tensors and writes one. That totals 8MH8MH feature-element accesses. The ideal fused path reads XX and γ\gamma and writes YY, totaling 3MH3MH logical accesses, plus the separate row-statistic output needed by this training implementation. This is an authored accounting model. Cache reuse, scalar traffic, dtype conversion, allocation, spills, and a compiler’s own fusion can all change actual memory transactions or runtime.

There is also a small but important padding rule. Triton may round a feature block up to a power of two. Padding lanes must contribute zero to the sum, stores must be masked, and the denominator remains the real hidden size, not the padded block width. Otherwise the kernel implements a different normalization.

Precision is part of the operation

Using wider accumulation protects the squared sum from low-precision overflow and reduces rounding error, but it does not specify the entire result. The location of the cast back to the activation dtype matters too. For low-precision input and weights of the same dtype, the pinned implementation distinguishes these paths:

Casting modeRow statisticNormalization and learned scale
LlamaPromote input to fp32 before squaring and reductionNormalize in fp32, cast the normalized row back, then apply the weight
GemmaPromote input and weight to fp32Normalize and apply the effective weight in fp32, then cast the output back

The weight offset is separate from the casting mode: γ=w+o\gamma = w + o, with an offset such as one supporting a different parameterization. The source also exposes a none casting mode. A caller must match the model’s epsilon, offset, dtype, and cast boundary rather than assuming all modes are interchangeable. Pinned cast branches

The mathematical reference below uses Python binary64 numbers. This makes its equations and finite differences easy to inspect, but it cannot establish bitwise agreement with a Triton kernel. A device-level check must compare the intended low-precision forward and backward graphs, including tolerances and non-power-of-two shapes. Different reduction trees and fused arithmetic can change the final low bits even when the real-valued equation agrees.

Backward has two different reductions

The forward denominator depends on every feature in its row. Let gj=∂L/∂yjg_j=\partial L/\partial y_j and qj=gjγjq_j=g_j\gamma_j. The direct path through yj=γjxjry_j=\gamma_j x_j r contributes rqjr q_j. The derivative of the shared reciprocal RMS adds a correction:

\begin{aligned} \hat{x}_j &= r x_j \\ c &= \frac{1}{H}\sum_k q_k\hat{x}_k \\ \frac{\partial L}{\partial x_j} &= r(q_j-c\hat{x}_j) \end{aligned}

This is a reduction across features within one row. Sending an upstream gradient to one output can therefore change the gradient of every input feature in that row. Select a single output below and inspect the direct and correction terms separately. Liger paper, Eq. 2

One output gradient reaches the whole row

Follow the upstream signal into the direct path and the shared RMS correction. Then accumulate the learned-weight gradient across identical token rows.

3
Step 1 / 8

One upstream gradient. Only the selected output receives dy = 1. Its weight routes this signal into the normalization derivative.

dy0 = 1

dy1 = 0

dy2 = 0

dy3 = 0

Direct: r × q

Reduce q × x̂ → c

dx0
—

dx1
—

dx2
—

dx3
—

Teal follows the direct derivative. Amber gathers one row statistic, then reaches every feature. Paths show dependencies even when their numeric contribution is zero.

Direct contribution

0
—
1
—
2
—
3
—

RMS correction

0
—
1
—
2
—
3
—

Shared γ: sum contributions across token rows

r1
r2
r3
·
·
·
·
·

Accumulated dγ after 0 of 3 identical rows

0
0.000
1
0.000
2
0.000
3
0.000
Complete input gradient = direct path + RMS correction
FeaturexdyDirectCorrectiondx
01.00001.00000.3651-0.01220.3530
12.00000.00000.0000-0.0243-0.0243
23.00000.00000.0000-0.0365-0.0365
34.00000.00000.0000-0.0487-0.0487

Row correction c = mean((dy × γ) × x̂) = 0.09129. One row contributes [0.365, 0.000, 0.000, 0.000] to dγ.

Animated derivatives of the real-valued formula with ε = 10⁻⁶. Path motion shows dependency, not duration or gradient magnitude; signed bars carry the magnitudes. Each row retains its own dx while dγ is summed.

The scale gradient has a different boundary. Every token row uses the same γj\gamma_j, so all rows contribute to the same parameter:

\frac{\partial L}{\partial \gamma_j}=\sum_{i=1}^{M}g_{ij}\hat{x}_{ij}

This is a reduction across rows for each feature. It is a sum; any averaging associated with the loss is already reflected in the incoming gradients. Repeating the same row doubles its scale contribution while leaving each copy’s input derivative unchanged.

The pinned Liger backward groups rows, accumulates fp32 partial weight gradients, and reduces the partial buffer afterward. Thus the fused forward does not imply that the complete training operation is a single kernel. Caching one reciprocal RMS per row saves recomputation of that statistic; the backward still needs the original input and, when affine scaling is enabled, the weight. Pinned backward and final reduction

From the equations to a runnable reference

Each row below pairs the relevant original-paper crop with the equation and code extracted from the same downloadable source. The code exposes intermediates for teaching and uses ordinary Python lists; it is not the fused production implementation.

1. Compute one reciprocal RMS per row

Original paper

Equation 1: root mean square with epsilon

Equation 1 defines RMSNorm using mean square and an epsilon inside the square root.

Math and tensor shapes

\begin{aligned}x&\in\mathbb{R}^{H}\\ s&=H^{-1}\sum_j x_j^2\\ r&=(s+\epsilon)^{-1/2}\\ \hat{x}_j&=r x_j\end{aligned}

No mean is subtracted. The reference returns the normalized row and its reciprocal RMS. Python uses binary64; Liger’s Llama/Gemma paths promote inputs to fp32 before squaring. The reference is mathematical teaching code, not a bitwise model of Triton reductions.

Python implementation

reference.py · normalize
python
def normalize(x, epsilon=1e-6):
    """Return normalized features and the cached reciprocal RMS of one row."""
    validate_row(x)
    if not math.isfinite(epsilon) or epsilon <= 0:
        raise ValueError("Epsilon must be finite and positive")
    mean_square = math.fsum(value * value for value in x) / len(x)
    if not math.isfinite(mean_square):
        raise ValueError("Squared inputs exceed the reference's numeric range")
    rstd = 1.0 / math.sqrt(mean_square + epsilon)
    return [value * rstd for value in x], rstd

2. Apply the feature weight in the same forward kernel

Original paper

Section 3.2: fused normalization and learned scalingPinned Liger RMSNorm implementation

Section 3.2 combines normalization and scaling; the optional offset is an implementation parameterization.

Math and tensor shapes

\begin{aligned}\gamma&\in\mathbb{R}^{H}\\ y_j&=\hat{x}_j\gamma_j\\ \gamma_j&=w_j+o\end{aligned}

The scale is shared across token rows. The paper’s plain parameterization uses offset o = 0; the optional offset can represent w + 1. This reference returns intermediate values for inspection. A fused implementation keeps the normalized row local instead of materializing it in device memory.

Python implementation

reference.py · forward
python
def forward(x, weight, epsilon=1e-6, offset=0.0):
    """Normalize the final dimension and apply a learned feature scale."""
    validate_row(weight)
    if len(x) != len(weight) or not math.isfinite(offset):
        raise ValueError("Weight must match the row; offset must be finite")
    x_hat, rstd = normalize(x, epsilon)
    y = [value * (scale + offset)
         for value, scale in zip(x_hat, weight)]
    return y, x_hat, rstd

3. Differentiate through the row statistic

Original paper

Equation 2: input and scale gradients

Equation 2 includes a scalar dot product that corrects the direct gradient path.

Math and tensor shapes

\begin{aligned}q_j&=\frac{\partial L}{\partial y_j}\gamma_j\\ c&=H^{-1}\sum_j q_j\hat{x}_j\\ \frac{\partial L}{\partial x_j}&=r(q_j-c\hat{x}_j)\end{aligned}

The direct term r q is incomplete because changing one input also changes the denominator of every output in its row. The negative correction carries that dependency. The derivation is exact for positive epsilon in real arithmetic; low-precision backward follows the chosen model’s casting rules.

Python implementation

reference.py · backward
python
def backward(x, weight, dy, epsilon=1e-6, offset=0.0):
    """Differentiate one row, including dependence of its RMS on every x."""
    validate_row(dy)
    if len(dy) != len(x):
        raise ValueError("Upstream gradient must match the row")
    _, x_hat, rstd = forward(x, weight, epsilon, offset)
    q = [grad * (scale + offset) for grad, scale in zip(dy, weight)]
    projection = math.fsum(a * b for a, b in zip(q, x_hat)) / len(x)
    dx = [rstd * (grad - normalized * projection)
          for grad, normalized in zip(q, x_hat)]
    dw = [grad * normalized for grad, normalized in zip(dy, x_hat)]
    return dx, dw

4. Sum the scale gradient over token rows

Original paper

Equation 2 and the reduction across input rowsPinned Liger RMSNorm implementation

The sentence following Equation 2 requires aggregation because all rows use the same scale.

Math and tensor shapes

\begin{aligned}X&\in\mathbb{R}^{M\times H}\\ \frac{\partial L}{\partial w_j}&=\sum_{i=1}^{M}\frac{\partial L}{\partial Y_{ij}}\hat{X}_{ij}\end{aligned}

Input gradients stay separate for each row, but weight contributions must be summed, not averaged by default. The offset is constant so its derivative with respect to w is one. The pinned production implementation accumulates fp32 partial weight gradients and reduces those partials afterward.

Python implementation

reference.py · weights
python
def weight_gradient(rows, weight, upstream, epsilon=1e-6, offset=0.0):
    """Sum the contributions to the weight shared by all token rows."""
    if not rows or len(rows) != len(upstream):
        raise ValueError("Input rows and upstream rows must match")
    partials = [backward(x, weight, dy, epsilon, offset)[1]
                for x, dy in zip(rows, upstream)]
    return [math.fsum(row[column] for row in partials)
            for column in range(len(weight))]

Original excerpts: Hsu et al., Liger Kernel, v3 (2025-01-24); originally submitted 2024-10-14, PDF page 4. Source PDF, crop bounds and image hashes are recorded in the source manifest. The independently written Python reference uses binary64 arithmetic.

Download the reference, CPU tests, and README. The example requires only the Python standard library. From the repository root, run:

code
python3 examples/liger-rmsnorm/reference.py python3 -m unittest discover -s examples/liger-rmsnorm -p 'test_*.py' -v

The checks cover known values, constant and zero rows, epsilon-sensitive scaling, input and learned-weight finite differences, and reduction of shared-weight contributions across multiple rows. They verify the authored mathematical reference. They do not run Liger, measure GPU performance, or reproduce training convergence.

What fusion establishes, and what needs measurement

The concrete opportunity is less materialized intermediate data around a row reduction. Its value depends on hidden size, number of rows, dtype, GPU, compiler, and whether the comparison baseline already fuses operations. A fair benchmark must identify those choices and separate forward-only, forward-plus-backward, and full training-step measurements. The full Liger library’s reported training results cannot be attributed to RMSNorm alone. Liger paper, numerical experiments

Kernel integration also has a gradient-lifetime constraint. The pinned interface can reuse the incoming gradient’s buffer for the input gradient. That is valid only when another consumer no longer needs the original contents; the source documents cases requiring the non-in-place option. A correct equation does not by itself make buffer reuse safe. Pinned autograd interface

RMSNorm supplies one row statistic and one shared feature scale. The next useful contrast is Liger’s fused SwiGLU, where the fused activation has no row reduction, but still has a deliberate memory and backward boundary. Online softmax develops the complementary idea of carrying a compact normalization state across a sequence of values.

If you found this paper review helpful, consider sharing it with others.

Mastodon