<!-- examples/liger-rmsnorm/README.md -->

# Liger RMSNorm teaching reference

This example implements the real-valued forward and backward equations from
_Liger Kernel: Efficient Triton Kernels for LLM Training_, equations 1-2,
[arXiv:2410.10989v3](https://arxiv.org/abs/2410.10989v3). The paper was first
submitted in 2024; the selected revision is dated January 24, 2025. RMSNorm itself
was introduced by [Zhang and Sennrich (2019)](https://arxiv.org/abs/1910.07467).

```sh
python3 examples/liger-rmsnorm/reference.py
python3 -m unittest discover -s examples/liger-rmsnorm -p 'test_*.py' -v
```

The seven CPU tests cover known normalization values, zero rows, non-centering,
input and scale finite differences, shared weight reduction, epsilon behavior,
and invalid shapes and values. No external packages are required. Source regions
`normalize`, `forward`, `backward`, and `weights` feed the annotated paper page.
Edit this file's neighboring `reference.py`, then run
`node scripts/sync-paper-examples.mjs` to refresh excerpts and downloads.

The implementation uses Python binary64 arithmetic and `math.fsum`. It does not
model GPU execution, fp32 accumulation order, fp16/bf16 rounding, fused arithmetic,
or latency. Its gradients differentiate the real-valued RMSNorm formula, not a
quantizer. The offset parameter exposes the alternative `weight + 1` scale
parameterization without modeling a particular model's casting policy.

The accompanying review inspected the official Liger implementation at commit
[`37607058f981045df738fd719fc7b8062b25c183`](https://github.com/linkedin/Liger-Kernel/blob/37607058f981045df738fd719fc7b8062b25c183/src/liger_kernel/ops/rms_norm.py).
That code revision is distinct from the paper revision. This independently
written reference does not vendor the production kernel. No Liger GPU execution,
training convergence experiment, or performance benchmark is claimed.
