<!-- examples/liger-swiglu/README.md -->

# Liger fused SwiGLU teaching reference

Run `python3 examples/liger-swiglu/swiglu.py` from the repository root. Add
`--json` for deterministic output. Python 3.10 or later is sufficient; there are
no third-party dependencies.

The canonical regions `forward`, `backward`, and `traffic` are synchronized into
the paper page by `scripts/sync-paper-examples.mjs`. Edit this source before
regenerating excerpts or downloads.

The reference accepts two equally sized finite vectors of already projected
values. It computes SiLU(gate) times up and both input gradients. Its backward
recomputes SiLU instead of retaining an activation array. Dense projections,
autodiff integration, training, layout conversion, and GPU execution are outside
its scope. Unlike the inspected upstream two-input backward, this implementation
does not overwrite its input vectors.

The checks compare both derivatives against central finite differences of a
weighted scalar loss, exercise zero and negative gates, zero up values, large
finite inputs, and reject invalid vector lengths and nonfinite entries. Python
and browser arithmetic are binary64. They do not emulate FP16/BF16 casting or
establish GPU correctness or performance.

The traffic model counts five element transfers for a separate SiLU pass and
multiply pass versus three for a fused pointwise pass. It assumes each tensor
has the same storage dtype and each pass reads/writes it once. Caches, allocation,
packing, upstream/downstream projections, and backward are excluded. The one
SiLU tensor omitted from saved state is an incremental storage calculation, not
a measured reduction in peak memory.

Sources:

- [Liger Kernel, arXiv:2410.10989v3](https://arxiv.org/abs/2410.10989v3),
  Section 3.2, Equations 7–9; Section 4.1 reports benchmark scope.
- [Official two-input kernel](https://github.com/linkedin/Liger-Kernel/blob/37607058f981045df738fd719fc7b8062b25c183/src/liger_kernel/ops/swiglu.py).
- [Official MLP wrapper](https://github.com/linkedin/Liger-Kernel/blob/37607058f981045df738fd719fc7b8062b25c183/src/liger_kernel/transformers/swiglu.py).

Upstream code revision: `37607058f981045df738fd719fc7b8062b25c183`.
This is newer than the paper's v0.2.1 benchmark release. No paper benchmark is
reproduced by this CPU reference.
