TL;DR
- Liger fuses SiLU and elementwise multiplication on already projected inputs; the surrounding matrix multiplications remain separate.
- The fused forward removes the separate SiLU tensor, reducing modeled logical traffic from 5Nq to 3Nq bytes for the stated two-pass baseline.
- Backward recomputes SiLU from saved inputs to produce both input gradients, trading arithmetic for less retained activation state.
- The paper reports SwiGLU speed parity and lower peak memory on its A100 setup; dtype, buffer ownership, and the measured workload still determine whether a concrete integration is useful.
The small operation between large projections
A transformer feed-forward block spends much of its arithmetic on matrix multiplication. Between those large operations, it may also create an activation tensor only to multiply it by another tensor immediately afterward. The activation is cheap to compute, but storing it, reading it again, and keeping it for backward can be expensive.
Liger's fused SwiGLU targets this middle step: apply SiLU to one already projected input, then multiply it elementwise by the other. The backward pass can reconstruct SiLU from saved inputs instead of retaining a separate SiLU activation. The useful question is which intermediate disappears, which values remain live, and what that changes about training memory.
This is a focused reading of the SwiGLU kernel in Liger Kernel: Efficient Triton Kernels for LLM Training, first posted in 2024. The equations and source excerpts use the January 2025 v3 paper. The companion fused RMSNorm review examines a reduction kernel from the same paper. Here every output coordinate is independent, so there is no mean, norm, softmax denominator, or communication across channels inside the pointwise operation.
Start with the fusion boundary
Write the input to a feed-forward block as X with shape [T, D], where T combines batch and sequence positions. Two learned projections produce a gate tensor A and an up tensor B, both [T, H]. Biases, when present, belong to those projections. A downstream projection maps the gated result back to the model width D:
The dense projections are outside the fusion considered here. In the pinned Liger MLP wrapper, the gate and up linear layers run before the custom SiLU-and-multiply function, and the down linear layer runs afterward. A packed gate/up projection is another possible layout, but it does not make the elementwise kernel a fused matrix multiplication.
For one coordinate, rename the two projected values a and b. Then the entire pointwise calculation is:
The sigmoid factor is between zero and one. The full SiLU gate is not: it includes a, so it can be negative and grows approximately like a for large positive inputs. In particular, a negative gate is not a Boolean “off” switch. This distinction also explains the surprising gradient behavior later.
The gate can be negative
Play the projected gate through sigmoid and SiLU, then watch the up branch join the product. Change either input to move the curve marker and restart the signal path.
−24 ← zero → +24 · changing either input moves the signed output
Start with two projected values
The gate input a and up input b already exist. The projection matrix multiplications are outside this sequence.
SiLU(a) = a · σ(a)
gate a
Sigmoid σ(a)
0.11920
Between zero and one.
SiLU(a) = a · σ(a)
-0.23841
The full gate includes a, so its range is not [0, 1].
Output y = SiLU(a) · b
-0.47681
a = -2, b = 2: output -0.47681. A negative gate produces a negative output for this positive up value.
What fusion removes from the forward pass
Consider a baseline that already implements SiLU in one kernel and then launches a separate multiplication. For N = TH elements, its first kernel reads A and writes S = SiLU(A). Its second kernel reads S and B and writes Y. With the same storage width q for every tensor, that is five logical element transfers, or 5Nq bytes.
A fused pointwise pass reads A and B, computes the sigmoid and products locally, then writes Y. The SiLU intermediate never needs its own global-memory tensor. This model counts 3Nq bytes, eliminating one intermediate write and one read. That is a 40% reduction in the modeled forward traffic; it does not imply a 40% latency reduction or a fixed speedup.
Remove one intermediate round trip
Follow the packets between global memory and the pointwise kernels. The separate plan writes and rereads S; the fused plan keeps it local.
Storage width (H = 4096)
Global memory
0 transfers per element completed · shared scale: 5
Two kernels, one intermediate tensor
A and B start in global memory. S has not been computed yet. Cumulative logical traffic: 0 MiB.
Two-kernel forward total
40 MiB
Read a; write s; read s and b; write y.
Fused forward total
24 MiB
Read a and b; write y.
Avoided write + read
16 MiB
40% of this model's forward traffic.
Recomputing SiLU in backward can also avoid retaining one 8 MiB SiLU tensor. The projected a and b inputs are still needed. This separate saved-state count is not added to the forward traffic above.
Several boundaries matter when interpreting that count. The projected A and B tensors already exist, and the output Y must still reach the downstream projection. Cache reuse can change physical memory traffic. A compiler may already fuse the baseline. Layout conversion can add copies, and a very small workload can be dominated by launch overhead. The figure isolates a specific two-pass versus one-pass calculation so its accounting stays inspectable.
Forward fusion also creates an opportunity for backward: if SiLU is cheap enough to recompute, there is no need to keep S alive merely to differentiate the product. To see why, follow the two gradient branches.
Backward sends a different signal to each input
Let g be the incoming loss gradient ∂L/∂y. The up input b appears linearly, so its gradient is g times SiLU(a). The gate input a needs the product rule inside SiLU:
These are elementwise vector-Jacobian products when a, b, and g become tensors. The elementwise backward supplies gradients to the surrounding linear layers; those layers still compute their own input and weight gradients through matrix operations.
Two simple inputs reveal why the branches cannot be treated as interchangeable. At a = 0, the output is zero, but SiLU's local slope is ½, so the gate gradient can be nonzero. At b = 0, the gate gradient vanishes, while the up gradient can remain nonzero. Negative a can also give a negative SiLU slope: “sigmoid is positive” does not prove that the derivative of a times sigmoid is positive.
A zero output can still carry a gradient
Play g backward through the product and follow its two branches. The loss is L = g · y; the signed bars reveal the two terms in the gate derivative.
Gate branch: waiting for g
Up branch: waiting for g
SiLU′: add these two signed contributions
Negative ← zero → positive · common scale for both bars
Keep the projected inputs
Backward has saved a and b. The incoming gradient g tells us how changing y changes the loss.
Gate branch ∂L/∂a
g · b · [σ(a) + SiLU(a) · (1 − σ(a))]
- Analytic
- -0.1815685
- Finite difference
- -0.1815685
- Absolute error
- < 10⁻⁷
Up branch ∂L/∂b
g · SiLU(a)
- Analytic
- -0.2384058
- Finite difference
- -0.2384058
- Absolute error
- < 10⁻⁷
Output y = -0.47681. The local SiLU slope is -0.09078; both b and g determine the gate gradient's sign.
Recomputation uses the saved projected inputs A and B together with the incoming gradient. It evaluates sigmoid and SiLU again, then emits both gradients in one pointwise backward. This removes the need for a separately saved S tensor; it does not remove every saved activation or guarantee an identical reduction in the model's peak allocation. Peak memory depends on which other tensors are alive at the same time.
Read the source beside the equations
The rows below connect original PDF excerpts to the equations and a runnable CPU reference. The reference uses equal-length flattened vectors and binary64 arithmetic. It provides an independent way to check the mathematical boundary, including both derivatives and the logical traffic count. It does not train a network or execute a GPU kernel.
1. Fuse the gate activation and elementwise product
Original paper
Equations 7 and 8 distinguish the projected inputs from the pointwise kernel. The linked implementation uses one common two-input path plus hardware-specific variants.
Math and tensor shapes
The CPU reference accepts flattened projected vectors of equal length. Its stable sigmoid avoids a large positive exponential. It intentionally uses binary64 and does not emulate the upstream forward cast of SiLU to the up-input dtype. The scalar loop is an arithmetic reference, not GPU fusion.
Python implementation
def sigmoid(value):
"""Evaluate sigmoid without exponentiating a large positive number."""
if value >= 0:
return 1.0 / (1.0 + math.exp(-value))
exp_value = math.exp(value)
return exp_value / (1.0 + exp_value)
def swiglu(gate, up):
"""Map two projected vectors to SiLU(gate) * up, element by element."""
_validate_vectors(gate, up)
return [a * sigmoid(a) * b for a, b in zip(gate, up)]2. Recompute SiLU and send a gradient to each branch
Original paper
Equation 9 gives both vector-Jacobian products. The linked two-input backward recomputes SiLU from saved projected inputs.
Math and tensor shapes
The bar denotes an incoming loss gradient, not a new activation. No cross-coordinate reduction is needed. This reference returns fresh lists; the inspected two-input Triton path overwrites its saved a and b storage with gradients. That reuse has an ownership contract and is not a general permission to mutate shared activations.
Python implementation
def swiglu_backward(gate, up, upstream):
"""Recompute SiLU from saved inputs; return both vector-Jacobian products."""
_validate_vectors(gate, up, upstream)
d_gate, d_up = [], []
for a, b, gradient in zip(gate, up, upstream):
probability = sigmoid(a)
silu = a * probability
slope = probability + silu * (1.0 - probability)
d_gate.append(gradient * b * slope)
d_up.append(gradient * silu)
return d_gate, d_up3. Count the intermediate that fusion removes
Original paper
Equation 8 defines the boundary used for this authored traffic calculation. The paper does not report these counts as measured bandwidth or a speedup.
Math and tensor shapes
q is storage bytes per element. The two-pass baseline reads a, writes s, reads s and b, then writes y. The fused pass reads a and b and writes y. Backward recomputation can separately remove one Nq-byte saved SiLU activation; it still requires a and b. Actual peak allocation and latency require measurement.
Python implementation
def traffic(tokens, hidden, bytes_per_element=2):
"""Count logical forward bytes for two passes versus one fused pass.
Excludes projections, backward, caching, copies, and launch overhead.
The saved activation size is incremental storage, not peak GPU memory.
"""
if any(type(v) is not int or v <= 0 for v in (tokens, hidden)):
raise ValueError("Expected positive integer dimensions")
if bytes_per_element not in (2, 4):
raise ValueError("Expected 2 or 4 storage bytes per element")
tensor_bytes = tokens * hidden * bytes_per_element
return {
"separateBytes": 5 * tensor_bytes,
"fusedBytes": 3 * tensor_bytes,
"avoidedBytes": 2 * tensor_bytes,
"savedSiluBytes": tensor_bytes,
}Original excerpts: Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, Yanning Chen; v3 (2025-01-24); first preprint 2024-10-14. Source crops preserve original colors and are attributed under CC BY 4.0. PDF and crop hashes, page rectangles, and rendering scale are recorded in the source manifest. The Python is an independently authored teaching reference.
Run swiglu.py with Python 3.10 or later. The README records the scope; requirements.txt confirms that no external packages are needed. Add --json for the deterministic values used by the browser-model checks. The self-check compares both analytical derivatives with central finite differences of a weighted scalar loss, exercises zero and negative gates, and verifies stable sigmoid behavior at large finite inputs.
Shapes, precision, and ownership are part of the kernel
The equations specify matching [T, H] inputs. They do not specify strides, alignment, storage dtype, or who owns a saved tensor. Those details decide whether a concrete kernel can be substituted safely.
The inspected two-input implementation flattens leading dimensions into rows, masks padded columns, and promotes the gate calculation to FP32. In its forward path, SiLU is cast to the up-input dtype before multiplication. Its backward recomputes the activation and writes gradients into the saved input buffers. These are implementation choices at the pinned revision, which is newer than the paper's benchmark release.
This review follows the default gate_multiplier = down_multiplier = 1 path. A nonunit gate multiplier changes the sigmoid argument and adds a chain-rule factor to the gate gradient; a nonunit down multiplier introduces additional output scaling. The displayed equations and 3Nq forward traffic model cover the default path.
Intermediate casts mean mathematical equivalence does not promise bitwise equality across eager, compiled, and custom-kernel paths. Validate forward and backward separately against a reference that matches the intended rounding convention. The binary64 explorer tests the smooth equation; it does not model a finite difference through quantization steps.
Likewise, overwriting a saved buffer requires knowing that its previous contents are no longer needed. Shared aliases, repeated backward calls, and checkpointing can change that lifetime. The teaching reference returns new lists so the derivative exercise does not inherit those mutation assumptions. In a real integration, inspect the wrapper and test the actual autograd usage as well as the standalone formula.
What the paper's measurements establish
The relevant result is more specific than “fusion is faster.” Section 4.1 evaluates Liger-Kernel v0.2.1 on one NVIDIA A100 with 80 GB, varying SwiGLU sequence length over 4096, 8192, 12288, and 16384. The paper reports speed parity with the baseline for SwiGLU and GeGLU, while peak memory at length 16384 is approximately 1.6 times lower through activation recomputation. It reports medians over ten repeats, with 0.2 and 0.8 quantiles. These are paper-reported measurements, not measurements from this page. Paper, §4.1 and Figures 2c/3c
That result fits the mechanism: reducing retained state can be useful even when runtime is similar. The paper's end-to-end training improvements combine several kernels and cannot be assigned to SwiGLU alone. A current performance comparison would need a pinned implementation and baseline, matching shapes and precision, explicit forward/backward timing boundaries, and measured peak allocation. The logical byte calculation above supplies a hypothesis to investigate, not a substitute for those measurements.
The useful pattern to carry forward
Start by drawing the operation boundary. Identify the projected inputs that must exist, the local values that can be recomputed, and the outputs required by the next operation. Then derive both gradient paths before deciding which activations need to survive the forward pass. SwiGLU makes this reasoning unusually clear because the pointwise operation has no cross-coordinate dependency.
The fused RMSNorm review adds a reduction and asks which row statistics should be retained. Online softmax takes the next step: maintaining a changing normalizer without storing all intermediate state. For the broader performance framework, Making Deep Learning Go Brrrr connects these memory and recomputation choices to the bottleneck you actually measure.
