What changes in a mixture of experts?
A Transformer normally applies the same feed-forward network (FFN) to each token. A sparse mixture of experts (MoE) replaces that FFN with a bank of separate FFNs and a router. For each token, the router chooses a few experts, runs those FFNs and combines their outputs. Attention and other shared parts of the block still run normally.
Consider one token sent to two of eight experts. The model stores all eight expert parameter sets, but this token runs two. Those two outputs must still become one vector for the next layer. Start with that complete path:
Change the token to get a different fixed score distribution. Change Experts per token to see how selection, normalized weights and the mixed output change together. An “expert” is a parameter set; its name does not imply a human-defined domain such as mathematics or language.
From scores to a weighted output
For a token vector x, a linear router produces one logit per expert. Softmax turns the logits into probabilities:
In the renormalized top-k mixture shown above, select the k largest probabilities and divide them by their selected sum. Unselected experts have zero weight:
The figure uses small authored linear maps for E_i(x) so the result is inspectable. Real experts are wider nonlinear FFNs. Mixtral uses a renormalized top-2 mixture of SwiGLU experts; routing variants can use different gates. Mixtral, §2.1
What does k/N actually compare?
Let each equally sized expert store P parameters and cost F FFN work units per token. The expert bank stores N·P; top-k routing evaluates k·P expert parameters and costs approximately k·F expert work.
For eight experts with top-2 routing, 25% means one quarter of the work of evaluating all eight equal experts. Relative to one equally sized FFN, the expert work is 2×. It is not a prediction that a whole MoE model is four times faster than a dense model.
Increasing N at fixed k increases stored expert parameters without increasing the selected expert FFN work. However, the router produces more scores, and dispatch, communication, attention and other shared layers still cost work. Stored weights, optimizer state and capacity buffers also use memory. Comparing complete models requires naming their sizes, architecture, batch size, hardware and measured latency or throughput. The Mixtral report explicitly distinguishes active-parameter cost from memory and device utilization. Mixtral, §3
Capacity is a batch-level constraint
One token choosing two experts is a routing decision. Many tokens choosing the same expert is a scheduling problem: one expert may receive more work than its buffer can hold while another has unused slots.
The next illustration isolates top-1 dispatch for 128 tokens and eight experts. It compares an authored skew with round-robin assignment. It does not optimize a loss or reproduce trained routing. Each expert has:
Raising the capacity factor adds buffer space; it does not force the requested loads to become equal. Round-robin fits exactly at factor 1 here because 128 divides evenly across eight experts. A learned router must balance load while also selecting useful experts.
In Switch Transformer, tokens exceeding expert capacity skip that expert computation and continue through the block's residual path. “Dropped” refers to the expert dispatch, not deleting the token from the sequence. A capacity factor greater than 1 provides extra slots but does not guarantee zero overflow. Switch Transformer, §2.2
A load-balancing objective
Switch adds an auxiliary term to its training loss. For top-1 choices, f_i is the fraction of tokens whose highest-probability expert is i, before capacity clipping. P_i is the mean router probability for that expert across the batch:
The N factor keeps the value at α when both statistics are uniform. The discrete choices in f have no gradient; the P probabilities provide the differentiable path. This top-1 definition needs an explicit adaptation for other assignment schemes. Switch Transformer, equations 4–6
Published designs make different choices
These are architectural choices from the named papers, rather than a ranking of top-1 and top-2 quality:
The normalized top-1 option in the first figure has weight 1. It therefore differs from Switch's selected-probability output gate. In that normalized toy, a task loss alone has no router-weight gradient when k=1; a separate auxiliary loss can still update router probabilities.
GShard uses a top-2 routing design with group-level capacity, an auxiliary balancing objective and probabilistic second-expert dispatch. Its 600B translation model is a specific reported configuration. GShard, §2.2 and Table 1
Training stability also depends on the architecture and numerical precision. ST-MoE studies router z-loss, which penalizes large router log-normalizers, as one way to improve stability in its experiments. Initialization, clipping and noise are design choices to test; there is no universal expert-layer scaling or noise rule that guarantees stable training. ST-MoE, §3
A complete CPU example
This small PyTorch layer implements the renormalized mixture from the first figure with actual FFNs. It preserves [batch, sequence, d_model], passes the requested top_k to the router, runs only selected experts and accumulates weighted outputs. It has no capacity limit, distributed dispatch or pretrained weights.
The final helper separately computes the Switch-style top-1 auxiliary statistic. The first backward pass checks the mixture's task-loss path; the second checks that statistic's router-gradient path. This is a mechanics example, not a benchmark or a full Switch/Mixtral training recipe.
# src/pages/concepts/transformers/mixture-of-experts/index.mdx
import torch
from torch import nn
from torch.nn import functional as F
class Expert(nn.Module):
"""A two-layer feed-forward expert."""
def __init__(self, d_model, d_ff):
super().__init__()
self.ffn = nn.Sequential(
nn.Linear(d_model, d_ff, bias=False),
nn.ReLU(),
nn.Linear(d_ff, d_model, bias=False),
)
def forward(self, x):
return self.ffn(x)
class Router(nn.Module):
"""Select experts and normalize their mixture weights."""
def __init__(self, d_model, num_experts, top_k):
super().__init__()
if not 1 <= top_k <= num_experts:
raise ValueError("top_k must be between 1 and num_experts")
self.top_k = top_k
self.gate = nn.Linear(d_model, num_experts, bias=False)
def forward(self, x):
probs = F.softmax(self.gate(x), dim=-1)
selected_probs, indices = probs.topk(self.top_k, dim=-1)
weights = selected_probs / selected_probs.sum(dim=-1, keepdim=True)
return probs, indices, weights
class MoELayer(nn.Module):
"""Sparse renormalized mixture without a dispatch capacity limit."""
def __init__(self, d_model, d_ff, num_experts, top_k):
super().__init__()
self.experts = nn.ModuleList(
[Expert(d_model, d_ff) for _ in range(num_experts)]
)
self.router = Router(d_model, num_experts, top_k)
def forward(self, x):
x_flat = x.reshape(-1, x.shape[-1])
probs, indices, weights = self.router(x_flat)
output = torch.zeros_like(x_flat)
for expert_id, expert in enumerate(self.experts):
token_rows, slots = torch.where(indices == expert_id)
if token_rows.numel():
contribution = expert(x_flat[token_rows])
contribution = contribution * weights[token_rows, slots, None]
output = output.index_add(0, token_rows, contribution)
return output.reshape_as(x), probs, indices, weights
def switch_aux_loss(gate_probs, alpha=0.01):
"""Switch top-1 balance term, using choices before capacity clipping."""
num_experts = gate_probs.shape[-1]
choices = gate_probs.argmax(dim=-1)
fraction = F.one_hot(choices, num_experts).to(gate_probs.dtype).mean(dim=0)
mean_probability = gate_probs.mean(dim=0)
return alpha * num_experts * (fraction * mean_probability).sum()
torch.manual_seed(7)
x = torch.randn(2, 3, 4, device="cpu", requires_grad=True)
layer = MoELayer(d_model=4, d_ff=8, num_experts=4, top_k=2)
y, probs, indices, weights = layer(x)
task_loss = y.square().mean()
task_loss.backward()
print("Output shape:", tuple(y.shape)) # (2, 3, 4)
print("First token experts:", indices[0].tolist())
print("First token weights:", weights[0].detach().tolist())
print("Task router gradient:", layer.router.gate.weight.grad.norm().item())
# Fresh graph: a separately scoped top-1 auxiliary objective.
layer.zero_grad(set_to_none=True)
x.grad = None
fresh_probs, _, _ = layer.router(x.reshape(-1, 4))
aux_loss = switch_aux_loss(fresh_probs)
aux_loss.backward()
print("Top-1 balance term:", aux_loss.item())
print("Auxiliary router gradient:", layer.router.gate.weight.grad.norm().item())
For top_k=N, compare the result with the probability-weighted sum of every expert output. For top_k=1, verify the selected weight is 1 and the result is that expert's output. With top_k>1, gradients can flow through the normalized weights into the router, as well as through selected FFNs and the input. Top-k selection itself is discrete.
What to check in a real system
Use the same task and training budget when comparing model quality. For execution, inspect requested loads, overflow rates, stored and active parameters, peak memory, batch size, latency and throughput. If experts are sharded across devices, include dispatch and return communication: expert parallelism can trade memory distribution for network cost.
The three examples answer separate questions: which experts contribute to a token, whether a batch fits the dispatch capacity, and which denominator a compute fraction uses. Together they explain why MoE can add parameters without evaluating every expert on every token, while leaving training quality and measured performance as separate evidence.
Further Reading
- Switch Transformers Paper
- Mixtral Technical Report
- GShard: Scaling Giant Models
- ST-MoE: Designing Effective Sparse Expert Models
Related concepts
Compose sparse attention patterns, follow one query through masking and softmax, and separate fewer allowed pairs from actual sparse-kernel savings.
Follow a query-key distance through ALiBi head slopes, attention logits and normalized weights, then understand the evidence and limits of length extrapolation.
Compare head sharing with explicit projection dimensions and KV-cache bytes. Separate exact memory ratios from measured quality and latency, and see why FlashAttention is an independent choice.
See how preserving initial attention sinks stabilizes a rolling KV cache, why it costs recent slots, and why stable streaming does not mean infinite recall.
See how context capacity, attention masks, and KV caches differ. Move a planted fact through a token sequence to inspect what one query can access.
Trace a target query through source keys, attention weights, and a weighted value mix. Understand cross-attention origins, shapes, masks, and applications.
