The kernel is fine. The profile isn’t.
You’ve been here: the answers match the CPU, the tests are green, nothing is illegal. And yet the kernel is slow. Occupancy looks okay on paper. You’re not “out of FLOPs.” You open the profiler expecting to find a math bottleneck and get something blunter — the SM is waiting.
Two stories show up over and over.
Sometimes half the warp takes if and half takes else. You pictured free parallel SIMT. The machine runs one path, then the other. Idle lanes still sit there. Cost goes up; useful work per pass goes down.
Sometimes a warp issues a global load and needs the result for the next math. While it’s stuck, the SM has nobody else to run. Issue util drops to nothing until the data comes back. Same load with other ready warps? The wait barely shows up — someone else was always ready to issue.
Those are control and data hazards, GPU edition. The rest of this page is just those two stories with numbers you can quote and toys you can poke.
Two numbers
1. Divergence · throughput ≈ 1/N
For N equally populated, equal-cost paths, serialized path passes average 1/N active lanes: two-way → 50%, four-way → 25%. Relative to one uniform path of that same cost, this illustration issues N path bodies. It is not a general wall-clock slowdown formula: unequal path lengths, memory latency and compiler predication change the result.
2. Scoreboard · alone on a wait → util ~0%
A warp parked on a long load cannot issue dependent work. Alone on the SM, issue util can sit near 0% for hundreds of cycles. With other ready warps, the same wait is latency hiding — the SM keeps busy.
Flip the demos below until both numbers feel obvious.
Control hazard: warp divergence
Each color is a path group. Press Play and watch them take turns. Switch 4-way or 8-way and the useful bar falls toward 1/N.
2 distinct paths in one warp. Equal-cost, equally populated teaching paths: average active lanes = 1/2.
Authored illustration: N equal-cost paths with equal lane groups give 1/N average active lanes and N path passes versus one uniform path. Real slowdown depends on path costs, latency and compiler predication; path0…pathN above stand for equal-cost bodies, not runnable definitions.
Fix: Prefer warp-uniform branches on hot paths. Put divergence across warps (or blocks), not inside a warp. On hot loops, prefer predication or a rewrite over data-dependent multi-way control when you can.
Data hazard: long scoreboard
Same load → use chain. The tape is SM issue each cycle. Flip 1 warp vs 3 warps: alone, the middle goes hollow (util 0% while W0 waits); with helpers, that gap fills in.
Only W0 lives here. About to issue a load then depend on it.
The hollow amber gap is the whole lesson. Flip to 3 warps — same scoreboard wait on W0, util stays high because someone else can issue.
Fix: Break pure load→use chains so other warps (or independent work in the same warp) can issue. Raise useful occupancy for memory-bound kernels — not “max occupancy” as a religion.
Dictionary (CPU → GPU)
Same problem, two machines. Tap Control / Data / Structural — watch both sides move. The number in the middle is the GPU hit.
Flush / bubble the pipe
Serialize paths in the warp
Three old words. Three GPU numbers: ~1/N, 0% if alone, unit busy.
What to do
- Profile stall reasons (Nsight Compute) before rewriting math.
- Kill warp divergence on hot paths — or accept the 1/N tax deliberately.
- Break pure load→use chains so other warps can issue.
- Raise useful occupancy for memory-bound kernels.
- Coalesce so fewer long waits per useful byte.
Related concepts
How a single SM actually runs work: warps of 32, cutaway of cores and memory, divergence tax, latency-hiding occupancy, and coalesced loads — instruments, not a catalog.
Master GPU memory hierarchy from registers to global memory, understand coalescing patterns, bank conflicts, and optimization strategies for maximum performance
How page-locked memory changes CPU-to-GPU transfers, the extra staging in pageable paths, and the conditions needed to overlap copies with GPU work.
Deep dive into the CUDA context object: control vs data plane, inventory (memory, modules, streams, events, graphs), push/pop/setCurrent stacks, primary retain/release, flags and limits, isolation, cost, and traps.
Decision map for CUDA: a context is per-process GPU state, a stream is an in-order queue inside a context, and MPS co-schedules work across client contexts on Volta+. Pick the layer that matches the problem.
Why ordinary multi-process CUDA time slicing can leave SMs idle, how Volta+ MPS co-schedules separate client contexts, execution-resource caps, and when to choose MPS or MIG.
