Skip to main content

GPU Pipeline Hazards: Scoreboard Stalls and Warp Divergence

A fine kernel can still profile poorly: warp divergence taxes throughput ~1/N; alone on a long scoreboard wait, SM issue util can hit 0%. Interactive demos.

The kernel is fine. The profile isn’t.

You’ve been here: the answers match the CPU, the tests are green, nothing is illegal. And yet the kernel is slow. Occupancy looks okay on paper. You’re not “out of FLOPs.” You open the profiler expecting to find a math bottleneck and get something blunter — the SM is waiting.

Two stories show up over and over.

Sometimes half the warp takes if and half takes else. You pictured free parallel SIMT. The machine runs one path, then the other. Idle lanes still sit there. Cost goes up; useful work per pass goes down.

Sometimes a warp issues a global load and needs the result for the next math. While it’s stuck, the SM has nobody else to run. Issue util drops to nothing until the data comes back. Same load with other ready warps? The wait barely shows up — someone else was always ready to issue.

Those are control and data hazards, GPU edition. The rest of this page is just those two stories with numbers you can quote and toys you can poke.

Two numbers

1. Divergence · throughput ≈ 1/N
For N equally populated, equal-cost paths, serialized path passes average 1/N active lanes: two-way → 50%, four-way → 25%. Relative to one uniform path of that same cost, this illustration issues N path bodies. It is not a general wall-clock slowdown formula: unequal path lengths, memory latency and compiler predication change the result.

2. Scoreboard · alone on a wait → util ~0%
A warp parked on a long load cannot issue dependent work. Alone on the SM, issue util can sit near 0% for hundreds of cycles. With other ready warps, the same wait is latency hiding — the SM keeps busy.

Flip the demos below until both numbers feel obvious.

Control hazard: warp divergence

Each color is a path group. Press Play and watch them take turns. Switch 4-way or 8-way and the useful bar falls toward 1/N.

Warp divergence
Step 1 / 3
50%
modeled lane utilization
2×
equal-cost path passes
lane 0lane 31
kernel · highlight = path running now
__global__ void k(float* a) {
int tid = threadIdx.x;
if (tid % 2 == 0) {
a[tid] += 1.0f; // even
} else {
a[tid] -= 1.0f; // odd
}
// reconverge
}

2 distinct paths in one warp. Equal-cost, equally populated teaching paths: average active lanes = 1/2.

Authored illustration: N equal-cost paths with equal lane groups give 1/N average active lanes and N path passes versus one uniform path. Real slowdown depends on path costs, latency and compiler predication; path0…pathN above stand for equal-cost bodies, not runnable definitions.

Fix: Prefer warp-uniform branches on hot paths. Put divergence across warps (or blocks), not inside a warp. On hot loops, prefer predication or a rewrite over data-dependent multi-way control when you can.

Data hazard: long scoreboard

Same load → use chain. The tape is SM issue each cycle. Flip 1 warp vs 3 warps: alone, the middle goes hollow (util 0% while W0 waits); with helpers, that gap fills in.

Scoreboard stall
Step 1 / 12
x = a[tid]loadscoreboardwaitb[tid] = x * 2uses x
0%
SM issue util while W0 waits on the load
full window 42%
cycle 0 · so far 100%
SM issue each cycle · hollow gap = alone on scoreboard
cycle 0cycle 11
W0
ready
only warp
W1
empty
no one home
W2
empty
no one home

Only W0 lives here. About to issue a load then depend on it.

The hollow amber gap is the whole lesson. Flip to 3 warps — same scoreboard wait on W0, util stays high because someone else can issue.

Fix: Break pure load→use chains so other warps (or independent work in the same warp) can issue. Raise useful occupancy for memory-bound kernels — not “max occupancy” as a religion.

Dictionary (CPU → GPU)

Same problem, two machines. Tap Control / Data / Structural — watch both sides move. The number in the middle is the GPU hit.

Same hazards · different machine
Same problem: Wrong next instruction for some work
CPU
Branch hazard
F
D
X
M
W
pipeline rolling…

Flush / bubble the pipe

~1/N
useful thr
GPU
Warp divergence
ifelse
one PC · about to split

Serialize paths in the warp

CPU does
Flush / bubble the pipe
GPU does
Serialize paths in the warp

Three old words. Three GPU numbers: ~1/N, 0% if alone, unit busy.

What to do

  1. Profile stall reasons (Nsight Compute) before rewriting math.
  2. Kill warp divergence on hot paths — or accept the 1/N tax deliberately.
  3. Break pure load→use chains so other warps can issue.
  4. Raise useful occupancy for memory-bound kernels.
  5. Coalesce so fewer long waits per useful byte.
GPU & High-Performance Computing
GPU Streaming Multiprocessor (SM)

How a single SM actually runs work: warps of 32, cutaway of cores and memory, divergence tax, latency-hiding occupancy, and coalesced loads — instruments, not a catalog.

GPU & High-Performance Computing
GPU Memory Hierarchy & Optimization

Master GPU memory hierarchy from registers to global memory, understand coalescing patterns, bank conflicts, and optimization strategies for maximum performance

Language & Framework Internals
Pinned Memory and DMA Transfers in PyTorch

How page-locked memory changes CPU-to-GPU transfers, the extra staging in pageable paths, and the conditions needed to overlap copies with GPU work.

GPU & High-Performance Computing
CUDA Contexts: Ownership, Current Stack, Isolation

Deep dive into the CUDA context object: control vs data plane, inventory (memory, modules, streams, events, graphs), push/pop/setCurrent stacks, primary retain/release, flags and limits, isolation, cost, and traps.

GPU & High-Performance Computing
CUDA Context vs Streams vs MPS: Which Layer Fixes What

Decision map for CUDA: a context is per-process GPU state, a stream is an in-order queue inside a context, and MPS co-schedules work across client contexts on Volta+. Pick the layer that matches the problem.

GPU & High-Performance Computing
CUDA Multi-Process Service (MPS): Concurrent Client Contexts

Why ordinary multi-process CUDA time slicing can leave SMs idle, how Volta+ MPS co-schedules separate client contexts, execution-resource caps, and when to choose MPS or MIG.

If you found this explanation helpful, consider sharing it with others.

Mastodon