Skip to main content

GPU Streaming Multiprocessor (SM)

How a single SM actually runs work: warps of 32, cutaway of cores and memory, divergence tax, latency-hiding occupancy, and coalesced loads — instruments, not a catalog.

You launched 100k threads. They still run 32 at a time.

An A100 has 108 Streaming Multiprocessors. Your kernel grid is a sea of blocks. Every block that is currently live sits entirely on one SM — with that SM’s register file, shared memory, and four warp schedulers.

None of that looks like a CPU core. The SM does not speculate for one thread. It keeps dozens of warps resident and issues whoever is ready. Miss the warp size, the occupancy math, or the memory pattern and the hardware is mostly waiting — while the launch still “succeeds.”

This page is one SM: what is inside it, how work is packed, and the three taxes that show up in every slow kernel review.

Two numbers

1. 32
A warp is 32 threads with one instruction stream. Divergence and coalescing are both warp-shaped problems.

2. ~500 cycles
Round-trip to HBM when caches miss. The SM does not make memory faster — it hides that gap by switching warps. No spare warps → hollow idle cycles.

What’s on the floor

CUDA cores, Tensor Cores, schedulers, register file, shared/L1 — all in one workshop. Click a bay for the role that matters to software.

Inside one SM
4 issue
4 issue
pick ready warps every cycle · free context switch

Click a bay. The floorplan stays; only the readout changes.

Warp schedulers

Four schedulers (typical recent SM). Each picks a warp with ready operands and issues. All warp state lives in the register file, so switching warps costs nothing — that is latency hiding.

4 / SM · 1–2 instr/issue · zero-cost switch

Schematic of an Ampere/Ada-class SM — not a die photo. Datacenter SKUs often omit RT cores; consumer RTX includes them. Counts vary by generation.

Threads, warps, blocks — one SM

A block never splits across SMs. Warps of 32 are the issue unit. Dial block size and how many blocks you try to park here; the stage recomposes the resident set.

Thread → warp → block on one SM
threads/block
blocks/SM
32 warps
resident on this SM · cap 64 warps / 2048 threads (datacenter model)

A block never splits across SMs. Warps of 32 are the issue unit. Grid is many blocks; only a few live here at once.

SM · resident blocks
block 0
W0
W1
W2
W3
W4
W5
W6
W7
8 warps · 256 threads · shared mem + barriers stay here
block 1
W0
W1
W2
W3
W4
W5
W6
W7
8 warps · 256 threads · shared mem + barriers stay here
block 2
W0
W1
W2
W3
W4
W5
W6
W7
8 warps · 256 threads · shared mem + barriers stay here
block 3
W0
W1
W2
W3
W4
W5
W6
W7
8 warps · 256 threads · shared mem + barriers stay here
warps / block
8
threads on SM
1024
warps on SM
32 / 64

SIMT issues instructions to active lanes of a 32-thread warp. Volta+ retains per-thread program counters and call stacks; use explicit synchronization when threads exchange data. Different blocks do not share a block barrier.

Hierarchy in one line: thread → warp (32) → block (≤1024 threads, one SM) → grid (whole launch). Shared memory and __syncthreads are block-scoped. Different blocks on the same SM do not share a barrier.

Divergence is a throughput tax

Warp instructions issue with active-lane masks. In this equally populated, equal-cost teaching illustration, 32 serialized paths average 1/32 active lanes. This is not a universal measured slowdown: path cost and compiler predication matter. Volta+ retains per-thread PCs and call stacks; use explicit synchronization for communication.

Warp divergence
paths
50%
equal-cost lane utilization vs uniform · 2 serialized passes

Warp instructions execute with active-lane masks. Volta+ keeps per-thread PCs. This equal-cost illustration masks other paths off (amber idle); it does not imply implicit synchronization.

pass 1 / 2 · active path group 0serialized
Hardware runs path 0 for its lanes; the other 16 lanes sit masked in this equally populated illustration.

Authored equal-cost, equally populated paths: average active lanes = 1 / path count. Actual slowdown depends on path lengths, latencies and compiler predication; active-lane utilization alone is not a timer.

Structure data so threads in a warp stay on the same path when you can. When branching is unavoidable, push the split across warps, not inside one.

Latency is hidden by spare warps

Issue → memory stall → need another ready warp. With too few residents, the issue tape shows amber gaps. With enough, the stall disappears into other work.

Latency hiding · cycle tape
resident
BUSY
cycle 0 · util so far 100% · full-run 50%

Toy single-issue scheduler: issue one cycle, then 4-cycle memory stall. Real SMs have four schedulers and stalls of hundreds of cycles — same idea, bigger numbers.

issue tape · teal = unit busy · amber hollow = idle cycle
W0
issuing
W1
ready
With only 2 warps and a 4-cycle stall, the scheduler runs out of ready work — hollow amber gaps on the tape.

Occupancy exists so this tape stays teal under real HBM latency (~hundreds of cycles). Not enough warps → units idle even if arithmetic intensity is fine.

Real stalls are hundreds of cycles, not four — which is why occupancy (warps waiting in the wings) is a first-class tuning knob, not a vanity metric.

Occupancy is three knobs, one SM

Registers per thread, shared memory per block, and threads per block all compete. One of them is usually the limiter. High occupancy helps memory-bound code hide latency; compute-bound kernels can live lower if each thread stays busy.

Occupancy knobs
preset
75%
occupancy · 48/64 warps · 6 blocks · limited by shared mem

Registers, shared memory, and block size fight for the same SM. The amber gauge is the limiter right now.

warp slots on SM (64 max)
register file75%
shared memory · limiter100%
thread slots75%

A100/Hopper/Blackwell-class caps (2048 threads, 64 warps, 65,536 regs, ~96 KB usable shared teaching budget). Real occupancy also depends on allocation granularity — use Nsight / NVIDIA occupancy calculator for production numbers.

Loads that walk together

For an aligned base and 4-byte loads on NVIDIA CC 6.0+, 32 consecutive words touch four 32-byte sectors; a one-word offset touches five, and the demo’s unwrapped stride touches 32. These are analytical traffic counts, not bandwidth measurements. Shared-memory tiling exists so you pay global once and reuse on-chip.

Global load · coalescing
pattern
4 sectors
32-byte sectors · aligned base · 4-byte words

Consecutive 4-byte addresses from lanes 0–31 collapse into few memory transactions. Scatter multiplies traffic for the same 32 values.

warp lanes · beat 1/4 (lanes 0–7)
L00
L11
L22
L33
L44
L55
L66
L77
L88
L99
L1010
L1111
L1212
L1313
L1414
L1515
L1616
L1717
L1818
L1919
L2020
L2121
L2222
L2323
L2424
L2525
L2626
L2727
L2828
L2929
L3030
L3131
touched sectors · gaps omitted · hover for byte range
sector 0sector 1sector 2sector 3
Lanes map to consecutive addresses — four 32-byte sectors for 128 useful bytes.

Authored traffic calculation for NVIDIA compute capability 6.0+: count distinct 32-byte sectors, irrespective of lane order. The four visual beats inspect lanes, not separate requests. Cache reuse and replay can change observed traffic; these are not bandwidth timings.

LevelScopeLatency (order of)Role
Registersper thread~0live locals
Shared / L1per block / SM~tens of cyclesreuse, cooperation
L2GPU-wide~hundredsshared cache
HBMGPU-wide~500 cyclesbulk store

L1 and shared often share the same SRAM bank; the split is configurable. Prefer explicit shared tiles when reuse is regular; lean on L1 when access is irregular.

What to do

  1. Size blocks in multiples of 32 — usually 128–512 threads. Tiny blocks waste slots; max-size blocks fight register/shared limits.
  2. Keep warps uniform — same branch path inside a warp when performance matters.
  3. Budget occupancy for memory-bound kernels — enough warps to cover HBM latency; use the occupancy calculator / Nsight, not guesswork.
  4. Coalesce global loads — consecutive addresses per warp; tile through shared when you reuse.
  5. Treat registers as occupancy fuel — spilling or 96+ regs/thread often kills resident warps.
  6. Profile the SM, not the launch — Nsight Compute: warp stall reasons, memory throughput, achieved occupancy.

Generations (short)

The SIMT model stuck; specialized units grew.

GenEraNotable on the SM
Volta2017Tensor Cores, independent thread scheduling
Turing2018RT cores (consumer), concurrent FP32+INT32
Ampere2020128 FP32/SM class, larger shared/L1, sparse TC
Hopper / Ada2022more shared, FP8 / Transformer Engine paths
Blackwell2024denser TC formats (e.g. FP4 class)

You still write for warps, occupancy, and the memory ladder. New silicon adds engines; it does not remove the taxes above.

Further reading

GPU & High-Performance Computing
GPU Pipeline Hazards: Scoreboard Stalls and Warp Divergence

A fine kernel can still profile poorly: warp divergence taxes throughput ~1/N; alone on a long scoreboard wait, SM issue util can hit 0%. Interactive demos.

GPU & High-Performance Computing
GPU Memory Hierarchy & Optimization

Master GPU memory hierarchy from registers to global memory, understand coalescing patterns, bank conflicts, and optimization strategies for maximum performance

GPU & High-Performance Computing
CUDA Contexts: Ownership, Current Stack, Isolation

Deep dive into the CUDA context object: control vs data plane, inventory (memory, modules, streams, events, graphs), push/pop/setCurrent stacks, primary retain/release, flags and limits, isolation, cost, and traps.

GPU & High-Performance Computing
CUDA Context vs Streams vs MPS: Which Layer Fixes What

Decision map for CUDA: a context is per-process GPU state, a stream is an in-order queue inside a context, and MPS co-schedules work across client contexts on Volta+. Pick the layer that matches the problem.

GPU & High-Performance Computing
CUDA Multi-Process Service (MPS): Concurrent Client Contexts

Why ordinary multi-process CUDA time slicing can leave SMs idle, how Volta+ MPS co-schedules separate client contexts, execution-resource caps, and when to choose MPS or MIG.

GPU & High-Performance Computing
CUDA Streams: Asynchronous Execution and Concurrency

A CUDA stream is an in-order queue of GPU ops. Overlap H→D, kernels, and D→H across streams — and avoid the legacy default-stream fences between blocking streams in one context.

If you found this explanation helpful, consider sharing it with others.

Mastodon