You launched 100k threads. They still run 32 at a time.
An A100 has 108 Streaming Multiprocessors. Your kernel grid is a sea of blocks. Every block that is currently live sits entirely on one SM — with that SM’s register file, shared memory, and four warp schedulers.
None of that looks like a CPU core. The SM does not speculate for one thread. It keeps dozens of warps resident and issues whoever is ready. Miss the warp size, the occupancy math, or the memory pattern and the hardware is mostly waiting — while the launch still “succeeds.”
This page is one SM: what is inside it, how work is packed, and the three taxes that show up in every slow kernel review.
Two numbers
1. 32
A warp is 32 threads with one instruction stream. Divergence and coalescing are both warp-shaped problems.
2. ~500 cycles
Round-trip to HBM when caches miss. The SM does not make memory faster — it hides that gap by switching warps. No spare warps → hollow idle cycles.
What’s on the floor
CUDA cores, Tensor Cores, schedulers, register file, shared/L1 — all in one workshop. Click a bay for the role that matters to software.
Click a bay. The floorplan stays; only the readout changes.
Warp schedulers
Four schedulers (typical recent SM). Each picks a warp with ready operands and issues. All warp state lives in the register file, so switching warps costs nothing — that is latency hiding.
4 / SM · 1–2 instr/issue · zero-cost switch
Schematic of an Ampere/Ada-class SM — not a die photo. Datacenter SKUs often omit RT cores; consumer RTX includes them. Counts vary by generation.
Threads, warps, blocks — one SM
A block never splits across SMs. Warps of 32 are the issue unit. Dial block size and how many blocks you try to park here; the stage recomposes the resident set.
A block never splits across SMs. Warps of 32 are the issue unit. Grid is many blocks; only a few live here at once.
- warps / block
- 8
- threads on SM
- 1024
- warps on SM
- 32 / 64
SIMT issues instructions to active lanes of a 32-thread warp. Volta+ retains per-thread program counters and call stacks; use explicit synchronization when threads exchange data. Different blocks do not share a block barrier.
Hierarchy in one line: thread → warp (32) → block (≤1024 threads, one SM) → grid (whole launch). Shared memory and __syncthreads are block-scoped. Different blocks on the same SM do not share a barrier.
Divergence is a throughput tax
Warp instructions issue with active-lane masks. In this equally populated, equal-cost teaching illustration, 32 serialized paths average 1/32 active lanes. This is not a universal measured slowdown: path cost and compiler predication matter. Volta+ retains per-thread PCs and call stacks; use explicit synchronization for communication.
Warp instructions execute with active-lane masks. Volta+ keeps per-thread PCs. This equal-cost illustration masks other paths off (amber idle); it does not imply implicit synchronization.
Authored equal-cost, equally populated paths: average active lanes = 1 / path count. Actual slowdown depends on path lengths, latencies and compiler predication; active-lane utilization alone is not a timer.
Structure data so threads in a warp stay on the same path when you can. When branching is unavoidable, push the split across warps, not inside one.
Latency is hidden by spare warps
Issue → memory stall → need another ready warp. With too few residents, the issue tape shows amber gaps. With enough, the stall disappears into other work.
Toy single-issue scheduler: issue one cycle, then 4-cycle memory stall. Real SMs have four schedulers and stalls of hundreds of cycles — same idea, bigger numbers.
Occupancy exists so this tape stays teal under real HBM latency (~hundreds of cycles). Not enough warps → units idle even if arithmetic intensity is fine.
Real stalls are hundreds of cycles, not four — which is why occupancy (warps waiting in the wings) is a first-class tuning knob, not a vanity metric.
Occupancy is three knobs, one SM
Registers per thread, shared memory per block, and threads per block all compete. One of them is usually the limiter. High occupancy helps memory-bound code hide latency; compute-bound kernels can live lower if each thread stays busy.
Registers, shared memory, and block size fight for the same SM. The amber gauge is the limiter right now.
A100/Hopper/Blackwell-class caps (2048 threads, 64 warps, 65,536 regs, ~96 KB usable shared teaching budget). Real occupancy also depends on allocation granularity — use Nsight / NVIDIA occupancy calculator for production numbers.
Loads that walk together
For an aligned base and 4-byte loads on NVIDIA CC 6.0+, 32 consecutive words touch four 32-byte sectors; a one-word offset touches five, and the demo’s unwrapped stride touches 32. These are analytical traffic counts, not bandwidth measurements. Shared-memory tiling exists so you pay global once and reuse on-chip.
Consecutive 4-byte addresses from lanes 0–31 collapse into few memory transactions. Scatter multiplies traffic for the same 32 values.
Authored traffic calculation for NVIDIA compute capability 6.0+: count distinct 32-byte sectors, irrespective of lane order. The four visual beats inspect lanes, not separate requests. Cache reuse and replay can change observed traffic; these are not bandwidth timings.
| Level | Scope | Latency (order of) | Role |
|---|---|---|---|
| Registers | per thread | ~0 | live locals |
| Shared / L1 | per block / SM | ~tens of cycles | reuse, cooperation |
| L2 | GPU-wide | ~hundreds | shared cache |
| HBM | GPU-wide | ~500 cycles | bulk store |
L1 and shared often share the same SRAM bank; the split is configurable. Prefer explicit shared tiles when reuse is regular; lean on L1 when access is irregular.
What to do
- Size blocks in multiples of 32 — usually 128–512 threads. Tiny blocks waste slots; max-size blocks fight register/shared limits.
- Keep warps uniform — same branch path inside a warp when performance matters.
- Budget occupancy for memory-bound kernels — enough warps to cover HBM latency; use the occupancy calculator / Nsight, not guesswork.
- Coalesce global loads — consecutive addresses per warp; tile through shared when you reuse.
- Treat registers as occupancy fuel — spilling or 96+ regs/thread often kills resident warps.
- Profile the SM, not the launch — Nsight Compute: warp stall reasons, memory throughput, achieved occupancy.
Generations (short)
The SIMT model stuck; specialized units grew.
| Gen | Era | Notable on the SM |
|---|---|---|
| Volta | 2017 | Tensor Cores, independent thread scheduling |
| Turing | 2018 | RT cores (consumer), concurrent FP32+INT32 |
| Ampere | 2020 | 128 FP32/SM class, larger shared/L1, sparse TC |
| Hopper / Ada | 2022 | more shared, FP8 / Transformer Engine paths |
| Blackwell | 2024 | denser TC formats (e.g. FP4 class) |
You still write for warps, occupancy, and the memory ladder. New silicon adds engines; it does not remove the taxes above.
Further reading
Related concepts
A fine kernel can still profile poorly: warp divergence taxes throughput ~1/N; alone on a long scoreboard wait, SM issue util can hit 0%. Interactive demos.
Master GPU memory hierarchy from registers to global memory, understand coalescing patterns, bank conflicts, and optimization strategies for maximum performance
Deep dive into the CUDA context object: control vs data plane, inventory (memory, modules, streams, events, graphs), push/pop/setCurrent stacks, primary retain/release, flags and limits, isolation, cost, and traps.
Decision map for CUDA: a context is per-process GPU state, a stream is an in-order queue inside a context, and MPS co-schedules work across client contexts on Volta+. Pick the layer that matches the problem.
Why ordinary multi-process CUDA time slicing can leave SMs idle, how Volta+ MPS co-schedules separate client contexts, execution-resource caps, and when to choose MPS or MIG.
A CUDA stream is an in-order queue of GPU ops. Overlap H→D, kernels, and D→H across streams — and avoid the legacy default-stream fences between blocking streams in one context.
