The FLOPS are fine. The bus is not.
You bought the top SKU. Tensor Cores light up on paper. Token generation still crawls. The profiler is not lying about compute util — there simply are not enough bytes arriving from device memory.
HBM is DRAM moved into the package: stacks of thinned dies, vertical copper vias, a silicon interposer with a bus a thousand lanes wide. DDR is a highway to a distant warehouse. HBM builds the warehouse next to the factory.
This page is why that packaging exists, how the tower is wired, and when your kernel actually hits the wall.
Two numbers
1. ~106× vs ~11×
From P100 (2016) to B200 (2024), dense FP16 compute grew roughly two orders of magnitude. HBM bandwidth grew about one — and that is with stacked DRAM. The gap is the product problem.
2. ~1 FLOP/byte
LLM decode at batch 1: each token reads the weights once and does almost nothing with them. On an H100-class ridge near ~300 FLOP/B, that is well under 1% of peak compute. Inference speed is HBM bandwidth.
The wall keeps growing
Even with HBM, FLOPS outrun bytes. Scrub generations and watch the gap sticky climb.
FLOPS climbed ~106×; bandwidth ~11× — with HBM. Without stacks, the wall would be worse.
HBM is the third lever after more channels and faster pins: put DRAM on package and make the bus absurdly wide. The wall still grows — FLOPS outpace even stacked DRAM.
Three levers — HBM is width
More channels. Faster pins. Wider on-package bus. Flip the lever; the toy bus changes shape.
Bandwidth = width × rate × efficiency. HBM maximizes width and minimizes distance; pin-rate strategies maximize rate on a board.
HBM is lever three pushed to the end: if board SI is expensive, put memory in the package and spend packaging cost on width instead.
A thousand slow, short wires beat sixty-four fast, long ones when the job is terabytes per second next to a 700 W die.
What’s in the package
Stacks beside the GPU on an interposer — or GDDR chips out on the board. Click bays for the role of each layer.
Proximity is the product. HBM stacks ride the interposer beside the die; GDDR lives on the PCB with fewer, faster pins.
Interposer
Passive silicon layer under GPU and stacks. Makes a 1024–2048-bit bus routable at all. Everything important is millimeters apart.
1024b+ · sub-µm pitch · on-package only
Schematic, not a die photo. Stack counts and widths vary by product; the qualitative claim is proximity + width over board-level pin speed.
How the tower wires itself
Through-silicon vias: etch, insulate, copper-fill, thin to ~50 µm, micro-bump, stack. One bad via can scrap the tower — a large part of HBM’s price.
Cross-section of one die. Watch a hole become a copper column that punches through after the wafer is thinned — then bonds to the next die.
Step 1/8 · Finished DRAM wafer
Transistors on top, metal stack above them, ~700 µm of bulk silicon below. Signals can leave the die edges — nothing yet punches straight through.
Teaching cross-section with industry-typical sizes (via 5–10 µm, die thinned to ~50 µm, micro-bump ~25 µm pitch). Real flows vary by vendor; one open or shorted TSV can scrap the whole HBM tower — a big piece of the cost story.
| Parameter | Typical | Why it matters |
|---|---|---|
| TSV diameter | 5–10 µm | thousands fit without eating the array |
| Die after thin | ~50 µm | short vias, stackable height |
| Interface | 1024b (HBM3) / 2048b (HBM4) | bandwidth via width |
| Micro-bump pitch | ~25 µm | die-to-die density |
Generations (per stack, peaks)
| Gen | Era | Pin class | Interface | BW / stack | Cap / stack class |
|---|---|---|---|---|---|
| HBM / HBM2 | 2015–17 | 1–2 Gb/s | 1024b | 128–256 GB/s | 4–8 GB |
| HBM2E | 2020 | ~3.6 Gb/s | 1024b | ~461 GB/s | 16 GB |
| HBM3 | 2022 | ~6.4 Gb/s | 1024b | ~819 GB/s | 24 GB |
| HBM3E | 2024 | ~9.6 Gb/s | 1024b | ~1.2 TB/s | 36 GB |
| HBM4 | 2025–26 | 8+ Gb/s | 2048b | 2+ TB/s | 64 GB class |
Device bandwidth = stacks × bin rate. H100 SXM ~5× HBM3 → ~3.35 TB/s; B200 ~8× HBM3E → ~8 TB/s — both under raw JEDEC stack peaks. Read the datasheet, not only this table.
HBM is the bottom of the on-package ladder
Registers → shared/L1 → L2 → HBM. Everything above exists to avoid paying ~500 cycles and the interposer trip.
Everything above HBM exists to avoid touching it. Bandwidth is a budget; reuse is how you stay under it.
Stacked DRAM on the interposer — the bulk store for weights, activations, KV cache. Enormous by DRAM standards; still the bottleneck for most ML.
H100-class teaching numbers. Below HBM you leave the package (host, peer GPU, network) — another order-of-magnitude cliff. See multi-GPU communication for that slope.
Programmer rule: every byte from HBM should be reused as many times as possible from registers and shared memory before the next global load. Coalesce warps; tile through shared. (See Streaming Multiprocessor for warp occupancy and coalescing instruments.)
Roofline: is HBM your limit?
Arithmetic intensity (FLOP per byte from memory) places you under a roof: slanted memory limit, flat compute limit. The corner is the ridge.
Attainable = min(peak TFLOPS, AI × HBM TB/s). Drag the plot or scrub intensity. Left of the ridge is bandwidth; right is compute.
- GPU
- H100 SXM · HBM3
- peak (dense FP16/BF16)
- 989 TFLOPS
- HBM bandwidth
- 3.35 TB/s
- ridge
- 295 FLOP/B
- this point
- 3.35 TF · 0.34%
Teaching model: marketed dense TFLOPS and HBM BW. Real kernels sit under the roof (occupancy, TC path, cache hits, precision). Intensities are order-of-magnitude — profile with Nsight for the true bound.
Try LLM decode vs big GEMM on H100. Decode sits left of the ridge — buying more TFLOPS without bandwidth does nothing. GEMM can sit right if shapes and Tensor Core paths cooperate.
Programming habits (not a different API)
cudaMalloc already gave you HBM. Using it well is still:
- Coalesce — consecutive addresses per warp → full-width transactions.
- Tile and reuse — stage in shared memory; raise intensity toward the ridge.
- Profile the bound — Nsight Compute: memory throughput, achieved occupancy, not just “kernel finished.”
// Teaching sketch: coalesced tile load + reuse from shared (production: cuBLAS / CUTLASS)
__global__ void tiled_gemm(const float* A, const float* B, float* C, int M, int N, int K) {
const int TILE = 32;
__shared__ float As[TILE][TILE];
__shared__ float Bs[TILE][TILE];
int row = blockIdx.y * TILE + threadIdx.y;
int col = blockIdx.x * TILE + threadIdx.x;
float sum = 0.f;
for (int t = 0; t < K; t += TILE) {
As[threadIdx.y][threadIdx.x] =
(row < M && t + threadIdx.x < K) ? A[row * K + t + threadIdx.x] : 0.f;
Bs[threadIdx.y][threadIdx.x] =
(col < N && t + threadIdx.y < K) ? B[(t + threadIdx.y) * N + col] : 0.f;
__syncthreads();
#pragma unroll
for (int k = 0; k < TILE; k++)
sum += As[threadIdx.y][k] * Bs[k][threadIdx.x];
__syncthreads();
}
if (row < M && col < N) C[row * N + col] = sum;
}
Heat and cost (the real stack limits)
Heat. Eight to sixteen dies share one escape path into the same cooler as a 700 W+ GPU. DRAM leakage rises with temperature → more refresh → less useful bandwidth. Stacks throttle in the ~95 °C class; thermal design bounds height as much as process does.
Money. TSVs, interposers, and whole-tower yield make HBM far more expensive per GB than DDR or GDDR. The economics only close when bandwidth is the product — training and serving large models.
| Memory | Relative $/GB (order) | Bandwidth unit | Home |
|---|---|---|---|
| DDR5 | ~1× | tens of GB/s / channel | CPUs |
| GDDR7 | ~3–4× | ~100+ GB/s / chip | graphics |
| HBM3E | ~15–25× | ~1 TB/s / stack | AI / HPC |
Prices are approximate and move with AI demand.
What to do
- Measure intensity — is the kernel left or right of the ridge on your GPU?
- Raise reuse before buying a bigger SKU — tiling, fusion, FlashAttention-style fewer HBM trips.
- Coalesce and vectorize global loads; fix strided layouts first.
- Size for occupancy when memory-bound — spare warps hide HBM latency (SM page).
- Believe the datasheet BW and the profiler, not marketing TFLOPS alone, for decode-heavy work.
- Budget multi-GPU traffic separately — HBM is top of the cliff; NVLink and Ethernet are further drops (multi-GPU).
Further reading
- JEDEC HBM4 (JESD270-4) announcement
- What Every Programmer Should Know About Memory — Drepper; DRAM fundamentals under every stack
- NVIDIA Hopper Architecture In-Depth — HBM3 integration on a flagship GPU
Related concepts
Master GPU memory hierarchy from registers to global memory, understand coalescing patterns, bank conflicts, and optimization strategies for maximum performance
How page-locked memory changes CPU-to-GPU transfers, the extra staging in pageable paths, and the conditions needed to overlap copies with GPU work.
Structure of Arrays vs Array of Structures as instruments: cache-line fill, SIMD gather vs contiguous load, GPU coalescing, and AoSoA hybrids — when layout is a 10× decision.
Deep dive into the CUDA context object: control vs data plane, inventory (memory, modules, streams, events, graphs), push/pop/setCurrent stacks, primary retain/release, flags and limits, isolation, cost, and traps.
Decision map for CUDA: a context is per-process GPU state, a stream is an in-order queue inside a context, and MPS co-schedules work across client contexts on Volta+. Pick the layer that matches the problem.
Why ordinary multi-process CUDA time slicing can leave SMs idle, how Volta+ MPS co-schedules separate client contexts, execution-resource caps, and when to choose MPS or MIG.
