Skip to main content

High Bandwidth Memory (HBM)

How HBM feeds AI GPUs: the memory wall, width-over-speed packaging, TSVs and interposers, the on-package hierarchy, and the roofline that decides if your kernel is bandwidth-bound.

The FLOPS are fine. The bus is not.

You bought the top SKU. Tensor Cores light up on paper. Token generation still crawls. The profiler is not lying about compute util — there simply are not enough bytes arriving from device memory.

HBM is DRAM moved into the package: stacks of thinned dies, vertical copper vias, a silicon interposer with a bus a thousand lanes wide. DDR is a highway to a distant warehouse. HBM builds the warehouse next to the factory.

This page is why that packaging exists, how the tower is wired, and when your kernel actually hits the wall.

Two numbers

1. ~106× vs ~11×
From P100 (2016) to B200 (2024), dense FP16 compute grew roughly two orders of magnitude. HBM bandwidth grew about one — and that is with stacked DRAM. The gap is the product problem.

2. ~1 FLOP/byte
LLM decode at batch 1: each token reads the weights once and does almost nothing with them. On an H100-class ridge near ~300 FLOP/B, that is well under 1% of peak compute. Inference speed is HBM bandwidth.

The wall keeps growing

Even with HBM, FLOPS outrun bytes. Scrub generations and watch the gap sticky climb.

Memory wall · dense FP16 vs HBM BW
1.0× gap
compute growth ÷ bandwidth growth vs P100 · P100 (2016)

FLOPS climbed ~106×; bandwidth ~11× — with HBM. Without stacks, the wall would be worse.

1×
compute
1.0×
HBM BW
P100: dense FP16 peak ~21.2 TFLOPS, HBM ~0.732 TB/s. Gap vs P100 growth rates = 1.0× (teaching model, product peaks).

HBM is the third lever after more channels and faster pins: put DRAM on package and make the bus absurdly wide. The wall still grows — FLOPS outpace even stacked DRAM.

Three levers — HBM is width

More channels. Faster pins. Wider on-package bus. Flip the lever; the toy bus changes shape.

Three levers for bandwidth
lever
width
Wider bus, on-package · 1024–2048b · mm-scale · stack cost

Bandwidth = width × rate × efficiency. HBM maximizes width and minimizes distance; pin-rate strategies maximize rate on a board.

toy bus · each tick is a data wire32 shown
distance
mm (interposer)
per-pin rate
moderate
HBM: 1024–2048 bits of short, efficient wires on the interposer. A thousand slow, short wires beat sixty-four fast, long ones for AI bandwidth.

HBM is lever three pushed to the end: if board SI is expensive, put memory in the package and spend packaging cost on width instead.

A thousand slow, short wires beat sixty-four fast, long ones when the job is terabytes per second next to a 700 W die.

What’s in the package

Stacks beside the GPU on an interposer — or GDDR chips out on the board. Click bays for the role of each layer.

Package · HBM vs GDDR
layout
mm away
warehouse next to the factory · 1024–2048-bit class bus

Proximity is the product. HBM stacks ride the interposer beside the die; GDDR lives on the PCB with fewer, faster pins.

stack 0
stack 1

Interposer

Passive silicon layer under GPU and stacks. Makes a 1024–2048-bit bus routable at all. Everything important is millimeters apart.

1024b+ · sub-µm pitch · on-package only

Schematic, not a die photo. Stack counts and widths vary by product; the qualitative claim is proximity + width over board-level pin speed.

How the tower wires itself

Through-silicon vias: etch, insulate, copper-fill, thin to ~50 µm, micro-bump, stack. One bad via can scrap the tower — a large part of HBM’s price.

How a TSV is made
700 µm
thick bulk Si · no vertical path yet

Cross-section of one die. Watch a hole become a copper column that punches through after the wafer is thinned — then bonds to the next die.

die cross-section · front (circuits) upactive circuitsmetal layers (10–15)bulk silicon (~700 µm)700 µmsiliconcopper TSVliner / bumpsetched hole

Step 1/8 · Finished DRAM wafer

Transistors on top, metal stack above them, ~700 µm of bulk silicon below. Signals can leave the die edges — nothing yet punches straight through.

Teaching cross-section with industry-typical sizes (via 5–10 µm, die thinned to ~50 µm, micro-bump ~25 µm pitch). Real flows vary by vendor; one open or shorted TSV can scrap the whole HBM tower — a big piece of the cost story.

ParameterTypicalWhy it matters
TSV diameter5–10 µmthousands fit without eating the array
Die after thin~50 µmshort vias, stackable height
Interface1024b (HBM3) / 2048b (HBM4)bandwidth via width
Micro-bump pitch~25 µmdie-to-die density

Generations (per stack, peaks)

GenEraPin classInterfaceBW / stackCap / stack class
HBM / HBM22015–171–2 Gb/s1024b128–256 GB/s4–8 GB
HBM2E2020~3.6 Gb/s1024b~461 GB/s16 GB
HBM32022~6.4 Gb/s1024b~819 GB/s24 GB
HBM3E2024~9.6 Gb/s1024b~1.2 TB/s36 GB
HBM42025–268+ Gb/s2048b2+ TB/s64 GB class

Device bandwidth = stacks × bin rate. H100 SXM ~5× HBM3 → ~3.35 TB/s; B200 ~8× HBM3E → ~8 TB/s — both under raw JEDEC stack peaks. Read the datasheet, not only this table.

HBM is the bottom of the on-package ladder

Registers → shared/L1 → L2 → HBM. Everything above exists to avoid paying ~500 cycles and the interposer trip.

On-package hierarchy
~500 cyc
~500 cyc
HBM · whole GPU · 2–8 TB/s device

Everything above HBM exists to avoid touching it. Bandwidth is a budget; reuse is how you stay under it.

Stacked DRAM on the interposer — the bulk store for weights, activations, KV cache. Enormous by DRAM standards; still the bottleneck for most ML.

H100-class teaching numbers. Below HBM you leave the package (host, peer GPU, network) — another order-of-magnitude cliff. See multi-GPU communication for that slope.

Programmer rule: every byte from HBM should be reused as many times as possible from registers and shared memory before the next global load. Coalesce warps; tile through shared. (See Streaming Multiprocessor for warp occupancy and coalescing instruments.)

Roofline: is HBM your limit?

Arithmetic intensity (FLOP per byte from memory) places you under a roof: slanted memory limit, flat compute limit. The corner is the ridge.

Roofline · intensity dial
GPU
memory-bound
0.34% of peak · 3.35 TFLOPS · AI 1.0 FLOP/B · ridge ~295

Attainable = min(peak TFLOPS, AI × HBM TB/s). Drag the plot or scrub intensity. Left of the ridge is bandwidth; right is compute.

0.11101001k10k0.11101001k10karithmetic intensity (FLOP / byte)attainable TFLOPSridge ≈ 295 FLOP/B3.35 TFmemory roofcompute roof
At 1.0 FLOP/B you sit left of the ridge (~295). Bandwidth caps you at 3.35 TFLOPS (0.34% of 989 peak). LLM decode (~1 FLOP/B) is almost pure HBM traffic.
GPU
H100 SXM · HBM3
peak (dense FP16/BF16)
989 TFLOPS
HBM bandwidth
3.35 TB/s
ridge
295 FLOP/B
this point
3.35 TF · 0.34%

Teaching model: marketed dense TFLOPS and HBM BW. Real kernels sit under the roof (occupancy, TC path, cache hits, precision). Intensities are order-of-magnitude — profile with Nsight for the true bound.

Try LLM decode vs big GEMM on H100. Decode sits left of the ridge — buying more TFLOPS without bandwidth does nothing. GEMM can sit right if shapes and Tensor Core paths cooperate.

Programming habits (not a different API)

cudaMalloc already gave you HBM. Using it well is still:

  1. Coalesce — consecutive addresses per warp → full-width transactions.
  2. Tile and reuse — stage in shared memory; raise intensity toward the ridge.
  3. Profile the bound — Nsight Compute: memory throughput, achieved occupancy, not just “kernel finished.”
C++
// Teaching sketch: coalesced tile load + reuse from shared (production: cuBLAS / CUTLASS)
__global__ void tiled_gemm(const float* A, const float* B, float* C, int M, int N, int K) {
    const int TILE = 32;
    __shared__ float As[TILE][TILE];
    __shared__ float Bs[TILE][TILE];
    int row = blockIdx.y * TILE + threadIdx.y;
    int col = blockIdx.x * TILE + threadIdx.x;
    float sum = 0.f;
    for (int t = 0; t < K; t += TILE) {
        As[threadIdx.y][threadIdx.x] =
            (row < M && t + threadIdx.x < K) ? A[row * K + t + threadIdx.x] : 0.f;
        Bs[threadIdx.y][threadIdx.x] =
            (col < N && t + threadIdx.y < K) ? B[(t + threadIdx.y) * N + col] : 0.f;
        __syncthreads();
        #pragma unroll
        for (int k = 0; k < TILE; k++)
            sum += As[threadIdx.y][k] * Bs[k][threadIdx.x];
        __syncthreads();
    }
    if (row < M && col < N) C[row * N + col] = sum;
}

Heat and cost (the real stack limits)

Heat. Eight to sixteen dies share one escape path into the same cooler as a 700 W+ GPU. DRAM leakage rises with temperature → more refresh → less useful bandwidth. Stacks throttle in the ~95 °C class; thermal design bounds height as much as process does.

Money. TSVs, interposers, and whole-tower yield make HBM far more expensive per GB than DDR or GDDR. The economics only close when bandwidth is the product — training and serving large models.

MemoryRelative $/GB (order)Bandwidth unitHome
DDR5~1×tens of GB/s / channelCPUs
GDDR7~3–4×~100+ GB/s / chipgraphics
HBM3E~15–25×~1 TB/s / stackAI / HPC

Prices are approximate and move with AI demand.

What to do

  1. Measure intensity — is the kernel left or right of the ridge on your GPU?
  2. Raise reuse before buying a bigger SKU — tiling, fusion, FlashAttention-style fewer HBM trips.
  3. Coalesce and vectorize global loads; fix strided layouts first.
  4. Size for occupancy when memory-bound — spare warps hide HBM latency (SM page).
  5. Believe the datasheet BW and the profiler, not marketing TFLOPS alone, for decode-heavy work.
  6. Budget multi-GPU traffic separately — HBM is top of the cliff; NVLink and Ethernet are further drops (multi-GPU).

Further reading

GPU & High-Performance Computing
GPU Memory Hierarchy & Optimization

Master GPU memory hierarchy from registers to global memory, understand coalescing patterns, bank conflicts, and optimization strategies for maximum performance

Language & Framework Internals
Pinned Memory and DMA Transfers in PyTorch

How page-locked memory changes CPU-to-GPU transfers, the extra staging in pageable paths, and the conditions needed to overlap copies with GPU work.

Systems & Architecture
SoA vs AoS: Data Layout Optimization

Structure of Arrays vs Array of Structures as instruments: cache-line fill, SIMD gather vs contiguous load, GPU coalescing, and AoSoA hybrids — when layout is a 10× decision.

GPU & High-Performance Computing
CUDA Contexts: Ownership, Current Stack, Isolation

Deep dive into the CUDA context object: control vs data plane, inventory (memory, modules, streams, events, graphs), push/pop/setCurrent stacks, primary retain/release, flags and limits, isolation, cost, and traps.

GPU & High-Performance Computing
CUDA Context vs Streams vs MPS: Which Layer Fixes What

Decision map for CUDA: a context is per-process GPU state, a stream is an in-order queue inside a context, and MPS co-schedules work across client contexts on Volta+. Pick the layer that matches the problem.

GPU & High-Performance Computing
CUDA Multi-Process Service (MPS): Concurrent Client Contexts

Why ordinary multi-process CUDA time slicing can leave SMs idle, how Volta+ MPS co-schedules separate client contexts, execution-resource caps, and when to choose MPS or MIG.

If you found this explanation helpful, consider sharing it with others.

Mastodon