Skip to main content

HPC Performance Optimization: Scaling, Profiling, and Tuning

Amdahl and Gustafson ceilings, strong vs weak scaling, roofline bounds, and hiding all-reduce behind compute — the levers that decide whether more GPUs actually buy science.

More GPUs. Same wall clock.

You doubled the allocation. The job did not get twice as fast. Nsight shows fat all-reduce bars, hollow SM gaps, and a data loader that still owns every step boundary. The cluster is fine. The schedule and the serial fraction are not.

HPC performance is not one trick. It is three questions in order: what is the theoretical ceiling, are you scaling the problem or just the hardware, and which roof are you actually hitting — memory, compute, or the network.

Two numbers

1. Amdahl · max speedup = 1/s
If 10% of the work never parallelizes, you cap at 10× even with infinite ranks. Cutting serial work from 10% → 5% doubles the ceiling. Buying another rack does not.

2. Overlap · compute should stay dense while collectives run
Blocking all-reduce puts the interconnect on the critical path. Async collectives (DDP buckets, MPI_Iallreduce) stack communication under the next layer’s backward pass — same idea as CUDA stream pipelines, one level up.

Flip the instruments until those feel mechanical.

The serial ceiling

Fixed problem size. More processors. How high can speedup go?

Here ss is serial time divided by the one-processor runtime of the fixed workload.

S(p) = \\frac{1}{s + \\frac{1-s}{p}}
Amdahl ceiling
10× max
ceiling = 1/s · s=10%
@p=16 6.4×
efficiency 40%
Speedup vs processors · dashed = 1/s ceiling

Fixed problem, more processors. Speedup@16 = 6.4× of a theoretical max 10× (1/s).

Infinite processors still stop at 1/s. Cutting serial work from 10% → 5% doubles the ceiling — hardware alone never will. Real clusters also pay communication Amdahl never models.

Strong vs weak scaling

Same cluster. Two different questions.

  • Strong — fixed global problem. Wall time should fall toward 1/p.
  • Weak — fixed work per rank. Wall time should stay flat.
Strong vs weak
10% eff
at p=32 · wall 31u · fixed problem
actual
ideal
Wall time vs ranks · should fall
1
2
4
8
16
32

Fixed global problem. Ideal time falls as 1/p. At 32 ranks wall ≈ 31 (vs ideal 3). Efficiency 10% — the amber lift is serial + communication.

Strong scaling answers: "same model, more GPUs, how much faster?" Curves flatten when serial + collectives eat the win.

Hide the network

Distributed training dies on communication latency long before single-kernel FLOPs run out. The backward pass is friendly: finish layer N grads, start all-reduce, keep computing layer N−1.

text
Layer N backward:   [==compute==]
Layer N all-reduce:              [===comm===]
Layer N-1 backward:              [==compute==]   ← overlaps
Comm–compute overlap
Step 1 / 13
50%
compute busy in this backward window · 0% of communication overlaps compute
single lane
t0
Backward pass · amber = all-reduce while compute idle
compute
all-reduce
Communication overlap pseudocode
for layer in reversed(layers):    grads = layer.backward(...)    all_reduce(grads)           # wait

t0: compute layer 1; its collective can start only after its gradients are ready.

Authored teaching model: 3 layers, 2 compute + 2 collective units each; one compute lane and one serialized collective lane. Both modes perform 6 compute and 6 collective units. 12 units to completion; 6 communication units are exposed. Each cell is one time unit; both modes use the same 12-cell axis. Real overlap depends on asynchronous progress and shared resource contention.

Blocking collectives put the network on the critical path. Every all-reduce is time the SMs are not doing useful backward work.

python
# PyTorch DDP does bucketing + async all-reduce for you.
# Manual MPI sketch of the same idea:
from mpi4py import MPI

req = comm.Iallreduce(MPI.IN_PLACE, grads_n, op=MPI.SUM)
# Keep grads_n alive and do not read or modify it until Wait completes.
grads_nm1 = layer_nm1.backward(...)  # independent work/buffer
req.Wait()
# grads_n now contains the reduced result.

The teaching tape keeps the same three layers and collective work in both modes, includes the last collective drain, and assumes communication progresses independently. Later compute can hide earlier communication, but the final ready gradient still has to be reduced; real runtimes also contend for GPU and memory resources.

Which roof are you under?

The roofline puts a kernel between peak bandwidth and peak FLOPs using operational intensity (FLOPs / byte). Most DL non-GEMM ops are memory-bound. Large matmuls sit under the compute roof. Multi-node jobs add a third roof: collectives.

Where to push
Bound
Low operational intensity

Few FLOPs per byte from HBM. You hit the bandwidth roof first — elementwise, LayerNorm, many attention parts.

↓ bytes
Move
What to do

Fuse kernels, reuse cache, cut traffic, better layouts. More ALUs alone will not move the needle.

Operational intensity (FLOPs/byte) places a kernel under the bandwidth roof or the compute roof. Distributed jobs add a third roof: the network. Optimize against the roof you are actually hitting.

For an interactive A100/H100 ridge plot, see HBM memory.

Load imbalance is a fake serial fraction

Even perfect math dies if ranks finish at different times:

  • Variable-length sequences without bucketing
  • Mixed GPU SKUs (fast ranks wait at every barrier)
  • Pipeline bubbles at fill and drain

Static partitions work when cost is uniform. Otherwise you need dynamic scheduling, sequence packing, or topology-aware placement — treat stragglers as serial work you invented.

Profile first

ToolJob
Nsight SystemsTimeline: gaps, H2D, kernels, NCCL
Nsight ComputeOne kernel: occupancy, stalls, roofline
VTune / perfHost: loader, threads, cache
bash
nsys profile --trace=cuda,nvtx,osrt,nccl python train.py
ncu --target-processes all -k regex:gemm python train.py

Five minutes of Systems often beats five days of kernel micro-opts on the wrong 2% bar.

What to do

  1. Measure the serial + sync fraction — if s is large, more GPUs are a tax.
  2. Pick strong or weak on purpose — report efficiency at the rank count you will run.
  3. Overlap collectives — async all-reduce / DDP buckets; do not block every layer.
  4. Name the roof — memory → fuse & cut traffic; compute → tensor cores & precision; network → overlap & topology.
  5. Profile before tuning — Systems for gaps, Compute for why, one change at a time.
bash
# Cheap first pass: is the GPU even busy?
nvidia-smi dmon -s u
# Then the timeline that tells the truth
nsys profile -o report --trace=cuda,nvtx,osrt,nccl python train.py

Further reading

If you found this explanation helpful, consider sharing it with others.

Mastodon