More GPUs. Same wall clock.
You doubled the allocation. The job did not get twice as fast. Nsight shows fat all-reduce bars, hollow SM gaps, and a data loader that still owns every step boundary. The cluster is fine. The schedule and the serial fraction are not.
HPC performance is not one trick. It is three questions in order: what is the theoretical ceiling, are you scaling the problem or just the hardware, and which roof are you actually hitting — memory, compute, or the network.
Two numbers
1. Amdahl · max speedup = 1/s
If 10% of the work never parallelizes, you cap at 10× even with infinite ranks. Cutting serial work from 10% → 5% doubles the ceiling. Buying another rack does not.
2. Overlap · compute should stay dense while collectives run
Blocking all-reduce puts the interconnect on the critical path. Async collectives (DDP buckets, MPI_Iallreduce) stack communication under the next layer’s backward pass — same idea as CUDA stream pipelines, one level up.
Flip the instruments until those feel mechanical.
The serial ceiling
Fixed problem size. More processors. How high can speedup go?
Here is serial time divided by the one-processor runtime of the fixed workload.
Fixed problem, more processors. Speedup@16 = 6.4× of a theoretical max 10× (1/s).
Infinite processors still stop at 1/s. Cutting serial work from 10% → 5% doubles the ceiling — hardware alone never will. Real clusters also pay communication Amdahl never models.
Strong vs weak scaling
Same cluster. Two different questions.
- Strong — fixed global problem. Wall time should fall toward 1/p.
- Weak — fixed work per rank. Wall time should stay flat.
Fixed global problem. Ideal time falls as 1/p. At 32 ranks wall ≈ 31 (vs ideal 3). Efficiency 10% — the amber lift is serial + communication.
Strong scaling answers: "same model, more GPUs, how much faster?" Curves flatten when serial + collectives eat the win.
Hide the network
Distributed training dies on communication latency long before single-kernel FLOPs run out. The backward pass is friendly: finish layer N grads, start all-reduce, keep computing layer N−1.
Layer N backward: [==compute==]
Layer N all-reduce: [===comm===]
Layer N-1 backward: [==compute==] ← overlaps
for layer in reversed(layers): grads = layer.backward(...) all_reduce(grads) # waitt0: compute layer 1; its collective can start only after its gradients are ready.
Authored teaching model: 3 layers, 2 compute + 2 collective units each; one compute lane and one serialized collective lane. Both modes perform 6 compute and 6 collective units. 12 units to completion; 6 communication units are exposed. Each cell is one time unit; both modes use the same 12-cell axis. Real overlap depends on asynchronous progress and shared resource contention.
Blocking collectives put the network on the critical path. Every all-reduce is time the SMs are not doing useful backward work.
# PyTorch DDP does bucketing + async all-reduce for you.
# Manual MPI sketch of the same idea:
from mpi4py import MPI
req = comm.Iallreduce(MPI.IN_PLACE, grads_n, op=MPI.SUM)
# Keep grads_n alive and do not read or modify it until Wait completes.
grads_nm1 = layer_nm1.backward(...) # independent work/buffer
req.Wait()
# grads_n now contains the reduced result.
The teaching tape keeps the same three layers and collective work in both modes, includes the last collective drain, and assumes communication progresses independently. Later compute can hide earlier communication, but the final ready gradient still has to be reduced; real runtimes also contend for GPU and memory resources.
Which roof are you under?
The roofline puts a kernel between peak bandwidth and peak FLOPs using operational intensity (FLOPs / byte). Most DL non-GEMM ops are memory-bound. Large matmuls sit under the compute roof. Multi-node jobs add a third roof: collectives.
Few FLOPs per byte from HBM. You hit the bandwidth roof first — elementwise, LayerNorm, many attention parts.
Fuse kernels, reuse cache, cut traffic, better layouts. More ALUs alone will not move the needle.
Operational intensity (FLOPs/byte) places a kernel under the bandwidth roof or the compute roof. Distributed jobs add a third roof: the network. Optimize against the roof you are actually hitting.
For an interactive A100/H100 ridge plot, see HBM memory.
Load imbalance is a fake serial fraction
Even perfect math dies if ranks finish at different times:
- Variable-length sequences without bucketing
- Mixed GPU SKUs (fast ranks wait at every barrier)
- Pipeline bubbles at fill and drain
Static partitions work when cost is uniform. Otherwise you need dynamic scheduling, sequence packing, or topology-aware placement — treat stragglers as serial work you invented.
Profile first
| Tool | Job |
|---|---|
| Nsight Systems | Timeline: gaps, H2D, kernels, NCCL |
| Nsight Compute | One kernel: occupancy, stalls, roofline |
| VTune / perf | Host: loader, threads, cache |
nsys profile --trace=cuda,nvtx,osrt,nccl python train.py
ncu --target-processes all -k regex:gemm python train.py
Five minutes of Systems often beats five days of kernel micro-opts on the wrong 2% bar.
What to do
- Measure the serial + sync fraction — if s is large, more GPUs are a tax.
- Pick strong or weak on purpose — report efficiency at the rank count you will run.
- Overlap collectives — async all-reduce / DDP buckets; do not block every layer.
- Name the roof — memory → fuse & cut traffic; compute → tensor cores & precision; network → overlap & topology.
- Profile before tuning — Systems for gaps, Compute for why, one change at a time.
# Cheap first pass: is the GPU even busy?
nvidia-smi dmon -s u
# Then the timeline that tells the truth
nsys profile -o report --trace=cuda,nvtx,osrt,nccl python train.py
Further reading
- Roofline model (Berkeley Lab)
- NVIDIA Nsight Systems
- Scalability — But at What COST? — McSherry et al.
- HPC Wiki — Performance profiling
Related concepts
Master GPU memory hierarchy from registers to global memory, understand coalescing patterns, bank conflicts, and optimization strategies for maximum performance
OpenMP parallel programming: fork-join model, scheduling, data races, false sharing, NUMA thread affinity, and GPU offloading.
Why DataLoader num_workers matters: processes hide load latency behind GPU work, how to find the sweet spot, and the memory/GIL pitfalls that come with the pool.
Replay branch outcomes to compare predictors, then connect local and global history to speculative execution, recovery and profiling.
Trace data, structural and control hazards, understand forwarding and load-use stalls, and connect register dependencies to scheduling.
Master sequential vs strided memory access patterns. Learn how cache efficiency and hardware prefetching affect application performance.
