How GPUs talk: the bandwidth cliff from HBM to Ethernet, NVLink 5 and GB200 NVL72 topologies, ring AllReduce step by step, and choosing between NCCL, Gloo, and MPI.

Multi-GPU Communication: NVLink, PCIe, and NCCL

A deep dive into NCCL internals: communicators and channels, how it picks ring/tree/NVLS algorithms and LL/LL128/Simple protocols, reading NCCL_DEBUG logs, and tuning and debugging distributed training.

NCCL: How NVIDIA Collective Communication Works

Data, tensor and pipeline parallelism split the batch, weight matrices or layers across GPUs. Compare memory, traffic and batch limits, then combine them.

AllReduce

Concepts Related to AllReduce

Multi-GPU Communication: NVLink, PCIe, and NCCL

NCCL: How NVIDIA Collective Communication Works

Data vs Tensor vs Pipeline Parallelism: When to Use Each