Multi-GPU Communication: NVLink, PCIe, and NCCL
How GPUs talk: the bandwidth cliff from HBM to Ethernet, NVLink 5 and GB200 NVL72 topologies, ring AllReduce step by step, and choosing between NCCL, Gloo, and MPI.
14 min readConcept
Tagged with
NCCL
Explore machine learning concepts related to NCCL. Clear explanations and practical insights.
3
Concepts Found
Concepts Related to NCCL
NCCL: How NVIDIA Collective Communication Works
A deep dive into NCCL internals: communicators and channels, how it picks ring/tree/NVLS algorithms and LL/LL128/Simple protocols, reading NCCL_DEBUG logs, and tuning and debugging distributed training.
12 min readConcept
DataParallel vs DistributedDataParallel
Compare PyTorch DataParallel vs DistributedDataParallel for multi-GPU training. Learn GIL limitations, NCCL AllReduce, and DDP best practices.
6 min readConcept