Tagged with

allreduce

Explore machine learning concepts related to allreduce. Clear explanations and practical insights.

Concepts Found

Concepts Related to allreduce

January 29, 2025

Multi-GPU Communication: NVLink, PCIe, and NCCL

How GPUs talk: the bandwidth cliff from HBM to Ethernet, NVLink 5 and GB200 NVL72 topologies, ring AllReduce step by step, and choosing between NCCL, Gloo, and MPI.

multi-GPU NVLink PCIe NCCL distributed training GPU interconnect NVSwitch AllReduce InfiniBand

11 min readConcept

January 15, 2025

NCCL: How NVIDIA Collective Communication Works

A deep dive into NCCL internals: communicators and channels, how it picks ring/tree/NVLS algorithms and LL/LL128/Simple protocols, reading NCCL_DEBUG logs, and tuning and debugging distributed training.

GPU NCCL Distributed Training Multi-GPU Communication Primitives AllReduce

9 min readConcept