Tagged with

GPU interconnect

Explore machine learning concepts related to GPU interconnect. Clear explanations and practical insights.

Concepts Found

Concepts Related to GPU interconnect

January 29, 2025

Multi-GPU Communication: NVLink, PCIe, and NCCL

How GPUs talk: the bandwidth cliff from HBM to Ethernet, NVLink 5 and GB200 NVL72 topologies, ring AllReduce step by step, and choosing between NCCL, Gloo, and MPI.

Multi-GPU NVLink PCIe NCCL Distributed Training GPU interconnect NVSwitch AllReduce InfiniBand

14 min readConcept