Multi-GPU Communication: NVLink, PCIe, and NCCL
How GPUs talk: the bandwidth cliff from HBM to Ethernet, NVLink 5 and GB200 NVL72 topologies, ring AllReduce step by step, and choosing between NCCL, Gloo, and MPI.
14 min readConcept
Tagged with
Distributed Training
Explore machine learning concepts related to Distributed Training. Clear explanations and practical insights.
5
Concepts Found
Concepts Related to Distributed Training
Slurm GPU Allocation for Distributed Training
Complete guide to GPU allocation on Slurm — --gres flags, CUDA_VISIBLE_DEVICES remapping, GPU topology and NVLink binding, MIG partitioning, production job scripts, and debugging common GPU errors.
18 min readConcept
Distributed Parallelism in Deep Learning
GPU distributed parallelism: Data Parallel (DDP), Tensor Parallel, Pipeline Parallel, and ZeRO optimization for training large AI models.
20 min readConcept
NCCL: How NVIDIA Collective Communication Works
A deep dive into NCCL internals: communicators and channels, how it picks ring/tree/NVLS algorithms and LL/LL128/Simple protocols, reading NCCL_DEBUG logs, and tuning and debugging distributed training.
12 min readConcept
Data vs Tensor vs Pipeline Parallelism: When to Use Each
Data, tensor and pipeline parallelism split the batch, weight matrices or layers across GPUs. Compare memory, traffic and batch limits, then combine them.
11 min readConcept