Tagged with

NVLink

Explore machine learning concepts related to NVLink. Clear explanations and practical insights.

Concepts Found

Concepts Related to NVLink

January 29, 2025

Multi-GPU Communication: NVLink, PCIe, and NCCL

How GPUs talk: the bandwidth cliff from HBM to Ethernet, NVLink 5 and GB200 NVL72 topologies, ring AllReduce step by step, and choosing between NCCL, Gloo, and MPI.

Multi-GPU NVLink PCIe NCCL Distributed Training GPU interconnect NVSwitch AllReduce InfiniBand

14 min readConcept

September 30, 2026

Data vs Tensor vs Pipeline Parallelism: When to Use Each

Data, tensor and pipeline parallelism split the batch, weight matrices or layers across GPUs. Compare memory, traffic and batch limits, then combine them.

Distributed Training Data Parallel Tensor Parallel Pipeline Parallel ZeRO NVLink AllReduce

11 min readConcept