Multi-GPU Communication: NVLink, PCIe, and NCCL

A single GPU never waits for a network. The moment you split a model or a batch across two of them, your training speed stops being a compute number and becomes a plumbing number: how fast can device A's gradients reach device B? Synchronizing 1 GB of gradients takes about 0.6 ms over NVLink 5, 16 ms over a PCIe link (64 GB/s), and 80 ms over a 100-gigabit Ethernet link — same model, same math, a 130× spread decided entirely by wiring.

This page is about that wiring: the physical links, the shapes they're arranged in, the collective operations that run over them, and the libraries that pick the algorithms.

The Bandwidth Cliff

Every byte a GPU exchanges lives somewhere on a steep slope. On-package HBM moves terabytes per second; every step away from the die — to a sibling GPU, out of the chassis, across the network — costs roughly an order of magnitude:

The cliff explains nearly every design decision in distributed training. Tensor parallelism stays inside an NVLink domain because it communicates constantly; data parallelism tolerates crossing nodes because it synchronizes once per step; hierarchical AllReduce exists so traffic falls down the cliff exactly once instead of N times. If you remember one picture from this page, make it this one. (For the top of the cliff, see the HBM page.)

Interconnect Technologies

PCIe: the universal floor

Every GPU speaks PCIe — it's how the CPU loads data, and on systems without NVLink it's also how GPUs reach each other.

Universal, vendor-neutral — and shared. GPU-to-GPU traffic over PCIe contends with storage and network devices on the same fabric, which is why even Gen6 is the cliff's middle ledge, not its top.

NVLink: the scale-up fabric

NVLink is NVIDIA's point-to-point GPU link. The table lists both the per-link rate and the number that actually matters — the per-GPU aggregate with all links in use:

All figures bidirectional. Latency sits around a microsecond, and transfers bypass the CPU entirely. The catch: NVLink is NVIDIA-only, and since Ada (RTX 40-series) it's gone from consumer cards entirely — multi-GPU on gaming hardware means PCIe, full stop.

NVSwitch: from node to rack

Point-to-point links don't scale to "everyone talks to everyone." NVSwitch is dedicated switch silicon that gives full bisection bandwidth — every GPU pair communicates at the full per-GPU rate simultaneously.

• HGX H100 node: 8 GPUs through 4 NVSwitch chips — 900 GB/s any-to-any.
• GB200 NVL72 rack: the 2026 flagship dissolves the node boundary — 72 Blackwell GPUs across 18 compute trays wired through 9 NVLink-switch trays into one NVLink domain with 130 TB/s of aggregate bandwidth. Each NVLink 5 Switch chip carries 144 ports at 14.4 TB/s. The "node" is now the rack.

InfiniBand, RoCE, and EFA: crossing machines

Past the rack, traffic rides the network:

GPUDirect RDMA lets the NIC read and write GPU memory directly — no CPU bounce buffer — and modern training nodes pair one NIC per GPU to widen the pipe. Switch-side, Quantum-X800 adds SHARP in-network reduction: the switch itself sums tensors mid-flight, and NCCL 2.27 uses it on both InfiniBand and NVLink fabrics.

GPU Topologies

How your GPUs are wired sets your scaling ceiling before you write a line of code. Explore the four shapes that matter — click any device or link:

Four regimes, in ascending order of money:

1. Consumer PCIe box — two cards sharing the PCIe fabric. Fine for experimentation; AllReduce tops out at the slot bandwidth, and there is no NVLink escape hatch on 40/50-series cards.
2. HGX 8-GPU node — the workhorse. NVSwitch makes topology invisible inside the node: no lucky or unlucky GPU pairs.
3. GB200 NVL72 rack — 72 GPUs, one NVLink domain. Models that needed multi-node tensor parallelism now fit inside a single fabric running at NVLink speed.
4. Multi-node cluster — NVLink inside each node or rack, InfiniBand/EFA between them, and an ~18× bandwidth gap at the boundary (1.8 TB/s in-rack vs 100 GB/s per XDR port) that every framework works around with hierarchy.

Collective Operations

Distributed training speaks a small vocabulary of collectives. Watch each one move actual chunks around a 4-GPU ring:

What each is for, with the PyTorch one-liner:

AllReduce — everyone contributes, everyone gets the sum. The heartbeat of data parallelism; DDP calls it on your gradients after every backward pass.

torch.distributed.all_reduce(tensor, op=dist.ReduceOp.SUM)

Broadcast — one rank sends, all receive. Weight initialization at startup.

torch.distributed.broadcast(tensor, src=0)

AllGather — everyone shares their shard, everyone ends with all shards. Tensor-parallel activations; ZeRO parameter gathering.

torch.distributed.all_gather(tensor_list, tensor)

ReduceScatter — sum like AllReduce, but each rank keeps only its slice. ZeRO gradient sharding; exactly half of a ring AllReduce.

torch.distributed.reduce_scatter(output, input_list)

Point-to-point — direct send/recv between two ranks. Pipeline parallelism passing activations stage to stage.

torch.distributed.send(tensor, dst=next_rank)
torch.distributed.recv(tensor, src=prev_rank)

The ring algorithm in the stepper moves 2(N−1)/N of the data per GPU regardless of N — which is why ring AllReduce is bandwidth-optimal for large messages, while latency-bound small messages prefer tree algorithms. NCCL picks per message size; you rarely should.

Communication Libraries

NCCL detects your topology (NVLink, PCIe, InfiniBand), then picks ring, tree, or SHARP-offloaded algorithms per message. As of 2.27 it also runs SHARP reductions on NVLink fabrics and offers symmetric memory across domains up to a full NVL72.

torch.distributed.init_process_group(backend='nccl')

RCCL is NCCL's API twin for AMD — PyTorch even selects it through the 'nccl' backend name on ROCm. oneCCL covers Intel (backend='ccl' after importing oneccl_bindings_for_pytorch). Gloo (backend='gloo') works everywhere but optimizes nothing — keep it for CPUs and debugging. MPI integrates with schedulers and existing HPC tooling. MSCCL lets you write custom collectives when the built-in algorithms don't fit your topology.

Choosing Your Setup

1. Vendor picks the library: NVIDIA → NCCL, AMD → RCCL, Intel → oneCCL, mixed or CPU → Gloo/MPI.
2. Scale picks the fabric: 2 GPUs → PCIe is survivable; 4–8 → you want an NVSwitch node; dozens → NVL72 or multi-node InfiniBand; hundreds+ → multi-node is mandatory, design for the hierarchy.
3. Consumer hardware reality check: no NVLink since the 3090 — a dual-4090/5090 box is PCIe-only, and that's a constraint to design around, not a bug to fix.

Performance: Living With the Cliff

Bandwidth vs latency. Large transfers (>1 MB) are bandwidth-bound — interconnect choice dominates. Small transfers (<100 KB) are latency-bound — even PCIe looks fine, and the fix is batching, not faster links. DDP's gradient bucketing (bucket_cap_mb, default 25) exists precisely to turn many small AllReduces into few large ones.

Overlap communication with compute. While layer N's gradients sync, layer N−1's are still being computed. Bigger buckets use bandwidth better; smaller buckets start syncing sooner. It's a tunable, not a constant.

Respect the hierarchy. Reduce within the NVLink domain first, cross the network once with the result. DeepSpeed and Megatron do this automatically; manual setups use separate intra- and inter-node process groups.

Further Reading

• NCCL Documentation - algorithms, environment variables, and tuning straight from the source
• NVIDIA GB200 NVL72 - the rack-as-one-GPU architecture in detail
• PyTorch Distributed notes - how DDP buckets, overlaps, and calls the collectives on this page

Related concepts