Multi-GPU Communication: NVLink vs PCIe, NCCL, and Distributed Training

Why GPU Communication Matters

Training models too large for one GPU requires splitting work across multiple devices. This introduces a fundamental challenge: keeping GPUs synchronized.

• Data Parallelism: Each GPU processes different data, but all need to average gradients after every backward pass
• Model Parallelism: Different GPUs hold different model layers, requiring activation exchange during forward/backward passes
• Tensor Parallelism: Matrix operations split across GPUs need to combine partial results

The communication overhead can easily dominate training time if you choose the wrong interconnect, library, or algorithm.

Interconnect Technologies

Not all GPU connections are equal. The bandwidth and latency of your interconnect determines the maximum scaling efficiency.

PCIe: The Universal Standard

PCIe (Peripheral Component Interconnect Express) is the universal standard for connecting GPUs to CPUs and to each other (via PCIe switches).

Pros: Universal compatibility, works with any GPU vendor Cons: Shared bandwidth with other devices, lower than NVLink, involves CPU chipset

NVLink: NVIDIA's High-Speed Lane

NVLink is NVIDIA's proprietary high-speed GPU interconnect, offering 10-30x higher bandwidth than PCIe.

Pros: Direct GPU-to-GPU without CPU involvement, very low latency (~0.7μs) Cons: NVIDIA-only, requires compatible systems (DGX, HGX), premium cost

NVSwitch: Full-Mesh Connectivity

NVSwitch is a dedicated switch chip that connects all GPUs in a server with full bisection bandwidth—any GPU can talk to any other at full speed simultaneously.

• DGX A100: 8 GPUs connected via 6 NVSwitches, 600 GB/s any-to-any
• DGX H100: 8 GPUs connected via 4 NVSwitches, 900 GB/s any-to-any

This eliminates bottlenecks in AllReduce patterns where every GPU needs to communicate with every other.

InfiniBand and RoCE: Cross-Node Networking

For clusters with more than 8 GPUs, you need high-speed networking between nodes:

GPU Direct RDMA allows network cards to read/write GPU memory directly, bypassing the CPU for cross-node transfers.

GPU Topologies

Your GPU topology—how devices are physically connected—determines your scaling ceiling and which algorithms work best.

Consumer: PCIe-Connected GPUs

Typical gaming PC or workstation with 2 GPUs:

• Connected via PCIe switch or CPU lanes
• 32-64 GB/s bandwidth (shared)
• AllReduce involves CPU chipset

Scaling limit: Works for small models, but PCIe becomes bottleneck quickly.

Workstation: Partial NVLink

High-end workstations (RTX 4090 NVLink bridge, older Quadro setups):

• NVLink between some GPU pairs (600+ GB/s)
• PCIe fallback for cross-pair communication
• Mixed topology requires careful scheduling

Scaling limit: 4-8 GPUs, but cross-pair communication is slow.

DGX/HGX: Full NVSwitch Mesh

NVIDIA's purpose-built systems:

• Full bisection bandwidth via NVSwitch
• 600-900 GB/s any-to-any
• Optimal for AllReduce (ring or tree works equally well)

Scaling limit: 8 GPUs per node—beyond requires InfiniBand.

Multi-Node Clusters

For training the largest models:

• Intra-node: NVSwitch (600+ GB/s)
• Inter-node: InfiniBand (50-100 GB/s)
• 12x bandwidth difference means hierarchical communication is essential

Scaling limit: Network complexity, job scheduling, fault tolerance.

Communication Patterns

Collective operations are the building blocks of distributed training. Understanding when to use each is critical for performance.

AllReduce

The most important operation for data-parallel training. Every GPU contributes data, and all receive the reduced (summed/averaged) result.

# PyTorch DDP does this automatically after backward()
torch.distributed.all_reduce(tensor, op=dist.ReduceOp.SUM)

Use case: Gradient synchronization after backward pass

Broadcast

One GPU sends data to all others.

# Distribute initial weights from rank 0
torch.distributed.broadcast(tensor, src=0)

Use case: Distributing model weights at initialization

AllGather

Each GPU shares its unique data with all others. All GPUs end up with all data.

# Each GPU has different tensor, all collect all tensors
torch.distributed.all_gather(tensor_list, tensor)

Use case: Collecting activations in tensor parallelism

ReduceScatter

Opposite of AllGather: reduce across GPUs, then scatter different chunks to different GPUs.

# ZeRO optimizer: each GPU gets different gradient shard
torch.distributed.reduce_scatter(output, input_list)

Use case: ZeRO optimizer stages, memory-efficient gradient handling

Point-to-Point

Direct GPU-to-GPU transfer without involving all GPUs.

# Pipeline parallelism: send activations to next stage
torch.distributed.send(tensor, dst=next_rank)
torch.distributed.recv(tensor, src=prev_rank)

Use case: Pipeline parallelism activation exchange

Communication Libraries

Multiple libraries implement these primitives. Choose based on your hardware.

NCCL (NVIDIA)

The gold standard for NVIDIA GPUs. Automatically detects topology, selects optimal algorithms, and achieves near-theoretical bandwidth.

# PyTorch auto-selects NCCL for CUDA tensors
torch.distributed.init_process_group(backend='nccl')

• Topology-aware: Detects NVLink, PCIe, InfiniBand
• Algorithm selection: Ring, tree, or collnet based on message size
• Closed-source: Can't customize algorithms

RCCL (AMD)

API-compatible with NCCL for AMD ROCm GPUs. Similar topology detection for MI200/MI300 series.

# For AMD GPUs with ROCm
torch.distributed.init_process_group(backend='nccl')  # RCCL provides NCCL-compatible interface

oneCCL (Intel)

Intel's library for CPUs and Intel Data Center GPUs (Ponte Vecchio).

# For Intel devices
import oneccl_bindings_for_pytorch
torch.distributed.init_process_group(backend='ccl')

Gloo

Cross-platform fallback when vendor libraries unavailable. Works on CPU and GPU.

# Useful for development/testing or CPU-only environments
torch.distributed.init_process_group(backend='gloo')

Limitation: No topology optimization, lower performance than vendor libs.

MPI

The classic HPC standard. Maximum flexibility, works everywhere.

# Requires MPI installation (OpenMPI, MPICH)
torch.distributed.init_process_group(backend='mpi')

Best for: HPC environments with existing MPI infrastructure.

MSCCL (Microsoft)

Extends NCCL with custom algorithm support. You can define your own communication patterns.

Best for: Research, custom topologies, Azure clusters.

torch.distributed.init_process_group(
    backend='nccl',
    init_method='env://',
    world_size=8,
    rank=local_rank
)

# Set before training
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,GRAPH

# DDP wraps model and handles sync
model = DDP(model, device_ids=[local_rank])

# Or manually:
for param in model.parameters():
    dist.all_reduce(param.grad)

# DDP does this with gradient bucketing
# Tune bucket size for your model:
model = DDP(model, bucket_cap_mb=25)

Choosing the Right Setup

Decision Tree

1. What GPU vendor?

   • NVIDIA → NCCL
   • AMD → RCCL
   • Intel → oneCCL
   • Mixed/CPU → Gloo or MPI

2. How many GPUs?

   • 1-2: PCIe usually sufficient
   • 4-8: NVLink strongly recommended
   • 8+: Multi-node with InfiniBand

3. Need custom algorithms?

   • No → Use vendor default
   • Yes → MSCCL or MPI

Performance Considerations

Bandwidth vs Latency

• Large messages (>1MB): Bandwidth-bound, NVLink helps most
• Small messages (<100KB): Latency-bound, even PCIe is similar

This is why:

• Gradient bucketing batches small parameter gradients into larger transfers
• Tree algorithms are better for small messages (lower latency)
• Ring algorithms are better for large messages (optimal bandwidth)

Communication Overlap

Modern frameworks overlap communication with computation:

1. Compute gradients for layer N+1
2. While computing, AllReduce gradients for layer N
3. By the time layer 1 gradients are done, sync is complete

bucket_cap_mb controls this tradeoff:

• Larger: Better bandwidth utilization
• Smaller: Earlier start of communication, more overlap

Multi-Node Efficiency

The 12x bandwidth gap between intra-node (NVLink) and inter-node (InfiniBand) means:

• Hierarchical AllReduce: Reduce within node first, then across nodes
• Model placement matters: Keep layers that communicate frequently on same node
• Gradient compression: Reduces cross-node traffic at cost of some accuracy

Related Concepts

• NCCL Communication: Deep dive into NCCL internals, algorithms, and debugging
• Distributed Parallelism: Data, model, pipeline, and tensor parallelism strategies
• GPU Memory Hierarchy: Understanding HBM, L2 cache, and memory bandwidth
• HBM Memory: High Bandwidth Memory that feeds NVLink