Overview
Modern deep learning models have grown too large to fit on a single GPU. Training GPT-4 or Llama 70B requires distributing computation across hundreds or thousands of GPUs. Distributed parallelism provides three orthogonal strategies to scale training: Data Parallel (split the batch), Tensor Parallel (split the layers), and Pipeline Parallel (split the model stages).
Understanding when to use each strategy—and how to combine them in 3D parallelism—is essential for efficient large-scale training. This guide covers the fundamentals of each approach, their trade-offs, and practical implementation with PyTorch and DeepSpeed.
Key Concepts
Data Parallelism (DDP)
Replicate the full model on each GPU and split the training batch. All-reduce gradients after backward pass to keep models synchronized.
Tensor Parallelism (TP)
Split individual weight matrices across GPUs. Each GPU computes partial results that are combined via all-reduce or all-gather per layer.
Pipeline Parallelism (PP)
Partition the model into sequential stages. Each GPU processes a different set of layers, passing activations forward through the pipeline.
3D Parallelism
Combine DP, TP, and PP to scale to thousands of GPUs. TP within nodes (NVLink), PP across nodes, DP for throughput scaling.
ZeRO Optimization
Partition optimizer states, gradients, and parameters across data-parallel ranks to reduce memory footprint without changing the parallelism model.
Communication Collectives
All-reduce (average gradients), all-gather (collect distributed data), reduce-scatter (scatter reduced results) form the backbone of distributed training.
The figure sets the three strategies side by side: what each one splits across the GPUs, how the GPUs communicate, and what each GPU holds in memory.
Data parallelism
Split the batch
Each GPU has full model
All-reduce gradients
after backward pass
Memory: Full model per GPU
Tensor parallelism
Split each layer horizontally
All-reduce after each layer
High bandwidth needed
Memory: 1/N per GPU
Pipeline parallelism
Split layers vertically
- GPU 0: Layers 0-7
- GPU 1: Layers 8-15
- GPU 2: Layers 16-23
- GPU 3: Layers 24-31
Micro-batches keep GPUs busy
Memory: 1/P layers per GPU
Data Parallelism: Split the Batch
Data parallelism is the simplest and most widely used strategy. Each GPU holds a complete copy of the model and processes a different portion of the training batch. After computing local gradients, GPUs synchronize via all-reduce to average gradients before updating weights.
The figure steps through one training iteration on four GPUs. Press Play, or pick a step, to see where the local work ends and the all-reduce begins.
Load Data: Each GPU loads its portion of the batch.
All GPUs must complete their batch before gradients are synchronized, so the slowest GPU determines throughput.
When to Use Data Parallelism
- Model fits comfortably in single GPU memory
- Need to increase training throughput
- Simple setup with minimal code changes
PyTorch DistributedDataParallel Setup
import os
import torch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data import DataLoader, DistributedSampler
# Launch with torchrun: one process per GPU.
local_rank = int(os.environ['LOCAL_RANK'])
torch.cuda.set_device(local_rank)
dist.init_process_group(backend='nccl')
# Supply MyModel, dataset, criterion, and num_epochs for your application.
model = DDP(MyModel().cuda(local_rank), device_ids=[local_rank])
sampler = DistributedSampler(dataset)
dataloader = DataLoader(dataset, batch_size=32, sampler=sampler)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)
for epoch in range(num_epochs):
sampler.set_epoch(epoch) # Reshuffle each epoch consistently across ranks.
for inputs, labels in dataloader:
inputs = inputs.cuda(local_rank)
labels = labels.cuda(local_rank)
optimizer.zero_grad(set_to_none=True)
loss = criterion(model(inputs), labels)
loss.backward() # DDP synchronizes gradient buckets.
optimizer.step()
dist.destroy_process_group()- Synchronous training: All GPUs must complete before the next iteration
- Gradient averaging: All-reduce divides by world size automatically
- Bucket fusion: Small gradients are grouped for efficient communication
- Find unused parameters: Set
find_unused_parameters=Truefor dynamic graphs
Tensor Parallelism: Split Within Layers
When a single layer is too large for one GPU, tensor parallelism splits weight matrices horizontally. Each GPU performs part of the matrix multiplication, then combines results via collective communication.
The figure splits one linear layer over two GPUs. Switch between column and row parallel to compare what each GPU holds and which collective rebuilds the output.
- Weight split
- Along columns (output dimension)
- Each GPU
- Computes partial output features
- Combine
- All-Gather concatenates results, unless a row-parallel layer follows
- Used in
- MLP first linear layer, attention QKV projection
Column Parallel: Split output features, All-Gather at end.
Column vs Row Parallel
In transformer MLPs, we pair column parallel (first linear) with row parallel (second linear):
- Column Parallel: Split output dimension, each GPU computes partial features and keeps its slice, with no communication
- Row Parallel: Split input dimension (already partitioned), each GPU computes partial output, all-reduce to sum
This pairing needs no all-gather between the two matmuls, only one all-reduce per MLP block in the forward pass and one in the backward pass. With attention split the same way, each transformer layer needs two all-reduces forward and two backward.
In tensor parallelism, we split weight matrices across GPUs. But there are exactly two ways to slice a matrix: by columns or by rows. These aren't just different cuts—they have fundamentally different properties for how inputs must be distributed and how outputs must be combined.
Column Parallel: Split by Columns
In column-parallel mode, we slice the weight matrix vertically—each GPU gets a subset of the columns. Since each column of W produces one element of the output, each GPU computes a portion of the output features. The input is replicated and the output is split, so the full output needs an all-gather.
Step 1: Replicate Input X on All GPUs
Same input everywhere
Step 2: Partition W by Columns
Step 3: Local Compute
Step 4: Partial Outputs (each GPU has 1/4 of output features)
Step 5: All-Gather to Get Full Output
Result: Full Output on Every GPU
If W has shape [H_in, H_out] and we use TP = N GPUs:
Each GPU i computes:
Final output requires concatenation:
Row Parallel: Split by Rows
In row-parallel mode, we slice the weight matrix horizontally—each GPU gets a subset of the rows. Since each row of W corresponds to one input feature, the input X must also be split. Each GPU computes a partial result with the same shape, and an all-reduce sums them.
Step 1: Split Input X Along Last Dim
Step 2: Partition W by Rows
Step 3: Local Compute
Step 4: Partial Outputs (same shape, different values — need to SUM)
All have shape [B, S, H_out] but contain PARTIAL sums
Step 5: All-Reduce (Sum) to Get Final Output
Result: Full Output on Every GPU
If W has shape [H_in, H_out] and we use TP = N GPUs:
Input X must also be split (along last dimension):
Each GPU i computes a partial product:
Final output requires summation:
The Critical Difference
| Aspect | Column Parallel | Row Parallel |
|---|---|---|
| Weight split direction | Vertically (by columns) | Horizontally (by rows) |
| Input requirement | Replicated (same on all GPUs) | Split along last dimension |
| Partial output shape | [B, S, H_out/N] — different features | [B, S, H_out] — partial sums |
| Communication op | All-Gather (concatenate) | All-Reduce (sum) |
| Output state | Replicated | Replicated |
| Ring send volume per GPU | M × (N−1)/N bytes | 2M × (N−1)/N bytes |
The Perfect Pair: Why They're Complementary
Megatron-style tensor parallelism rests on one observation: column-parallel output is already split in exactly the format row-parallel needs as input. Feeding one into the other needs no communication, so the pair costs one all-reduce at the end instead of two separate operations.
Column-parallel produces split output along the feature dimension. Row-parallel needs split input along the feature dimension, so the two match.
Communication Volume Analysis
M is the full output tensor size, summed or concatenated across N GPUs. For a ring collective, count sent and received bytes separately:
| All-Gather (Column Parallel) | All-Reduce (Row Parallel) | |
|---|---|---|
| Each GPU sends | M × (N−1)/N bytes | 2M × (N−1)/N bytes |
| Each GPU receives | M × (N−1)/N bytes | 2M × (N−1)/N bytes |
| Send + receive per GPU | 2M × (N−1)/N bytes | 4M × (N−1)/N bytes |
For N=8 and M=8 MiB, each rank sends and receives 7 MiB for all-gather, or 14 MiB for all-reduce. This counts forwarded shards, not just the rank’s initial shard. All-reduce costs 2× the bandwidth of all-gather. This is why the Megatron pattern places row-parallel at the end of each block—where we need replicated output anyway (for residual connections).
Column vs row: two sides of the same coin
Column Parallel: Keep input whole, split output features. Good for the "up projection" where we expand dimensions.
Row Parallel: Split input features, keep output whole. Good for the "down projection" where we reduce dimensions.
Tensor parallelism requires communication after every layer, not just after the backward pass. This demands high-bandwidth interconnects like NVLink (H100: 450 GB/s per direction per GPU). A 100–400 Gb/s InfiniBand port provides 12.5–50 GB/s per direction before overhead; cross-node TP performance depends on aggregate NIC capacity and workload.
Pipeline Parallelism: Split by Stages
Pipeline parallelism partitions the model into sequential stages, with each GPU responsible for a subset of layers. Micro-batching keeps GPUs busy by processing multiple smaller batches simultaneously.
The figure lays out the forward and backward passes of each stage over time. Switch schedules to compare where the bubbles fall.
- Efficiency
- ~60%
- Bubble ratio
- (p-1)/(m+p-1)
- p
- 4 pipeline stages
- m
- 4 micro-batches
All forwards first, then all backwards. Better pipelining with micro-batches.
The Bubble Problem
In naive pipelining, GPUs sit idle waiting for activations from previous stages. This creates "bubbles" of wasted compute time. The bubble ratio depends on the schedule:
- Naive: ~75% bubble time with 4 stages
- GPipe:
(p-1)/mof the ideal compute time, or(p-1)/(m+p-1)of the total step time (p=stages, m=micro-batches) - 1F1B: the same bubble as GPipe, plus lower activation memory
- Interleaved 1F1B:
(p-1)/(v·m)of the ideal compute time with v model chunks per GPU, at the cost of v times more communication
1F1B (One Forward One Backward) alternates forward and backward passes, allowing earlier activation release. GPipe stores m activations per stage until backward begins. With p stages, 1F1B uses only p activations in flight regardless of micro-batch count.
3D Parallelism: Combining All Three
For training the largest models (GPT-3, Llama 70B+), we combine all three strategies. The key insight is to match parallelism type to interconnect bandwidth:
Each small cube in the figure is one GPU. Move the sliders to change the data, tensor and pipeline sizes; the total is their product.
Communication Hierarchy
| Dimension | Communication Pattern | Frequency | Best Interconnect |
|---|---|---|---|
| TP | All-reduce/gather per layer | Very high | NVLink (H100: 450 GB/s per direction/GPU) |
| PP | Point-to-point between stages | Medium | HDR/NDR InfiniBand (25–50 GB/s per direction/port) |
| DP | All-reduce gradients per iteration | Low | Network (any bandwidth) |
Megatron-LM 3D Parallelism Configuration
# Training Llama 70B on 64 H100 GPUs (8 nodes × 8 GPUs)
# TP=8 within node (NVLink); PP=4 across nodes; DP=2 for larger batches
python train.py \
--tensor-model-parallel-size 8 \
--pipeline-model-parallel-size 4 \
--data-parallel-size 2 \
--micro-batch-size 1 \
--global-batch-size 1024 \
--num-layers 80 \
--hidden-size 8192 \
--num-attention-heads 64
# Total: 8 × 4 × 2 = 64 GPUsZeRO: Memory-Efficient Data Parallelism
ZeRO (Zero Redundancy Optimizer) reduces memory footprint by partitioning optimizer states, gradients, and parameters across data-parallel ranks instead of replicating them.
The figure stacks the model state each GPU holds under standard DDP and each ZeRO stage, with four data-parallel GPUs. Highlight a stage to read its breakdown.
Each ZeRO stage reduces memory at the cost of more communication. ZeRO-1 has minimal overhead (optimizer states rarely accessed). ZeRO-3 requires all-gather for every forward pass, adding ~50% communication overhead but dividing per-GPU model state by N_d, so model size can grow with the number of GPUs.
Memory Breakdown
For a model with Φ parameters using Adam optimizer with mixed precision:
| Component | Memory (per GPU) | Notes |
|---|---|---|
| FP16 Parameters | 2Φ | Model weights |
| FP16 Gradients | 2Φ | Computed during backward |
| FP32 Optimizer States | 12Φ | Adam: params + momentum + variance |
| Total | 16Φ | Standard DDP per GPU |
ZeRO Stages
With N_d data-parallel GPUs:
- ZeRO-1: Partition optimizer states → 4Φ + 12Φ/N_d per GPU (approaches 4× memory reduction as N_d grows)
- ZeRO-2: + Partition gradients → 2Φ + 14Φ/N_d per GPU (approaches 8× memory reduction)
- ZeRO-3: + Partition parameters → 16Φ/N_d per GPU (N_d× memory reduction)
DeepSpeed ZeRO-3 Configuration
{
"zero_optimization": {
"stage": 3,
"offload_optimizer": {
"device": "cpu",
"pin_memory": true
},
"offload_param": {
"device": "cpu",
"pin_memory": true
},
"overlap_comm": true,
"contiguous_gradients": true,
"sub_group_size": 1e9,
"reduce_bucket_size": "auto",
"stage3_prefetch_bucket_size": "auto",
"stage3_param_persistence_threshold": "auto"
},
"fp16": {
"enabled": true,
"loss_scale_window": 100
}
}PyTorch FSDP (Fully Sharded Data Parallel) implements ZeRO-3 natively in PyTorch. DeepSpeed ZeRO offers more configuration options and offloading capabilities, while FSDP provides tighter integration with PyTorch ecosystem.
Choosing the Right Strategy
For the three strategies side by side, with a decision flow and a figure of what each GPU holds, see data vs tensor vs pipeline parallelism.
The figure walks the decision as a series of yes or no questions. Answer them to reach a strategy, or pick an earlier step in the path to change an answer.
Does model fit in single GPU memory?
Strategy Comparison
| Strategy | Memory Efficiency | Communication | Complexity | Best For |
|---|---|---|---|---|
| DDP | Low (full replication) | Low (once per iter) | Simple | Small models, throughput scaling |
| ZeRO-1/2 | Medium | Low-Medium | Moderate | Memory-constrained DDP |
| ZeRO-3/FSDP | High | High | Moderate | Very large models, simple API |
| TP | High | Very High | Complex | Large models within node |
| PP | High | Medium | Complex | Cross-node, layer-wise split |
| 3D Parallel | Highest | Optimized | Most Complex | Largest models (100B+) |
Real-World Applications
Fine-tuning LLMs
Fine-tuning a 7B model on consumer GPUs
Pre-training Foundation Models
Training models from scratch at massive scale
Multi-GPU Inference
Serving large models that exceed single GPU memory
Research with Limited Resources
Training large models on university clusters
Production Training Pipelines
Consistent, reproducible large-scale training
Mixture-of-Experts Training
Training sparse models with expert parallelism
Best Practices
- Start with DDP: Begin with data parallelism and only add complexity when needed. DDP is well-tested and has minimal overhead.
- Match Parallelism to Interconnect: Use TP within nodes (NVLink), PP across nodes (InfiniBand), and DP anywhere. Never use TP across nodes without NVSwitch.
- Profile Before Optimizing: Use NVIDIA Nsight Systems to identify whether you are compute-bound or communication-bound before adding parallelism.
- Tune Micro-batch Size: For pipeline parallelism, use at least 4× as many micro-batches as stages to minimize bubble overhead.
- Enable Gradient Checkpointing: Trade compute for memory by recomputing activations during backward pass. Essential for very deep models.
- Use Mixed Precision: FP16/BF16 training halves memory usage and doubles throughput on Tensor Cores. Always enable when possible.
Common Pitfalls to Avoid
Tensor Parallel Across Nodes
Using TP across nodes without NVLink results in 10-30× slower layer computation due to InfiniBand latency
Incorrect World Size Calculation
3D parallelism requires world_size = DP × TP × PP; mismatched sizes cause deadlocks
Memory Fragmentation with ZeRO-3
Frequent all-gather and release of parameters can fragment GPU memory
Unbalanced Pipeline Stages
Stages with different computational costs create bubbles even with good scheduling
Advantages & Limitations
Advantages
- ✓Train models 100× larger than single GPU memory allows
- ✓Near-linear scaling efficiency with proper configuration
- ✓Flexibility to trade memory for compute and vice versa
- ✓Production-proven at scale (GPT-4, Llama, PaLM)
- ✓Well-supported by PyTorch, DeepSpeed, Megatron-LM
- ✓CPU/NVMe offloading extends effective memory further
Limitations
- ×Significant engineering complexity for 3D parallelism
- ×Requires high-bandwidth interconnects for efficiency
- ×Debugging distributed training failures is challenging
- ×Communication overhead limits scaling efficiency
- ×Requires careful hyperparameter tuning for each configuration
- ×Different parallelism strategies have different numerical behaviors
Further Reading
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (Shoeybi et al., 2019), reviewed on this site
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
- GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism
- PyTorch FSDP Tutorial
- DeepSpeed Documentation
- NVIDIA Megatron-LM Repository
Related concepts
Data, tensor and pipeline parallelism split the batch, weight matrices or layers across GPUs. Compare memory, traffic and batch limits, then combine them.
A deep dive into NCCL internals: communicators and channels, how it picks ring/tree/NVLS algorithms and LL/LL128/Simple protocols, reading NCCL_DEBUG logs, and tuning and debugging distributed training.
Follow PyTorch batches through worker queues, collation, ordered delivery and GPU transfer, then measure loading throughput and budget process memory.
How page-locked memory changes CPU-to-GPU transfers, the extra staging in pageable paths, and the conditions needed to overlap copies with GPU work.
Deep dive into the CUDA context object: control vs data plane, inventory (memory, modules, streams, events, graphs), push/pop/setCurrent stacks, primary retain/release, flags and limits, isolation, cost, and traps.
Decision map for CUDA: a context is per-process GPU state, a stream is an in-order queue inside a context, and MPS co-schedules work across client contexts on Volta+. Pick the layer that matches the problem.
