Skip to main content

Distributed Parallelism in Deep Learning

GPU distributed parallelism: Data Parallel (DDP), Tensor Parallel, Pipeline Parallel, and ZeRO optimization for training large AI models.

Overview

Modern deep learning models have grown too large to fit on a single GPU. Training GPT-4 or Llama 70B requires distributing computation across hundreds or thousands of GPUs. Distributed parallelism provides three orthogonal strategies to scale training: Data Parallel (split the batch), Tensor Parallel (split the layers), and Pipeline Parallel (split the model stages).

Understanding when to use each strategy—and how to combine them in 3D parallelism—is essential for efficient large-scale training. This guide covers the fundamentals of each approach, their trade-offs, and practical implementation with PyTorch and DeepSpeed.

Key Concepts

Data Parallelism (DDP)

Replicate the full model on each GPU and split the training batch. All-reduce gradients after backward pass to keep models synchronized.

Tensor Parallelism (TP)

Split individual weight matrices across GPUs. Each GPU computes partial results that are combined via all-reduce or all-gather per layer.

Pipeline Parallelism (PP)

Partition the model into sequential stages. Each GPU processes a different set of layers, passing activations forward through the pipeline.

3D Parallelism

Combine DP, TP, and PP to scale to thousands of GPUs. TP within nodes (NVLink), PP across nodes, DP for throughput scaling.

ZeRO Optimization

Partition optimizer states, gradients, and parameters across data-parallel ranks to reduce memory footprint without changing the parallelism model.

Communication Collectives

All-reduce (average gradients), all-gather (collect distributed data), reduce-scatter (scatter reduced results) form the backbone of distributed training.

The figure sets the three strategies side by side: what each one splits across the GPUs, how the GPUs communicate, and what each GPU holds in memory.

Three ways to split training across 4 GPUs

Data parallelism

Split the batch

Full Batch (32 samples)
GPU 0: 8
GPU 1: 8
GPU 2: 8
GPU 3: 8

Each GPU has full model

Model copy
Model copy
Model copy
Model copy

All-reduce gradients

after backward pass

Memory: Full model per GPU

Tensor parallelism

Split each layer horizontally

Linear Layer
[4096 × 16384]
GPU 0
[4096×4096]
GPU 1
[4096×4096]
GPU 2
[4096×4096]
GPU 3
[4096×4096]

All-reduce after each layer

High bandwidth needed

Best within node
NVLink: 600 GB/s

Memory: 1/N per GPU

Pipeline parallelism

Split layers vertically

  1. GPU 0: Layers 0-7
  2. GPU 1: Layers 8-15
  3. GPU 2: Layers 16-23
  4. GPU 3: Layers 24-31

Micro-batches keep GPUs busy

Bubble: idle time between stages

Memory: 1/P layers per GPU

Data Parallelism: Split the Batch

Data parallelism is the simplest and most widely used strategy. Each GPU holds a complete copy of the model and processes a different portion of the training batch. After computing local gradients, GPUs synchronize via all-reduce to average gradients before updating weights.

The figure steps through one training iteration on four GPUs. Press Play, or pick a step, to see where the local work ends and the all-reduce begins.

One DDP training step
Step 1 / 6
Step
GPU 0 (Rank 0)
Batch[0:8]
Model (full copy)
Forward Pass
Backward Pass
∇θ (gradients)
GPU 1 (Rank 1)
Batch[8:16]
Model (full copy)
Forward Pass
Backward Pass
∇θ (gradients)
GPU 2 (Rank 2)
Batch[16:24]
Model (full copy)
Forward Pass
Backward Pass
∇θ (gradients)
GPU 3 (Rank 3)
Batch[24:32]
Model (full copy)
Forward Pass
Backward Pass
∇θ (gradients)
All-reduce: average gradients across all GPUs → same ∇θ everywhere
All GPUs updated with identical weights, ready for the next iteration

Load Data: Each GPU loads its portion of the batch.

All GPUs must complete their batch before gradients are synchronized, so the slowest GPU determines throughput.

When to Use Data Parallelism

  • Model fits comfortably in single GPU memory
  • Need to increase training throughput
  • Simple setup with minimal code changes

PyTorch DistributedDataParallel Setup

python
import os
import torch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data import DataLoader, DistributedSampler

# Launch with torchrun: one process per GPU.
local_rank = int(os.environ['LOCAL_RANK'])
torch.cuda.set_device(local_rank)
dist.init_process_group(backend='nccl')

# Supply MyModel, dataset, criterion, and num_epochs for your application.
model = DDP(MyModel().cuda(local_rank), device_ids=[local_rank])
sampler = DistributedSampler(dataset)
dataloader = DataLoader(dataset, batch_size=32, sampler=sampler)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)

for epoch in range(num_epochs):
    sampler.set_epoch(epoch)  # Reshuffle each epoch consistently across ranks.
    for inputs, labels in dataloader:
        inputs = inputs.cuda(local_rank)
        labels = labels.cuda(local_rank)
        optimizer.zero_grad(set_to_none=True)
        loss = criterion(model(inputs), labels)
        loss.backward()  # DDP synchronizes gradient buckets.
        optimizer.step()

dist.destroy_process_group()
Key DDP Behaviors
  • Synchronous training: All GPUs must complete before the next iteration
  • Gradient averaging: All-reduce divides by world size automatically
  • Bucket fusion: Small gradients are grouped for efficient communication
  • Find unused parameters: Set find_unused_parameters=True for dynamic graphs

Tensor Parallelism: Split Within Layers

When a single layer is too large for one GPU, tensor parallelism splits weight matrices horizontally. Each GPU performs part of the matrix multiplication, then combines results via collective communication.

The figure splits one linear layer over two GPUs. Switch between column and row parallel to compare what each GPU holds and which collective rebuilds the output.

One linear layer on two GPUs
Split
Input X
[batch, 4096]
Same on all GPUs
Weight Matrix A
GPU 0
[4096, 8192]
GPU 1
[4096, 8192]
[4096, 16384]
Partial Outputs
Y₀ = X @ A₀
[batch, 8192]
Y₁ = X @ A₁
[batch, 8192]
ALL-GATHER
Concatenate
[Y₀ | Y₁]
→ [batch, 16384]
Full Output Y
[batch, 16384]
GPU 0GPU 1collective between GPUssame on both GPUs
Weight split
Along columns (output dimension)
Each GPU
Computes partial output features
Combine
All-Gather concatenates results, unless a row-parallel layer follows
Used in
MLP first linear layer, attention QKV projection

Column Parallel: Split output features, All-Gather at end.

Column vs Row Parallel

In transformer MLPs, we pair column parallel (first linear) with row parallel (second linear):

  1. Column Parallel: Split output dimension, each GPU computes partial features and keeps its slice, with no communication
  2. Row Parallel: Split input dimension (already partitioned), each GPU computes partial output, all-reduce to sum

This pairing needs no all-gather between the two matmuls, only one all-reduce per MLP block in the forward pass and one in the backward pass. With attention split the same way, each transformer layer needs two all-reduces forward and two backward.

In tensor parallelism, we split weight matrices across GPUs. But there are exactly two ways to slice a matrix: by columns or by rows. These aren't just different cuts—they have fundamentally different properties for how inputs must be distributed and how outputs must be combined.

Column Parallel: Split by Columns

In column-parallel mode, we slice the weight matrix vertically—each GPU gets a subset of the columns. Since each column of W produces one element of the output, each GPU computes a portion of the output features. The input is replicated and the output is split, so the full output needs an all-gather.

Column parallel: split W by columns (TP = 4)
Step 1 / 6

Step 1: Replicate Input X on All GPUs

X
[B,S,H]
GPU 0
X
[B,S,H]
GPU 1
X
[B,S,H]
GPU 2
X
[B,S,H]
GPU 3

Same input everywhere

Step 2: Partition W by Columns

W₀
W₁
W₂
W₃
W0
[H, H]
GPU 0
W1
[H, H]
GPU 1
W2
[H, H]
GPU 2
W3
[H, H]
GPU 3

Step 3: Local Compute

Each GPU computes:
Yᵢ = X · Wᵢ
[B,S,H] × [H,H] = [B,S,H]

Step 4: Partial Outputs (each GPU has 1/4 of output features)

Y0 = X · W0
[B, S, H] ← GPU 0
Y1 = X · W1
[B, S, H] ← GPU 1
Y2 = X · W2
[B, S, H] ← GPU 2
Y3 = X · W3
[B, S, H] ← GPU 3

Step 5: All-Gather to Get Full Output

ALL-GATHER: Concatenate [Y₀, Y₁, Y₂, Y₃] along last dimension
Each GPU collects all pieces → Every GPU now has complete Y

Result: Full Output on Every GPU

Y = [Y₀ | Y₁ | Y₂ | Y₃]
[B, S, H] | [B, S, H] | [B, S, H] | [B, S, H] = [B, S, 4H]
Column-Parallel Mathematics

If W has shape [H_in, H_out] and we use TP = N GPUs:

W = [W₀ | W₁ | ... | W_N-1] where each Wᵢ has shape [H_in, H_out/N]

Each GPU i computes:

Yᵢ = X · Wᵢ → shape [B, S, H_out/N]

Final output requires concatenation:

Y = AllGather([Y₀, Y₁, ..., Y_N-1]) → shape [B, S, H_out]

Row Parallel: Split by Rows

In row-parallel mode, we slice the weight matrix horizontally—each GPU gets a subset of the rows. Since each row of W corresponds to one input feature, the input X must also be split. Each GPU computes a partial result with the same shape, and an all-reduce sums them.

Row parallel: split W by rows (TP = 4)
Step 1 / 6

Step 1: Split Input X Along Last Dim

Full X: [B, S, H]
H = 4 chunks
X0
[B,S,H/4]
GPU 0
X1
[B,S,H/4]
GPU 1
X2
[B,S,H/4]
GPU 2
X3
[B,S,H/4]
GPU 3

Step 2: Partition W by Rows

W₀ [H/4, H_out]
W₁ [H/4, H_out]
W₂ [H/4, H_out]
W₃ [H/4, H_out]

Step 3: Local Compute

Each GPU computes:
Yᵢ = Xᵢ · Wᵢ
[B,S,H/4] × [H/4,H_out]
= [B,S,H_out]
Why Row-Parallel Needs Summation
Y = X·W = [X₀|X₁|X₂|X₃] · [W₀; W₁; W₂; W₃] = X₀·W₀ + X₁·W₁ + X₂·W₂ + X₃·W₃
Each GPU computes one term of this sum.

Step 4: Partial Outputs (same shape, different values — need to SUM)

Y0 = X0 · W0
[B, S, H_out] ← GPU 0
Y1 = X1 · W1
[B, S, H_out] ← GPU 1
Y2 = X2 · W2
[B, S, H_out] ← GPU 2
Y3 = X3 · W3
[B, S, H_out] ← GPU 3

All have shape [B, S, H_out] but contain PARTIAL sums

Step 5: All-Reduce (Sum) to Get Final Output

ALL-REDUCE (SUM): Y = Y₀ + Y₁ + Y₂ + Y₃
Each GPU sums all partials → Every GPU now has complete Y

Result: Full Output on Every GPU

Y = Y₀ + Y₁ + Y₂ + Y₃
[B, S, H_out] — replicated on all GPUs
Row-Parallel Mathematics

If W has shape [H_in, H_out] and we use TP = N GPUs:

W = [W₀; W₁; ...; W_N-1] where each Wᵢ has shape [H_in/N, H_out]

Input X must also be split (along last dimension):

X = [X₀ | X₁ | ... | X_N-1] where each Xᵢ has shape [B, S, H_in/N]

Each GPU i computes a partial product:

Yᵢ = Xᵢ · Wᵢ → shape [B, S, H_out]

Final output requires summation:

Y = AllReduce(Y₀ + Y₁ + ... + Y_N-1) → shape [B, S, H_out]

The Critical Difference

AspectColumn ParallelRow Parallel
Weight split directionVertically (by columns)Horizontally (by rows)
Input requirementReplicated (same on all GPUs)Split along last dimension
Partial output shape[B, S, H_out/N] — different features[B, S, H_out] — partial sums
Communication opAll-Gather (concatenate)All-Reduce (sum)
Output stateReplicatedReplicated
Ring send volume per GPUM × (N−1)/N bytes2M × (N−1)/N bytes

The Perfect Pair: Why They're Complementary

Megatron-style tensor parallelism rests on one observation: column-parallel output is already split in exactly the format row-parallel needs as input. Feeding one into the other needs no communication, so the pair costs one all-reduce at the end instead of two separate operations.

MLP block: column-parallel → row-parallel
Input X
[B, S, H]
Replicated
Column Parallel (FC1)
W₁: [H, 4H]
Each GPU: [H, H]
+ GeLU
Hidden State
Split (no communication)
Row Parallel (FC2)
W₂: [4H, H]
Each GPU: [H, H]
→ needs sum
ALL-REDUCE: Sum partial outputs from Row-Parallel
This is the only communication in the entire MLP forward pass.
Output Y
[B, S, H] — Replicated

Column-parallel produces split output along the feature dimension. Row-parallel needs split input along the feature dimension, so the two match.

Communication Volume Analysis

M is the full output tensor size, summed or concatenated across N GPUs. For a ring collective, count sent and received bytes separately:

All-Gather (Column Parallel)All-Reduce (Row Parallel)
Each GPU sendsM × (N−1)/N bytes2M × (N−1)/N bytes
Each GPU receivesM × (N−1)/N bytes2M × (N−1)/N bytes
Send + receive per GPU2M × (N−1)/N bytes4M × (N−1)/N bytes

For N=8 and M=8 MiB, each rank sends and receives 7 MiB for all-gather, or 14 MiB for all-reduce. This counts forwarded shards, not just the rank’s initial shard. All-reduce costs 2× the bandwidth of all-gather. This is why the Megatron pattern places row-parallel at the end of each block—where we need replicated output anyway (for residual connections).

Column vs row: two sides of the same coin

Column Parallel: Keep input whole, split output features. Good for the "up projection" where we expand dimensions.

Row Parallel: Split input features, keep output whole. Good for the "down projection" where we reduce dimensions.

High Bandwidth Required

Tensor parallelism requires communication after every layer, not just after the backward pass. This demands high-bandwidth interconnects like NVLink (H100: 450 GB/s per direction per GPU). A 100–400 Gb/s InfiniBand port provides 12.5–50 GB/s per direction before overhead; cross-node TP performance depends on aggregate NIC capacity and workload.

Pipeline Parallelism: Split by Stages

Pipeline parallelism partitions the model into sequential stages, with each GPU responsible for a subset of layers. Micro-batching keeps GPUs busy by processing multiple smaller batches simultaneously.

The figure lays out the forward and backward passes of each stage over time. Switch schedules to compare where the bubbles fall.

Pipeline schedule, 4 stages
Schedule
Time →
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
GPU 0
F0
F1
F2
F3
B3
B2
B1
B0
GPU 1
F0
F1
F2
F3
B3
B2
B1
B0
GPU 2
F0
F1
F2
F3
B3
B2
B1
B0
GPU 3
F0
F1
F2
F3
B3
B2
B1
B0
MB 0MB 1MB 2MB 3F = Forward (solid)B = Backward (light)bubble (idle)
Efficiency
~60%
Bubble ratio
(p-1)/(m+p-1)
p
4 pipeline stages
m
4 micro-batches

All forwards first, then all backwards. Better pipelining with micro-batches.

The Bubble Problem

In naive pipelining, GPUs sit idle waiting for activations from previous stages. This creates "bubbles" of wasted compute time. The bubble ratio depends on the schedule:

  • Naive: ~75% bubble time with 4 stages
  • GPipe: (p-1)/m of the ideal compute time, or (p-1)/(m+p-1) of the total step time (p=stages, m=micro-batches)
  • 1F1B: the same bubble as GPipe, plus lower activation memory
  • Interleaved 1F1B: (p-1)/(v·m) of the ideal compute time with v model chunks per GPU, at the cost of v times more communication
1F1B Memory Advantage

1F1B (One Forward One Backward) alternates forward and backward passes, allowing earlier activation release. GPipe stores m activations per stage until backward begins. With p stages, 1F1B uses only p activations in flight regardless of micro-batch count.

3D Parallelism: Combining All Three

For training the largest models (GPT-3, Llama 70B+), we combine all three strategies. The key insight is to match parallelism type to interconnect bandwidth:

Each small cube in the figure is one GPU. Move the sliders to change the data, tensor and pipeline sizes; the total is their product.

3D parallelism: DP × TP × PP
2
2
2
TPPPDP4567
DP: top faces, upTP: right facesPP: left faces

Communication Hierarchy

DimensionCommunication PatternFrequencyBest Interconnect
TPAll-reduce/gather per layerVery highNVLink (H100: 450 GB/s per direction/GPU)
PPPoint-to-point between stagesMediumHDR/NDR InfiniBand (25–50 GB/s per direction/port)
DPAll-reduce gradients per iterationLowNetwork (any bandwidth)

Megatron-LM 3D Parallelism Configuration

bash
# Training Llama 70B on 64 H100 GPUs (8 nodes × 8 GPUs)
# TP=8 within node (NVLink); PP=4 across nodes; DP=2 for larger batches
python train.py \
    --tensor-model-parallel-size 8 \
    --pipeline-model-parallel-size 4 \
    --data-parallel-size 2 \
    --micro-batch-size 1 \
    --global-batch-size 1024 \
    --num-layers 80 \
    --hidden-size 8192 \
    --num-attention-heads 64

# Total: 8 × 4 × 2 = 64 GPUs

ZeRO: Memory-Efficient Data Parallelism

ZeRO (Zero Redundancy Optimizer) reduces memory footprint by partitioning optimizer states, gradients, and parameters across data-parallel ranks instead of replicating them.

The figure stacks the model state each GPU holds under standard DDP and each ZeRO stage, with four data-parallel GPUs. Highlight a stage to read its breakdown.

Model state per GPU, N_d = 4 GPUs
Highlight
ParametersGradientsOptimizer StatesPartitioned (held by the other GPUs)

Each ZeRO stage reduces memory at the cost of more communication. ZeRO-1 has minimal overhead (optimizer states rarely accessed). ZeRO-3 requires all-gather for every forward pass, adding ~50% communication overhead but dividing per-GPU model state by N_d, so model size can grow with the number of GPUs.

Memory Breakdown

For a model with Φ parameters using Adam optimizer with mixed precision:

ComponentMemory (per GPU)Notes
FP16 Parameters2ΦModel weights
FP16 Gradients2ΦComputed during backward
FP32 Optimizer States12ΦAdam: params + momentum + variance
Total16ΦStandard DDP per GPU

ZeRO Stages

With N_d data-parallel GPUs:

  • ZeRO-1: Partition optimizer states → 4Φ + 12Φ/N_d per GPU (approaches 4× memory reduction as N_d grows)
  • ZeRO-2: + Partition gradients → 2Φ + 14Φ/N_d per GPU (approaches 8× memory reduction)
  • ZeRO-3: + Partition parameters → 16Φ/N_d per GPU (N_d× memory reduction)

DeepSpeed ZeRO-3 Configuration

json
{
  "zero_optimization": {
    "stage": 3,
    "offload_optimizer": {
      "device": "cpu",
      "pin_memory": true
    },
    "offload_param": {
      "device": "cpu",
      "pin_memory": true
    },
    "overlap_comm": true,
    "contiguous_gradients": true,
    "sub_group_size": 1e9,
    "reduce_bucket_size": "auto",
    "stage3_prefetch_bucket_size": "auto",
    "stage3_param_persistence_threshold": "auto"
  },
  "fp16": {
    "enabled": true,
    "loss_scale_window": 100
  }
}
ZeRO vs FSDP

PyTorch FSDP (Fully Sharded Data Parallel) implements ZeRO-3 natively in PyTorch. DeepSpeed ZeRO offers more configuration options and offloading capabilities, while FSDP provides tighter integration with PyTorch ecosystem.

Choosing the Right Strategy

For the three strategies side by side, with a decision flow and a figure of what each GPU holds, see data vs tensor vs pipeline parallelism.

The figure walks the decision as a series of yes or no questions. Answer them to reach a strategy, or pick an earlier step in the path to change an answer.

Choosing a parallelism strategy

Does model fit in single GPU memory?

Strategy Comparison

StrategyMemory EfficiencyCommunicationComplexityBest For
DDPLow (full replication)Low (once per iter)SimpleSmall models, throughput scaling
ZeRO-1/2MediumLow-MediumModerateMemory-constrained DDP
ZeRO-3/FSDPHighHighModerateVery large models, simple API
TPHighVery HighComplexLarge models within node
PPHighMediumComplexCross-node, layer-wise split
3D ParallelHighestOptimizedMost ComplexLargest models (100B+)

Real-World Applications

Fine-tuning LLMs

Fine-tuning a 7B model on consumer GPUs

ZeRO-3 + gradient checkpointing on 4×A100 40GB

Pre-training Foundation Models

Training models from scratch at massive scale

3D Parallelism: TP=8, PP=8, DP=16 on 1024 H100s

Multi-GPU Inference

Serving large models that exceed single GPU memory

Tensor parallelism for Llama 70B across 8 GPUs

Research with Limited Resources

Training large models on university clusters

ZeRO-2 with CPU offloading on 8 V100s

Production Training Pipelines

Consistent, reproducible large-scale training

DDP with gradient accumulation for batch size flexibility

Mixture-of-Experts Training

Training sparse models with expert parallelism

Expert Parallelism + DP for MoE layers

Best Practices

  • Start with DDP: Begin with data parallelism and only add complexity when needed. DDP is well-tested and has minimal overhead.
  • Match Parallelism to Interconnect: Use TP within nodes (NVLink), PP across nodes (InfiniBand), and DP anywhere. Never use TP across nodes without NVSwitch.
  • Profile Before Optimizing: Use NVIDIA Nsight Systems to identify whether you are compute-bound or communication-bound before adding parallelism.
  • Tune Micro-batch Size: For pipeline parallelism, use at least 4× as many micro-batches as stages to minimize bubble overhead.
  • Enable Gradient Checkpointing: Trade compute for memory by recomputing activations during backward pass. Essential for very deep models.
  • Use Mixed Precision: FP16/BF16 training halves memory usage and doubles throughput on Tensor Cores. Always enable when possible.

Common Pitfalls to Avoid

!

Tensor Parallel Across Nodes

Using TP across nodes without NVLink results in 10-30× slower layer computation due to InfiniBand latency

Solution: Keep TP within a single DGX node and use PP for cross-node distribution
!

Incorrect World Size Calculation

3D parallelism requires world_size = DP × TP × PP; mismatched sizes cause deadlocks

Solution: Verify process group setup: TP groups within nodes, PP groups across nodes, DP groups spanning everything
!

Memory Fragmentation with ZeRO-3

Frequent all-gather and release of parameters can fragment GPU memory

Solution: Use contiguous_gradients=true and tune prefetch bucket sizes
!

Unbalanced Pipeline Stages

Stages with different computational costs create bubbles even with good scheduling

Solution: Profile layer times and balance parameters per stage, accounting for attention vs FFN differences

Advantages & Limitations

Advantages

  • ✓Train models 100× larger than single GPU memory allows
  • ✓Near-linear scaling efficiency with proper configuration
  • ✓Flexibility to trade memory for compute and vice versa
  • ✓Production-proven at scale (GPT-4, Llama, PaLM)
  • ✓Well-supported by PyTorch, DeepSpeed, Megatron-LM
  • ✓CPU/NVMe offloading extends effective memory further

Limitations

  • ×Significant engineering complexity for 3D parallelism
  • ×Requires high-bandwidth interconnects for efficiency
  • ×Debugging distributed training failures is challenging
  • ×Communication overhead limits scaling efficiency
  • ×Requires careful hyperparameter tuning for each configuration
  • ×Different parallelism strategies have different numerical behaviors

Further Reading

GPU & High-Performance Computing
Data vs Tensor vs Pipeline Parallelism: When to Use Each

Data, tensor and pipeline parallelism split the batch, weight matrices or layers across GPUs. Compare memory, traffic and batch limits, then combine them.

GPU & High-Performance Computing
NCCL: How NVIDIA Collective Communication Works

A deep dive into NCCL internals: communicators and channels, how it picks ring/tree/NVLS algorithms and LL/LL128/Simple protocols, reading NCCL_DEBUG logs, and tuning and debugging distributed training.

Language & Framework Internals
PyTorch DataLoader Pipeline

Follow PyTorch batches through worker queues, collation, ordered delivery and GPU transfer, then measure loading throughput and budget process memory.

Language & Framework Internals
Pinned Memory and DMA Transfers in PyTorch

How page-locked memory changes CPU-to-GPU transfers, the extra staging in pageable paths, and the conditions needed to overlap copies with GPU work.

GPU & High-Performance Computing
CUDA Contexts: Ownership, Current Stack, Isolation

Deep dive into the CUDA context object: control vs data plane, inventory (memory, modules, streams, events, graphs), push/pop/setCurrent stacks, primary retain/release, flags and limits, isolation, cost, and traps.

GPU & High-Performance Computing
CUDA Context vs Streams vs MPS: Which Layer Fixes What

Decision map for CUDA: a context is per-process GPU state, a stream is an in-order queue inside a context, and MPS co-schedules work across client contexts on Volta+. Pick the layer that matches the problem.

If you found this explanation helpful, consider sharing it with others.

Mastodon