Distributed Parallelism in Deep Learning
GPU distributed parallelism: Data Parallel (DDP), Tensor Parallel, Pipeline Parallel, and ZeRO optimization for training large AI models.
20 min readConcept
Tagged with
ZeRO
Explore machine learning concepts related to ZeRO. Clear explanations and practical insights.
2
Concepts Found
Concepts Related to ZeRO
Data vs Tensor vs Pipeline Parallelism: When to Use Each
Data, tensor and pipeline parallelism split the batch, weight matrices or layers across GPUs. Compare memory, traffic and batch limits, then combine them.
11 min readConcept