Distributed Parallelism in Deep Learning
GPU distributed parallelism: Data Parallel (DDP), Tensor Parallel, Pipeline Parallel, and ZeRO optimization for training large AI models.
20 min readConcept
Tagged with
Tensor Parallel
Explore machine learning concepts related to Tensor Parallel. Clear explanations and practical insights.
2
Concepts Found
Concepts Related to Tensor Parallel
Data vs Tensor vs Pipeline Parallelism: When to Use Each
Data, tensor and pipeline parallelism split the batch, weight matrices or layers across GPUs. Compare memory, traffic and batch limits, then combine them.
11 min readConcept