Distributed Parallelism in Deep Learning
GPU distributed parallelism: Data Parallel (DDP), Tensor Parallel, Pipeline Parallel, and ZeRO optimization for training large AI models.
10 min readConcept
Tagged with
pytorch
Explore machine learning concepts related to pytorch. Clear explanations and practical insights.
5
Concepts Found
Concepts Related to pytorch
DataParallel vs DistributedDataParallel
Compare PyTorch DataParallel vs DistributedDataParallel for multi-GPU training. Learn GIL limitations, NCCL AllReduce, and DDP best practices.
6 min readConcept
PyTorch DataLoader Pipeline
Understanding how PyTorch DataLoader moves data from disk through CPU to GPU, including Dataset, Sampler, Workers, and Collate components.
4 min readConcept
Understanding num_workers
Deep dive into PyTorch DataLoader num_workers parameter: how parallel workers prefetch data, optimal configuration, and common pitfalls.
5 min readConcept
Pinned Memory and DMA Transfers
Understanding PyTorch pin_memory for faster CPU to GPU data transfers using DMA (Direct Memory Access) and page-locked memory.
5 min readConcept