DataParallel vs DistributedDataParallel
Compare PyTorch DataParallel vs DistributedDataParallel for multi-GPU training. Learn GIL limitations, NCCL AllReduce, and DDP best practices.
No direct links0 refs
🔥
PyTorch Internals
Deep dive into PyTorch data loading, memory management, and distributed training patterns.
4
Concepts
All PyTorch Internals Concepts
PyTorch DataLoader Pipeline
Understanding how PyTorch DataLoader moves data from disk through CPU to GPU, including Dataset, Sampler, Workers, and Collate components.
No direct links0 refs
Understanding num_workers
Deep dive into PyTorch DataLoader num_workers parameter: how parallel workers prefetch data, optimal configuration, and common pitfalls.
No direct links0 refs
Pinned Memory and DMA Transfers
Understanding PyTorch pin_memory for faster CPU to GPU data transfers using DMA (Direct Memory Access) and page-locked memory.
No direct links0 refs