Explore machine learning concepts related to PyTorch. Clear explanations and practical insights.
PyTorch DataLoader deep dive — Dataset, Sampler, Workers, Collate internals, num_workers throughput profiling, memory analysis, serialization costs, production patterns (LMDB, WebDataset), and bottleneck diagnosis.
Why pin_memory=True matters: pageable paths pay two host copies and block the CPU; pinned memory enables one DMA hop and real overlap with GPU compute.
Why DataLoader num_workers matters: processes hide load latency behind GPU work, how to find the sweet spot, and the memory/GIL pitfalls that come with the pool.
GPU distributed parallelism: Data Parallel (DDP), Tensor Parallel, Pipeline Parallel, and ZeRO optimization for training large AI models.
Compare PyTorch DataParallel vs DistributedDataParallel for multi-GPU training. Learn GIL limitations, NCCL AllReduce, and DDP best practices.