GPU Memory Hierarchy & Optimization
Master GPU memory hierarchy from registers to global memory, understand coalescing patterns, bank conflicts, and optimization strategies for maximum performance
8 min readConcept
Tagged with
cuda
Explore machine learning concepts related to cuda. Clear explanations and practical insights.
10
Concepts Found
Concepts Related to cuda
NVIDIA Unified Virtual Memory
NVIDIA Unified Virtual Memory (UVM): on-demand page migration, memory oversubscription, and simplified CPU-GPU memory management.
12 min readConcept
Pinned Memory and DMA Transfers in PyTorch
Complete guide to PyTorch pin_memory — how DMA transfers work, when pinning helps vs hurts, NUMA effects, profiling with torch.profiler, num_workers interaction, and debugging slow data loading.
8 min readConcept
CUDA Multi-Process Service (MPS): GPU Sharing for Concurrent Workloads
Complete guide to CUDA MPS — architecture, performance benchmarks vs time-slicing and MIG, thread percentage planning, production deployment with systemd and Kubernetes, profiling with nsys, and troubleshooting.
10 min readConcept
NVIDIA Device Files in /dev/
Understanding character devices, major/minor numbers, and the device file hierarchy created by NVIDIA drivers for GPU access in Linux.
11 min readConcept
How Docker Works with GPUs: Device Files, Bind Mounts, and Driver Stacks
Understand how containerized processes access GPU hardware through device files, bind mounts, and the NVIDIA container runtime. Learn the kernel driver vs user-space library distinction.
7 min readConcept
CUDA Context vs Streams vs MPS: Process Isolation, Concurrency, and Multi-Tenancy
How CUDA contexts, streams, and MPS compare: a context is a per-process container of GPU state, a stream is an in-order queue inside a context, and MPS lets multiple processes share a single GPU concurrently. Three layers, three different problems.
9 min readConcept
CUDA Streams: Asynchronous Execution and Concurrency
A CUDA stream is a queue of GPU operations that execute in order. Understanding streams is the difference between a GPU at 30% utilization and one running flat out — they are how kernels and memory copies overlap on real hardware.
8 min readConcept
Tensor Cores Explained: Mixed Precision & Matrix Acceleration
NVIDIA Tensor Cores explained: architecture-, precision-, and workload-dependent matrix acceleration for AI training and inference on CUDA GPUs.
10 min readConcept
GPU Streaming Multiprocessor (SM)
Deep dive into the fundamental processing unit of modern GPUs - the Streaming Multiprocessor architecture, execution model, and memory hierarchy
12 min readConcept