Tensor Cores: Mixed Precision & Matrix Acceleration
How NVIDIA Tensor Cores accelerate GEMM: warp-level MMA, precision dials (TF32/FP16/BF16/INT8/FP8), shape alignment and padding cost, AMP vs model.half(), and when the speedup actually shows up.
Explore machine learning concepts related to CUDA. Clear explanations and practical insights.
How NVIDIA Tensor Cores accelerate GEMM: warp-level MMA, precision dials (TF32/FP16/BF16/INT8/FP8), shape alignment and padding cost, AMP vs model.half(), and when the speedup actually shows up.
Master GPU memory hierarchy from registers to global memory, understand coalescing patterns, bank conflicts, and optimization strategies for maximum performance
How a single SM actually runs work: warps of 32, cutaway of cores and memory, divergence tax, latency-hiding occupancy, and coalesced loads — instruments, not a catalog.
How page-locked memory changes CPU-to-GPU transfers, the extra staging in pageable paths, and the conditions needed to overlap copies with GPU work.
Deep dive into the CUDA context object: control vs data plane, inventory (memory, modules, streams, events, graphs), push/pop/setCurrent stacks, primary retain/release, flags and limits, isolation, cost, and traps.
A CUDA stream is an in-order queue of GPU ops. Overlap H→D, kernels, and D→H across streams — and avoid the legacy default-stream fences between blocking streams in one context.
NVIDIA Unified Virtual Memory (UVM): on-demand page migration, memory oversubscription, and simplified CPU-GPU memory management.
Decision map for CUDA: a context is per-process GPU state, a stream is an in-order queue inside a context, and MPS co-schedules work across client contexts on Volta+. Pick the layer that matches the problem.
Why ordinary multi-process CUDA time slicing can leave SMs idle, how Volta+ MPS co-schedules separate client contexts, execution-resource caps, and when to choose MPS or MIG.
NVIDIA vs AMD for deep learning compared at both layers: the CUDA vs ROCm software moat, the microarchitecture (warp vs wavefront, SM vs CU, Tensor vs Matrix Cores), and the datacenter accelerators (H100/H200/B200 vs MI300X/MI325X).
How /dev/nvidia*, nvidiactl, nvidia-uvm, and DRI nodes map major/minor numbers to the driver, what CUDA opens first, and the minimum mount set for containers.
A fine kernel can still profile poorly: warp divergence taxes throughput ~1/N; alone on a long scoreboard wait, SM issue util can hit 0%. Interactive demos.
Understand how containerized processes access GPU hardware through device files, bind mounts, and the NVIDIA container runtime. Learn the kernel driver vs user-space library distinction.