CUDA, tensor cores, multi-GPU communication, and cluster-scale workload orchestration.
Complete guide to Slurm — architecture, core commands, job lifecycle, job scripts, array jobs, dependencies, monitoring with squeue/sacct, and troubleshooting failed jobs on HPC clusters.
20 min read · intermediate
How NVIDIA Tensor Cores accelerate GEMM: warp-level MMA, precision dials (TF32/FP16/BF16/INT8/FP8), shape alignment cliffs, AMP vs model.half(), and when the speedup actually shows up.
16 min read · intermediate
How HBM feeds AI GPUs: the memory wall, width-over-speed packaging, TSVs and interposers, the on-package hierarchy, and the roofline that decides if your kernel is bandwidth-bound.
14 min read · intermediate
Master GPU memory hierarchy from registers to global memory, understand coalescing patterns, bank conflicts, and optimization strategies for maximum performance
20 min read · intermediate
How GPUs talk: the bandwidth cliff from HBM to Ethernet, NVLink 5 and GB200 NVL72 topologies, ring AllReduce step by step, and choosing between NCCL, Gloo, and MPI.
14 min read · intermediate
How a single SM actually runs work: warps of 32, cutaway of cores and memory, divergence tax, latency-hiding occupancy, and coalesced loads — instruments, not a catalog.
16 min read · intermediate
Complete guide to GPU allocation on Slurm — --gres flags, CUDA_VISIBLE_DEVICES remapping, GPU topology and NVLink binding, MIG partitioning, production job scripts, and debugging common GPU errors.
18 min read · intermediate
Deep dive into the CUDA context object: control vs data plane, inventory (memory, modules, streams, events, graphs), push/pop/setCurrent stacks, primary retain/release, flags and limits, isolation, cost, and traps.
24 min read · intermediate
A CUDA stream is an in-order queue of GPU ops. Overlap H→D, kernels, and D→H across streams — and avoid the default-stream trap that serializes everything.
10 min read · intermediate
How Slurm decides which jobs run first — priority factors, fair-share scheduling, backfill, and monitoring commands (squeue, sinfo, sacct).
8 min read · intermediate
NVIDIA Unified Virtual Memory (UVM): on-demand page migration, memory oversubscription, and simplified CPU-GPU memory management.
25 min read · intermediate
Decision map for CUDA: a context is per-process GPU state, a stream is an in-order queue inside a context, and MPS shares one context across processes. Pick the layer that matches the problem.
12 min read · intermediate
Flynn's Classification explained — SISD, SIMD, MISD, MIMD with interactive architecture explorer, SIMD evolution from MMX to AMX, branch divergence visualization, and workload-architecture throughput comparison.
15 min read · intermediate
Complete MPI guide — point-to-point and collective communication with real C and mpi4py code, deadlock simulation, performance benchmarking, communicator splitting, and debugging on HPC clusters.
22 min read · intermediate
OpenMP parallel programming: fork-join model, scheduling, data races, false sharing, NUMA thread affinity, and GPU offloading.
12 min read · advanced
CUDA page migration and fault handling between CPU and GPU memory. Learn TLB management, DMA transfers, and memory optimization.
16 min read · advanced
How Slurm tracks resource consumption through account hierarchies, TRES billing, and resource limits — sacctmgr, sreport, and the association model explained.
8 min read · intermediate
Why exclusive CUDA contexts leave SMs idle under multi-process load, how MPS multiplexes clients through a shared context, thread percentage caps, and when to pick exclusive, MPS, or MIG.
14 min read · intermediate
Amdahl and Gustafson ceilings, strong vs weak scaling, roofline bounds, and hiding all-reduce behind compute — the levers that decide whether more GPUs actually buy science.
11 min read · intermediate
How sched/backfill works — the algorithm that lets small jobs run in gaps while large jobs wait, why accurate time limits matter, and the key tuning parameters (bf_interval, bf_window, bf_max_job_test).
7 min read · intermediate
NVIDIA vs AMD for deep learning compared at both layers: the CUDA vs ROCm software moat, the microarchitecture (warp vs wavefront, SM vs CU, Tensor vs Matrix Cores), and the datacenter accelerators (H100/H200/B200 vs MI300X/MI325X).
10 min read · intermediate
How /dev/nvidia*, nvidiactl, nvidia-uvm, and DRI nodes map major/minor numbers to the driver, what CUDA opens first, and the minimum mount set for containers.
14 min read · intermediate
How the NVIDIA GPU Operator turns drivers, container toolkit, device plugin, GFD, DCGM, and MIG into DaemonSets reconciling from a ClusterPolicy — boot order, request path, sharing modes, and ops traps.
18 min read · intermediate
Why the first nvidia-smi or CUDA open costs seconds, what that cold path rebuilds, how nvidia-persistenced holds device FDs on the host, and when persistence pays off for pods, CI, and batch jobs.
14 min read · intermediate
GPU distributed parallelism: Data Parallel (DDP), Tensor Parallel, Pipeline Parallel, and ZeRO optimization for training large AI models.
20 min read · advanced
A fine kernel can still profile poorly: warp divergence taxes throughput ~1/N; alone on a long scoreboard wait, SM issue util can hit 0%. Interactive demos.
11 min read · intermediate
A deep dive into NCCL internals: communicators and channels, how it picks ring/tree/NVLS algorithms and LL/LL128/Simple protocols, reading NCCL_DEBUG logs, and tuning and debugging distributed training.
12 min read · intermediate
Data, tensor and pipeline parallelism split the batch, weight matrices or layers across GPUs. Compare memory, traffic and batch limits, then combine them.
11 min read · advanced