Four terms recur throughout this page:
- Streaming Multiprocessor (SM): the fundamental processing unit containing CUDA cores, Tensor cores, RT cores, schedulers, and memory.
- Warp: a group of 32 threads that execute in SIMD fashion; the basic scheduling unit in GPU execution.
- Memory coalescing: combining memory accesses from multiple threads into fewer transactions for optimal bandwidth.
- Occupancy: the ratio of active warps to maximum warps per SM. Higher occupancy helps hide memory latency.
The explorer below has four views of an NVIDIA A100: its SM grid and memory system, memory access patterns, warp execution, and kernel launch configuration. Pick a view, then select an SM, switch the access pattern, toggle divergence, or move the launch sliders to see occupancy change.
108 Streaming Multiprocessors
SM 0 partitions
Each partition contains 16 CUDA cores (64 total per SM)
GPU Memory Hierarchy Overview
Modern GPUs feature a complex memory hierarchy designed to maximize throughput for parallel workloads. Understanding this hierarchy is crucial for achieving peak performance in GPU applications. One powerful technique for reducing memory traffic across these levels is kernel fusion, which combines multiple operations into a single GPU kernel to avoid redundant reads and writes.
Memory Types and Characteristics
1. Registers (Fastest, Private)
Location: On-chip, per-thread
Size: 256 KB per SM (65,536 × 32-bit)
Latency: 0 cycles (immediate)
Bandwidth: ~8 TB/s
Scope: Private to each thread
Key Points:
- Fastest storage in GPU
- Compiler-managed allocation
- Spilling to local memory impacts performance
- Limited to 255 registers per thread
2. Shared Memory (Fast, Shared within Block)
Location: On-chip, per SM
Size: 48-228 KB per SM (configurable)
Latency: ~20-30 cycles
Bandwidth: ~4 TB/s
Scope: Shared within thread block
Optimization Strategies:
- Use for data reuse within blocks
- Avoid bank conflicts (32 banks, 4-byte width)
- Implement tiling algorithms
- Coordinate thread access patterns
3. L1 Cache (Fast, Automatic)
Location: On-chip, per SM
Size: 128 KB (combined with shared memory)
Latency: ~30-40 cycles
Bandwidth: ~4 TB/s
Scope: Per SM, caches global/local memory
Configuration Options:
// Configure L1/Shared split
cudaFuncSetCacheConfig(kernel, cudaFuncCachePreferShared); // More shared memory
cudaFuncSetCacheConfig(kernel, cudaFuncCachePreferL1); // More L1 cache
cudaFuncSetCacheConfig(kernel, cudaFuncCachePreferEqual); // Balanced
4. L2 Cache (Medium, GPU-wide)
Location: On-chip, shared across GPU
Size: 40-60 MB (A100: 40 MB, H100: 50 MB)
Latency: ~200 cycles
Bandwidth: ~2-3 TB/s
Scope: All SMs, coherent
Features:
- Caches all memory accesses
- Persistent across kernel launches
- Supports atomic operations
- Hardware-managed coherency
5. Global Memory (Slow, Large)
Location: Off-chip DRAM/HBM
Size: 24-80 GB (HBM2e/HBM3)
Latency: ~400-600 cycles
Bandwidth: 1-2 TB/s
Scope: Entire GPU and host
Memory Coalescing Patterns
Coalesced Access (Optimal)
// Perfect coalescing - consecutive threads access consecutive addresses
__global__ void coalesced_access(float* data) {
int tid = blockIdx.x * blockDim.x + threadIdx.x;
float value = data[tid]; // Thread 0→data[0], Thread 1→data[1], etc.
}
Strided Access (Poor)
// Strided pattern - wastes bandwidth
__global__ void strided_access(float* data, int stride) {
int tid = blockIdx.x * blockDim.x + threadIdx.x;
float value = data[tid * stride]; // Non-consecutive access
}
Random Access (Worst)
// Random pattern - serialized transactions
__global__ void random_access(float* data, int* indices) {
int tid = blockIdx.x * blockDim.x + threadIdx.x;
float value = data[indices[tid]]; // Unpredictable pattern
}
Shared Memory Optimization
Bank Conflict Resolution
// Read patterns for one warp, 32 banks, 4-byte words.
// Assume the arrays have already been initialized and synchronized.
int lane = threadIdx.x % 32;
__shared__ float words[32 * 32];
float conflict = words[32 * lane]; // 32 distinct words, same bank: 32-way
float broadcast = words[lane % 4]; // Four words in distinct banks: broadcasts
// A transpose reads columns of a tile. Padding changes the row stride.
__shared__ float tile[32][33];
float column = tile[lane][0]; // stride 33: one distinct bank per lane
Matrix Transpose with Shared Memory
#define TILE_DIM 32
#define BLOCK_ROWS 8
__global__ void transpose_coalesced(float *odata, float *idata, int width, int height) {
__shared__ float tile[TILE_DIM][TILE_DIM + 1]; // +1 padding avoids bank conflicts
int x = blockIdx.x * TILE_DIM + threadIdx.x;
int y = blockIdx.y * TILE_DIM + threadIdx.y;
// Coalesced read from global memory
for (int j = 0; j < TILE_DIM; j += BLOCK_ROWS) {
if (x < width && (y + j) < height) {
tile[threadIdx.y + j][threadIdx.x] = idata[(y + j) * width + x];
}
}
__syncthreads();
// Transpose indices for write
x = blockIdx.y * TILE_DIM + threadIdx.x;
y = blockIdx.x * TILE_DIM + threadIdx.y;
// Coalesced write to global memory
for (int j = 0; j < TILE_DIM; j += BLOCK_ROWS) {
if (x < height && (y + j) < width) {
odata[(y + j) * height + x] = tile[threadIdx.x][threadIdx.y + j];
}
}
}
Memory Access Optimization Techniques
1. Texture Memory (Spatial Locality)
// Texture memory for 2D spatial locality
texture<float, cudaTextureType2D> texRef;
__global__ void texture_kernel(float* output, int width, int height) {
int x = blockIdx.x * blockDim.x + threadIdx.x;
int y = blockIdx.y * blockDim.y + threadIdx.y;
// Hardware-optimized 2D cache
output[y * width + x] = tex2D(texRef, x, y);
}
2. Constant Memory (Broadcast)
__constant__ float const_data[1024]; // 64 KB constant memory
__global__ void const_kernel(float* output) {
int tid = threadIdx.x;
// All threads read same value - broadcast optimization
output[tid] = const_data[0] * tid;
}
3. Unified Memory (Simplified Programming)
// Automatic migration between host and device
float *data;
cudaMallocManaged(&data, size);
// Access from both CPU and GPU
data[0] = 1.0f; // CPU write
kernel<<<grid, block>>>(data); // GPU access
cudaDeviceSynchronize();
float result = data[0]; // CPU read
Performance Analysis Tools
Memory Bandwidth Calculation
Effective Bandwidth (GB/s) = (Bytes Read + Bytes Written) / Elapsed Seconds / 1e9
Efficiency = (Effective Bandwidth / Theoretical Bandwidth) × 100%
Requested read/write bytes define effective bandwidth; memory-sector counters describe actual traffic and can expose wasted transfers. Occupancy is a separate latency-hiding metric, already reflected in the elapsed time, so it is not a bandwidth multiplier. See NVIDIA’s effective bandwidth definition.
Profiling Metrics
# NSight Compute memory metrics
ncu --metrics l1tex__t_bytes_pipe_lsu_mem_global_op_ld.sum,
l1tex__t_bytes_pipe_lsu_mem_global_op_st.sum,
smsp__sass_average_data_bytes_per_sector_mem_global_op_ld.pct,
smsp__sass_average_data_bytes_per_sector_mem_global_op_st.pct
./application
Memory Optimization Checklist
Design Phase
- Identify memory access patterns
- Calculate memory bandwidth requirements
- Plan data layout for coalescing
- Determine shared memory usage
Implementation Phase
- Implement coalesced access patterns
- Use shared memory for data reuse
- Avoid bank conflicts with padding
- Minimize register pressure
- Use appropriate memory types (texture, constant)
Optimization Phase
- Profile memory transactions
- Analyze bandwidth utilization
- Identify and fix inefficient patterns
- Tune L1/shared memory configuration
- Consider memory prefetching
Advanced Memory Patterns
Warp-level Memory Operations
// Warp shuffle for register-level data exchange
__global__ void warp_reduce(float* data) {
float value = data[threadIdx.x];
// Warp-level reduction without shared memory
for (int mask = 16; mask > 0; mask >>= 1) {
value += __shfl_xor_sync(0xffffffff, value, mask);
}
if (threadIdx.x % 32 == 0) {
data[threadIdx.x / 32] = value;
}
}
Memory-level Parallelism
// Increase memory-level parallelism with unrolling
__global__ void mlp_kernel(float* __restrict__ out,
const float* __restrict__ in,
int n) {
int tid = blockIdx.x * blockDim.x + threadIdx.x;
int stride = gridDim.x * blockDim.x;
// Process multiple elements per thread
float sum0 = 0, sum1 = 0, sum2 = 0, sum3 = 0;
for (int i = tid; i < n/4; i += stride) {
// Issue multiple loads simultaneously
float4 val = reinterpret_cast<const float4*>(in)[i];
sum0 += val.x;
sum1 += val.y;
sum2 += val.z;
sum3 += val.w;
}
out[tid] = sum0 + sum1 + sum2 + sum3;
}
Best Practices Summary
Do's
- Coalesce global memory accesses
- Use shared memory for data reuse
- Pad shared memory arrays to avoid bank conflicts
- Batch small transfers into larger ones
- Use vector loads (float2, float4) when possible
- Profile and measure actual bandwidth
Don'ts
- Random memory access patterns
- Excessive register usage causing spills
- Unaligned memory accesses
- Frequent host-device transfers
- Ignoring memory hierarchy
- Assuming CPU optimization strategies work on GPU
Performance Impact
Bandwidth Utilization by Pattern
For one 32-thread warp reading 4-byte words from an aligned base on NVIDIA compute capability 6.0+, count distinct 32-byte sectors. These are analytical traffic values, not measured bandwidth or latency. Permuting lanes within the same address set does not change the sector count.
| Access pattern | Sector utilization (unique bytes / sector bytes) | 32-byte sectors |
|---|---|---|
| Consecutive words | 100% | 4 |
| Consecutive, offset by one word | 80% | 5 |
| Stride 2 words | 50% | 8 |
| Stride 32 words | 12.5% | 32 |
| Scattered, one word per sector | 12.5% | 32 |
| Broadcast of one word | 12.5% (same word reused by all lanes) | 1 |
Conclusion
GPU memory hierarchy optimization is essential for achieving peak performance. Focus on:
- Coalesced access patterns for global memory
- Shared memory for data reuse and communication
- Avoiding bank conflicts through padding
- Proper occupancy to hide memory latency
- Profiling and measurement to validate optimizations
Master these concepts to unlock the full potential of GPU computing and achieve order-of-magnitude performance improvements in your applications.
Related concepts
A fine kernel can still profile poorly: warp divergence taxes throughput ~1/N; alone on a long scoreboard wait, SM issue util can hit 0%. Interactive demos.
How a single SM actually runs work: warps of 32, cutaway of cores and memory, divergence tax, latency-hiding occupancy, and coalesced loads — instruments, not a catalog.
How page-locked memory changes CPU-to-GPU transfers, the extra staging in pageable paths, and the conditions needed to overlap copies with GPU work.
Structure of Arrays vs Array of Structures as instruments: cache-line fill, SIMD gather vs contiguous load, GPU coalescing, and AoSoA hybrids — when layout is a 10× decision.
Deep dive into the CUDA context object: control vs data plane, inventory (memory, modules, streams, events, graphs), push/pop/setCurrent stacks, primary retain/release, flags and limits, isolation, cost, and traps.
Decision map for CUDA: a context is per-process GPU state, a stream is an in-order queue inside a context, and MPS co-schedules work across client contexts on Volta+. Pick the layer that matches the problem.
