Skip to main content

GPU Memory Hierarchy & Optimization

Master GPU memory hierarchy from registers to global memory, understand coalescing patterns, bank conflicts, and optimization strategies for maximum performance

Four terms recur throughout this page:

  • Streaming Multiprocessor (SM): the fundamental processing unit containing CUDA cores, Tensor cores, RT cores, schedulers, and memory.
  • Warp: a group of 32 threads that execute in SIMD fashion; the basic scheduling unit in GPU execution.
  • Memory coalescing: combining memory accesses from multiple threads into fewer transactions for optimal bandwidth.
  • Occupancy: the ratio of active warps to maximum warps per SM. Higher occupancy helps hide memory latency.

The explorer below has four views of an NVIDIA A100: its SM grid and memory system, memory access patterns, warp execution, and kernel launch configuration. Pick a view, then select an SM, switch the access pattern, toggle divergence, or move the launch sliders to see occupancy change.

NVIDIA A100 architecture
View

108 Streaming Multiprocessors

L2 Cache (40 MB)
MC0
MC1
MC2
MC3
MC4
MC5
HBM0
HBM1
HBM2
HBM3
HBM4
SML2 cacheMemory controllerHBM stack

SM 0 partitions

Partition 0
Partition 1
Partition 2
Partition 3

Each partition contains 16 CUDA cores (64 total per SM)

GPU Memory Hierarchy Overview

Modern GPUs feature a complex memory hierarchy designed to maximize throughput for parallel workloads. Understanding this hierarchy is crucial for achieving peak performance in GPU applications. One powerful technique for reducing memory traffic across these levels is kernel fusion, which combines multiple operations into a single GPU kernel to avoid redundant reads and writes.

Memory Types and Characteristics

1. Registers (Fastest, Private)

text
Location: On-chip, per-thread
Size: 256 KB per SM (65,536 × 32-bit)
Latency: 0 cycles (immediate)
Bandwidth: ~8 TB/s
Scope: Private to each thread

Key Points:

  • Fastest storage in GPU
  • Compiler-managed allocation
  • Spilling to local memory impacts performance
  • Limited to 255 registers per thread

2. Shared Memory (Fast, Shared within Block)

text
Location: On-chip, per SM
Size: 48-228 KB per SM (configurable)
Latency: ~20-30 cycles
Bandwidth: ~4 TB/s
Scope: Shared within thread block

Optimization Strategies:

  • Use for data reuse within blocks
  • Avoid bank conflicts (32 banks, 4-byte width)
  • Implement tiling algorithms
  • Coordinate thread access patterns

3. L1 Cache (Fast, Automatic)

text
Location: On-chip, per SM
Size: 128 KB (combined with shared memory)
Latency: ~30-40 cycles
Bandwidth: ~4 TB/s
Scope: Per SM, caches global/local memory

Configuration Options:

C++
// Configure L1/Shared split
cudaFuncSetCacheConfig(kernel, cudaFuncCachePreferShared);  // More shared memory
cudaFuncSetCacheConfig(kernel, cudaFuncCachePreferL1);      // More L1 cache
cudaFuncSetCacheConfig(kernel, cudaFuncCachePreferEqual);   // Balanced

4. L2 Cache (Medium, GPU-wide)

text
Location: On-chip, shared across GPU
Size: 40-60 MB (A100: 40 MB, H100: 50 MB)
Latency: ~200 cycles
Bandwidth: ~2-3 TB/s
Scope: All SMs, coherent

Features:

  • Caches all memory accesses
  • Persistent across kernel launches
  • Supports atomic operations
  • Hardware-managed coherency

5. Global Memory (Slow, Large)

text
Location: Off-chip DRAM/HBM
Size: 24-80 GB (HBM2e/HBM3)
Latency: ~400-600 cycles
Bandwidth: 1-2 TB/s
Scope: Entire GPU and host

Memory Coalescing Patterns

Coalesced Access (Optimal)

C++
// Perfect coalescing - consecutive threads access consecutive addresses
__global__ void coalesced_access(float* data) {
    int tid = blockIdx.x * blockDim.x + threadIdx.x;
    float value = data[tid];  // Thread 0→data[0], Thread 1→data[1], etc.
}

Strided Access (Poor)

C++
// Strided pattern - wastes bandwidth
__global__ void strided_access(float* data, int stride) {
    int tid = blockIdx.x * blockDim.x + threadIdx.x;
    float value = data[tid * stride];  // Non-consecutive access
}

Random Access (Worst)

C++
// Random pattern - serialized transactions
__global__ void random_access(float* data, int* indices) {
    int tid = blockIdx.x * blockDim.x + threadIdx.x;
    float value = data[indices[tid]];  // Unpredictable pattern
}

Shared Memory Optimization

Bank Conflict Resolution

C++
// Read patterns for one warp, 32 banks, 4-byte words.
// Assume the arrays have already been initialized and synchronized.
int lane = threadIdx.x % 32;
__shared__ float words[32 * 32];
float conflict = words[32 * lane];  // 32 distinct words, same bank: 32-way
float broadcast = words[lane % 4]; // Four words in distinct banks: broadcasts

// A transpose reads columns of a tile. Padding changes the row stride.
__shared__ float tile[32][33];
float column = tile[lane][0];      // stride 33: one distinct bank per lane

Matrix Transpose with Shared Memory

C++
#define TILE_DIM 32
#define BLOCK_ROWS 8

__global__ void transpose_coalesced(float *odata, float *idata, int width, int height) {
    __shared__ float tile[TILE_DIM][TILE_DIM + 1];  // +1 padding avoids bank conflicts

    int x = blockIdx.x * TILE_DIM + threadIdx.x;
    int y = blockIdx.y * TILE_DIM + threadIdx.y;

    // Coalesced read from global memory
    for (int j = 0; j < TILE_DIM; j += BLOCK_ROWS) {
        if (x < width && (y + j) < height) {
            tile[threadIdx.y + j][threadIdx.x] = idata[(y + j) * width + x];
        }
    }

    __syncthreads();

    // Transpose indices for write
    x = blockIdx.y * TILE_DIM + threadIdx.x;
    y = blockIdx.x * TILE_DIM + threadIdx.y;

    // Coalesced write to global memory
    for (int j = 0; j < TILE_DIM; j += BLOCK_ROWS) {
        if (x < height && (y + j) < width) {
            odata[(y + j) * height + x] = tile[threadIdx.x][threadIdx.y + j];
        }
    }
}

Memory Access Optimization Techniques

1. Texture Memory (Spatial Locality)

C++
// Texture memory for 2D spatial locality
texture<float, cudaTextureType2D> texRef;

__global__ void texture_kernel(float* output, int width, int height) {
    int x = blockIdx.x * blockDim.x + threadIdx.x;
    int y = blockIdx.y * blockDim.y + threadIdx.y;

    // Hardware-optimized 2D cache
    output[y * width + x] = tex2D(texRef, x, y);
}

2. Constant Memory (Broadcast)

C++
__constant__ float const_data[1024];  // 64 KB constant memory

__global__ void const_kernel(float* output) {
    int tid = threadIdx.x;
    // All threads read same value - broadcast optimization
    output[tid] = const_data[0] * tid;
}

3. Unified Memory (Simplified Programming)

C++
// Automatic migration between host and device
float *data;
cudaMallocManaged(&data, size);

// Access from both CPU and GPU
data[0] = 1.0f;  // CPU write
kernel<<<grid, block>>>(data);  // GPU access
cudaDeviceSynchronize();
float result = data[0];  // CPU read

Performance Analysis Tools

Memory Bandwidth Calculation

text
Effective Bandwidth (GB/s) = (Bytes Read + Bytes Written) / Elapsed Seconds / 1e9
Efficiency = (Effective Bandwidth / Theoretical Bandwidth) × 100%

Requested read/write bytes define effective bandwidth; memory-sector counters describe actual traffic and can expose wasted transfers. Occupancy is a separate latency-hiding metric, already reflected in the elapsed time, so it is not a bandwidth multiplier. See NVIDIA’s effective bandwidth definition.

Profiling Metrics

bash
# NSight Compute memory metrics
ncu --metrics l1tex__t_bytes_pipe_lsu_mem_global_op_ld.sum,
               l1tex__t_bytes_pipe_lsu_mem_global_op_st.sum,
               smsp__sass_average_data_bytes_per_sector_mem_global_op_ld.pct,
               smsp__sass_average_data_bytes_per_sector_mem_global_op_st.pct
               ./application

Memory Optimization Checklist

Design Phase

  • Identify memory access patterns
  • Calculate memory bandwidth requirements
  • Plan data layout for coalescing
  • Determine shared memory usage

Implementation Phase

  • Implement coalesced access patterns
  • Use shared memory for data reuse
  • Avoid bank conflicts with padding
  • Minimize register pressure
  • Use appropriate memory types (texture, constant)

Optimization Phase

  • Profile memory transactions
  • Analyze bandwidth utilization
  • Identify and fix inefficient patterns
  • Tune L1/shared memory configuration
  • Consider memory prefetching

Advanced Memory Patterns

Warp-level Memory Operations

C++
// Warp shuffle for register-level data exchange
__global__ void warp_reduce(float* data) {
    float value = data[threadIdx.x];

    // Warp-level reduction without shared memory
    for (int mask = 16; mask > 0; mask >>= 1) {
        value += __shfl_xor_sync(0xffffffff, value, mask);
    }

    if (threadIdx.x % 32 == 0) {
        data[threadIdx.x / 32] = value;
    }
}

Memory-level Parallelism

C++
// Increase memory-level parallelism with unrolling
__global__ void mlp_kernel(float* __restrict__ out,
                           const float* __restrict__ in,
                           int n) {
    int tid = blockIdx.x * blockDim.x + threadIdx.x;
    int stride = gridDim.x * blockDim.x;

    // Process multiple elements per thread
    float sum0 = 0, sum1 = 0, sum2 = 0, sum3 = 0;

    for (int i = tid; i < n/4; i += stride) {
        // Issue multiple loads simultaneously
        float4 val = reinterpret_cast<const float4*>(in)[i];
        sum0 += val.x;
        sum1 += val.y;
        sum2 += val.z;
        sum3 += val.w;
    }

    out[tid] = sum0 + sum1 + sum2 + sum3;
}

Best Practices Summary

Do's

  • Coalesce global memory accesses
  • Use shared memory for data reuse
  • Pad shared memory arrays to avoid bank conflicts
  • Batch small transfers into larger ones
  • Use vector loads (float2, float4) when possible
  • Profile and measure actual bandwidth

Don'ts

  • Random memory access patterns
  • Excessive register usage causing spills
  • Unaligned memory accesses
  • Frequent host-device transfers
  • Ignoring memory hierarchy
  • Assuming CPU optimization strategies work on GPU

Performance Impact

Bandwidth Utilization by Pattern

For one 32-thread warp reading 4-byte words from an aligned base on NVIDIA compute capability 6.0+, count distinct 32-byte sectors. These are analytical traffic values, not measured bandwidth or latency. Permuting lanes within the same address set does not change the sector count.

Access patternSector utilization (unique bytes / sector bytes)32-byte sectors
Consecutive words100%4
Consecutive, offset by one word80%5
Stride 2 words50%8
Stride 32 words12.5%32
Scattered, one word per sector12.5%32
Broadcast of one word12.5% (same word reused by all lanes)1

Conclusion

GPU memory hierarchy optimization is essential for achieving peak performance. Focus on:

  1. Coalesced access patterns for global memory
  2. Shared memory for data reuse and communication
  3. Avoiding bank conflicts through padding
  4. Proper occupancy to hide memory latency
  5. Profiling and measurement to validate optimizations

Master these concepts to unlock the full potential of GPU computing and achieve order-of-magnitude performance improvements in your applications.

GPU & High-Performance Computing
GPU Pipeline Hazards: Scoreboard Stalls and Warp Divergence

A fine kernel can still profile poorly: warp divergence taxes throughput ~1/N; alone on a long scoreboard wait, SM issue util can hit 0%. Interactive demos.

GPU & High-Performance Computing
GPU Streaming Multiprocessor (SM)

How a single SM actually runs work: warps of 32, cutaway of cores and memory, divergence tax, latency-hiding occupancy, and coalesced loads — instruments, not a catalog.

Language & Framework Internals
Pinned Memory and DMA Transfers in PyTorch

How page-locked memory changes CPU-to-GPU transfers, the extra staging in pageable paths, and the conditions needed to overlap copies with GPU work.

Systems & Architecture
SoA vs AoS: Data Layout Optimization

Structure of Arrays vs Array of Structures as instruments: cache-line fill, SIMD gather vs contiguous load, GPU coalescing, and AoSoA hybrids — when layout is a 10× decision.

GPU & High-Performance Computing
CUDA Contexts: Ownership, Current Stack, Isolation

Deep dive into the CUDA context object: control vs data plane, inventory (memory, modules, streams, events, graphs), push/pop/setCurrent stacks, primary retain/release, flags and limits, isolation, cost, and traps.

GPU & High-Performance Computing
CUDA Context vs Streams vs MPS: Which Layer Fixes What

Decision map for CUDA: a context is per-process GPU state, a stream is an in-order queue inside a context, and MPS co-schedules work across client contexts on Volta+. Pick the layer that matches the problem.

If you found this explanation helpful, consider sharing it with others.

Mastodon