NUMA Architecture: Non-Uniform Memory Access

What is NUMA?

NUMA (Non-Uniform Memory Access) is a memory design for multi-socket machines where latency depends on which socket owns the DRAM. A core can reach its own node's memory faster than memory attached to another socket.

That asymmetry is the whole story. Multi-socket servers and HPC boxes scale memory bandwidth by giving each socket a local controller — at the cost of a slower remote path over QPI, UPI, or Infinity Fabric.

Explore the topology

Local access stays on-socket (~90 ns). Remote access crosses QPI/UPI (~300 ns). Pair a core with a memory bank on either node to see which path you get.

Local vs remote path

Each load is a chain of hops. Local stays on the home socket. Remote pays for the interconnect before the far memory controller answers.

Placement policy

Same eight threads under three policies — aware (all local), unaware (half remote), and first-touch (pages piled on node 0).

Anatomy of a NUMA node

Each node typically includes:

• CPU socket — one or more processors and their cores
• Local DRAM — memory banks on that socket's controller
• Memory controller — schedules access to local banks
• Local I/O — PCIe roots and devices hung off this socket

Inter-socket interconnect

Modern links (illustrative peak rates):

• Intel QPI — up to ~25.6 GB/s
• Intel UPI — up to ~41.6 GB/s
• AMD Infinity Fabric — up to ~50 GB/s per link

These make remote memory possible, not free. Bandwidth is lower than local DDR, and contention from other cross-socket traffic hurts further.

Performance characteristics

Latency ladder

Bandwidth

Local memory can sustain full channel bandwidth (e.g. 100+ GB/s with multi-channel DDR). Remote traffic is capped by:

1. Interconnect bandwidth (often 25–50 GB/s)
2. Contention from other cross-socket streams
3. Coherence / protocol overhead

NUMA-aware programming

Best practices

1. Thread affinity — pin workers near their data

// Linux: sched_setaffinity
cpu_set_t cpuset;
CPU_ZERO(&cpuset);
CPU_SET(core_id, &cpuset);
sched_setaffinity(0, sizeof(cpuset), &cpuset);

2. Node-local allocation — place pages on the right node

// libnuma
numa_alloc_onnode(size, node_id);

3. First-touch — the first write decides the home node

#pragma omp parallel
{
    int thread_id = omp_get_thread_num();
    initialize_local_data(data[thread_id]); // touch on the right node
}

Common pitfalls

1. False sharing — cores on different nodes bouncing the same cache line
2. Remote allocation — one thread allocates for everyone
3. Thread migration — OS moves a thread off its data
4. Imbalanced memory — one node fills while others sit empty

Where NUMA shows up

Server shapes

• 2-socket — most common; clean dual-node story
• 4-socket — richer topology, more hop variance
• 8+ socket — HPC; multi-hop NUMA and careful placement

Cloud and virtualization

• VM topology — guests should mirror host NUMA boundaries
• Containers — Kubernetes / Docker NUMA-aware scheduling
• Automatic balancing — kernel NUMA balancing can help or surprise; know when to disable it

Tools

# Topology
numactl --hardware

# Per-node stats
numastat
numastat -m

# Perf counters
perf stat -e node-loads,node-load-misses ./application

# Intel PCM
pcm-numa

NUMA vs memory interleaving

Interleaving spreads addresses across banks/channels inside a node for bandwidth. NUMA spreads memory across nodes for capacity and socket-local bandwidth. Production systems usually do both:

1. Intra-node — channel interleaving
2. Inter-node — NUMA domains
3. Hybrid — some firmware can interleave across nodes (latency trade-off)

When it matters most

• Databases — partition data by node
• Scientific codes — domain decomposition aligned with topology
• Analytics — shard working sets with the hardware
• Hypervisors — place VMs on intact NUMA domains

Wrapping up

NUMA is how multi-socket machines stay scalable: local memory is fast and wide; remote memory works but costs ~3× latency and less bandwidth. The figures above encode the three habits that matter — see the topology, respect the path, place threads with their pages.

Related concepts