Python Shared Memory

What shared memory changes

Normal multiprocessing sends data between isolated address spaces: a Queue or Pipe pickles the object, copies the bytes through an IPC channel, and rebuilds it on the other side. multiprocessing.shared_memory changes the data path — the OS owns one named byte region and every process maps that same region into its own address space, so reads and writes land on the same underlying bytes. The explorer below measures exactly what that buys.

The trade is sharp. Shared memory skips serialization and copying for the payload, but it does not make Python objects magically shared: you decide the byte layout, synchronize writes, and own the cleanup. The three patterns below are the ones that matter.

Share a NumPy array

A SharedMemory block is raw bytes, and a NumPy ndarray can use shm.buf as its storage directly. The name alone is not enough — pass the shape and dtype alongside it, and size the block with array.nbytes:

import numpy as np
from multiprocessing import shared_memory

arr = np.arange(12, dtype=np.float64).reshape(3, 4)
shm = shared_memory.SharedMemory(create=True, size=arr.nbytes)
shared = np.ndarray(arr.shape, dtype=arr.dtype, buffer=shm.buf)
shared[:] = arr                                   # write once

# another process, handed only shm.name + shape + dtype:
attached = shared_memory.SharedMemory(name=shm.name)
view = np.ndarray(arr.shape, dtype=arr.dtype, buffer=attached.buf)  # zero-copy

shared and view are different Python objects over the same buffer. Every attached process calls close() on its own handle; exactly one owner calls unlink() to delete the block.

Write safely

Shared memory shares bytes, not atomic operations. A read-modify-write has three phases — read, compute, write back — and if two processes interleave them, an update vanishes. Wrap the full critical section in a lock:

from multiprocessing import Lock

lock = Lock()
def safe_increment(shm, lock):
    with lock:                                    # the whole read-modify-write
        v = int.from_bytes(shm.buf[0:4], "little")
        shm.buf[0:4] = (v + 1).to_bytes(4, "little")

For bulk numerical work, partitioning is faster than locking: give each worker a non-overlapping slice (image[start:stop]) so no two processes ever write the same bytes, and drop the lock entirely.

Lifetime: close vs. unlink

close() releases this process's mapping; unlink() deletes the block and must be called once, by the owner. Scatter that responsibility and you get either a leak (no one unlinks) or a use-after-free (someone unlinks early). When ownership is messy, let a manager own it:

from multiprocessing.managers import SharedMemoryManager

with SharedMemoryManager() as smm:
    shm = smm.SharedMemory(size=1024)   # the manager unlinks on context exit
    ...

Python 3.13 added track= to SharedMemory: track=True registers the block with the resource tracker (right within a multiprocessing family), but independent process trees each get their own tracker — there, use a single owner, a SharedMemoryManager, or track=False. On Windows track is ignored, since the OS deletes the block once all handles close.

Reference

Common pitfalls

API reference

Related concepts