Python Shared Memory

What Shared Memory Changes

Normal multiprocessing sends data between isolated process address spaces. A queue or pipe usually pickles the Python object, copies bytes through an IPC channel, and rebuilds an object on the other side.

multiprocessing.shared_memory changes the data path. The operating system owns one named byte region, and each Python process maps that same region into its own address space. The mapping addresses can differ, but reads and writes land on the same underlying bytes.

That trade is powerful and sharp. Shared memory avoids IPC serialization and copying for payload bytes once they are in the shared block, but it does not make Python objects magically shared. You still decide the byte layout, synchronize writes, and clean up lifetime.

Figure: Shared Memory Recipe Notebook

Create and attach

Use this when one process needs to publish a small byte payload for another process to read without queue serialization.

Step 1 of 4Create exact bytes

Process handles

Process Aowner

create=True, size=len(message)

Shared byte region

empty

...

empty

Block exists

The name can now be shared with another process.

Create exact bytes

Process A asks the OS for a named block sized exactly for the payload.

Recipe code

from multiprocessing import shared_memory

message = b"Hello from Process A!"
shm = shared_memory.SharedMemory(create=True, size=len(message))

try:
    shm.buf[:len(message)] = message

    # Pass shm.name to another process.
    reader = shared_memory.SharedMemory(name=shm.name)
    try:
        received = bytes(reader.buf[:len(message)])
        print(received.decode("utf-8"))
    finally:
        reader.close()
finally:
    shm.close()
    shm.unlink()

Safety checks

Allocate exact bytes

The block size is len(message), not a guessed constant.

Pass the generated name

The reader attaches by shm.name; it does not need a fixed name.

Close every handle

Both owner and reader close their process-local mappings.

Unlink once

Only the owner requests deletion of this shared block.

Why this works

SharedMemory gives each process a local handle to the same OS-owned bytes. The bytes still have to be written once, but after that the reader attaches by name and reads the shared payload directly.

Recipe 1: Create and Attach

Start with the smallest useful pattern: one owner creates a block, writes a byte payload, passes the generated name to another process, and unlinks the block once. Let Python generate the name unless you have a specific coordination protocol for fixed names.

from multiprocessing import shared_memory

message = b"Hello from Process A!"
shm = shared_memory.SharedMemory(create=True, size=len(message))

try:
    shm.buf[:len(message)] = message

    # Pass shm.name to another process.
    reader = shared_memory.SharedMemory(name=shm.name)
    try:
        received = bytes(reader.buf[:len(message)])
        print(received.decode("utf-8"))
    finally:
        reader.close()
finally:
    shm.close()
    shm.unlink()

The name is only an attachment handle. The shared region itself is raw bytes, so every reader needs the same length and encoding agreement that the writer used. Every attached process calls close() for its own handle; exactly one owner calls unlink() for the block.

NumPy arrays work well with shared memory because an ndarray can use shm.buf as its storage. The shared memory name is not enough by itself. Pass the shape and dtype along with the name, and allocate the exact number of bytes with array.nbytes.

import numpy as np
from multiprocessing import shared_memory

array = np.arange(12, dtype=np.float64).reshape(3, 4)
shm = shared_memory.SharedMemory(create=True, size=array.nbytes)

try:
    shared = np.ndarray(array.shape, dtype=array.dtype, buffer=shm.buf)
    shared[:] = array

    worker_shm = shared_memory.SharedMemory(name=shm.name)
    try:
        worker_view = np.ndarray(
            array.shape,
            dtype=array.dtype,
            buffer=worker_shm.buf,
        )
        worker_view[0, :] *= 2
    finally:
        worker_shm.close()

    result = shared.copy()
finally:
    shm.close()
    shm.unlink()

shared and worker_view are different Python objects. They share the same data buffer, but each process constructs its own metadata. Copy the final result before closing if the caller needs a normal array that outlives the shared memory handle.

Recipe 3: Write Safely

Shared memory shares bytes, not atomic operations. A read-modify-write update has three phases: read the old value, compute the new value, and write it back. If two processes interleave those phases, one update can disappear.

Use a lock around the full critical section when workers may touch the same bytes.

from multiprocessing import Lock, Process, shared_memory

def safe_increment(name, lock, count):
    shm = shared_memory.SharedMemory(name=name)
    try:
        for _ in range(count):
            with lock:
                value = int.from_bytes(shm.buf[0:4], "little")
                value += 1
                shm.buf[0:4] = value.to_bytes(4, "little")
    finally:
        shm.close()

shm = shared_memory.SharedMemory(create=True, size=4)
try:
    shm.buf[0:4] = (100).to_bytes(4, "little")
    lock = Lock()

    workers = [
        Process(target=safe_increment, args=(shm.name, lock, 1)),
        Process(target=safe_increment, args=(shm.name, lock, 1)),
    ]
    for worker in workers:
        worker.start()
    for worker in workers:
        worker.join()

    failures = [worker.exitcode for worker in workers if worker.exitcode != 0]
    if failures:
        raise RuntimeError(f"worker failure exit codes: {failures}")

    print(int.from_bytes(shm.buf[0:4], "little"))  # 102
finally:
    shm.close()
    shm.unlink()

For bulk numerical work, partitioning is often faster than locking. Give each worker a non-overlapping slice, then no two processes write the same bytes.

Recipe 4: Own Cleanup

close() and unlink() solve different problems. close() releases this process's mapping or handle. unlink() requests deletion of the shared memory block and should be called once per block by the owner.

track was added to SharedMemory in Python 3.13. On Unix-like systems, track=True registers the block with Python's resource tracker. Processes created through multiprocessing usually share the same tracker. Independent Python processes or subprocess trees can each get their own tracker; with track=True, the first process to exit may remove a block another process still expects. In that case, use a single owner, use SharedMemoryManager, or set track=False when another process is already responsible for cleanup.

On Windows, track is ignored because Windows deletes shared memory after all handles are closed.

from multiprocessing.managers import SharedMemoryManager
from multiprocessing import shared_memory

with SharedMemoryManager() as smm:
    shm = smm.SharedMemory(size=1024)
    name = shm.name

    participant = shared_memory.SharedMemory(name=name, track=True)
    try:
        participant.buf[0:5] = b"ready"
    finally:
        participant.close()

# The manager owns unlinking when the context exits.

Use a manager when ownership would otherwise be scattered across processes. Use manual close()/unlink() when ownership is simple and explicit.

Compact Reference

Common Pitfalls and Solutions

Pitfall	Problem	Fix
No synchronization	Two processes can read the same old value and overwrite each other.	Use a Lock/RLock/Condition, or partition memory so writers never touch the same bytes.
Leaking blocks	A POSIX shared memory block can outlive the creating process.	Call close() in every process and unlink() once when the block is no longer needed.
Independent resource trackers	Standalone Python processes can each create a tracker; the first exit may unlink the block.	Use SharedMemoryManager, a single owner process, or track=False when another owner handles cleanup.
Wrong byte size	The NumPy shape and dtype may require more bytes than the block contains.	Allocate array.nbytes or prod(shape) * dtype.itemsize and pass shape/dtype with the name.
Using the buffer after close	The memoryview becomes invalid for that process.	Copy any result you need before closing the SharedMemory handle.

Pitfall

No synchronization

Problem

Two processes can read the same old value and overwrite each other.

Fix

Use a Lock/RLock/Condition, or partition memory so writers never touch the same bytes.

Pitfall

Leaking blocks

Problem

A POSIX shared memory block can outlive the creating process.

Fix

Call close() in every process and unlink() once when the block is no longer needed.

Pitfall

Independent resource trackers

Problem

Standalone Python processes can each create a tracker; the first exit may unlink the block.

Fix

Use SharedMemoryManager, a single owner process, or track=False when another owner handles cleanup.

Pitfall

Wrong byte size

Problem

The NumPy shape and dtype may require more bytes than the block contains.

Fix

Allocate array.nbytes or prod(shape) * dtype.itemsize and pass shape/dtype with the name.

Pitfall

Using the buffer after close

Problem

The memoryview becomes invalid for that process.

Fix

Copy any result you need before closing the SharedMemory handle.

API Reference

API	Purpose	Detail
SharedMemory(...)	Create or attach to a named shared byte block.	Python 3.13 signature is SharedMemory(name=None, create=False, size=0, *, track=True).
shm.buf	memoryview over the shared bytes.	Use explicit slicing and encoding/decoding; do not access after close().
shm.name	Generated or user-provided identifier for the block.	Pass this value to other processes so they can attach to the same bytes.
shm.size	Total bytes available through this handle.	May be rounded up by the operating system; still size arrays explicitly.
shm.close()	Release this process mapping/handle.	Call it in every process when that process is done with the block.
shm.unlink()	Request deletion of the shared memory block.	Call once per block. On Windows, deletion happens when all handles close.
SharedMemoryManager	Own shared blocks from a manager process.	Use it when manual lifetime ownership would be fragile.

API

SharedMemory(...)

Purpose

Create or attach to a named shared byte block.

Detail

Python 3.13 signature is SharedMemory(name=None, create=False, size=0, *, track=True).

API

shm.buf

Purpose

memoryview over the shared bytes.

Detail

Use explicit slicing and encoding/decoding; do not access after close().

API

shm.name

Purpose

Generated or user-provided identifier for the block.

Detail

Pass this value to other processes so they can attach to the same bytes.

API

shm.size

Purpose

Total bytes available through this handle.

Detail

May be rounded up by the operating system; still size arrays explicitly.

API

shm.close()

Purpose

Release this process mapping/handle.

Detail

Call it in every process when that process is done with the block.

API

shm.unlink()

Purpose

Request deletion of the shared memory block.

Detail

Call once per block. On Windows, deletion happens when all handles close.

API

SharedMemoryManager

Purpose

Own shared blocks from a manager process.

Detail

Use it when manual lifetime ownership would be fragile.

Real-World Example: Partitioned Image Processing

This example avoids locks during pixel processing by giving each worker a non-overlapping region. The only shared output is the image buffer itself; each process owns a row range.

import numpy as np
from multiprocessing import Process, shared_memory

def process_rows(name, shape, dtype, start, stop):
    shm = shared_memory.SharedMemory(name=name)
    try:
        image = np.ndarray(shape, dtype=dtype, buffer=shm.buf)
        block = image[start:stop]
        gray = (
            0.299 * block[..., 0]
            + 0.587 * block[..., 1]
            + 0.114 * block[..., 2]
        ).astype(dtype)
        block[..., 0] = gray
        block[..., 1] = gray
        block[..., 2] = gray
    finally:
        shm.close()

def parallel_grayscale(image, workers=4):
    shm = shared_memory.SharedMemory(create=True, size=image.nbytes)
    try:
        shared = np.ndarray(image.shape, dtype=image.dtype, buffer=shm.buf)
        shared[:] = image

        rows = np.linspace(0, image.shape[0], workers + 1, dtype=int)
        processes = [
            Process(
                target=process_rows,
                args=(shm.name, image.shape, image.dtype, rows[i], rows[i + 1]),
            )
            for i in range(workers)
        ]

        for process in processes:
            process.start()
        for process in processes:
            process.join()

        failures = [p.exitcode for p in processes if p.exitcode != 0]
        if failures:
            raise RuntimeError(f"worker failure exit codes: {failures}")

        return shared.copy()
    finally:
        shm.close()
        shm.unlink()

Partitioning work so writers never touch the same bytes is the fastest way to use shared memory safely. If workers need to update shared counters, queues, or overlapping regions, add synchronization around those specific operations.