Skip to main content

CUDA Contexts: Ownership, Current Stack, Isolation

Deep dive into the CUDA context object: control vs data plane, inventory (memory, modules, streams, events, graphs), push/pop/setCurrent stacks, primary retain/release, flags and limits, isolation, cost, and traps.

You got a pointer. Nobody said context.

You call cudaSetDevice(0) and cudaMalloc. A device pointer comes back. Nothing in that call site says context, yet the driver just retained a primary CUDA context for process × GPU 0: host-side tables, device virtual address space, default stream, and a place to hang every later module, event, and graph.

Most production bugs that look like “random CUDA errors” are really wrong current context, unbalanced push/pop, handle used after destroy, or cross-context pointer. This page is that machinery — what the context is, what lives inside it, and how threads bind to it.

Two numbers

1. Current is per CPU thread
Process-wide “we initialized CUDA” is not enough. Each thread has its own current context (often via a stack). A context-requiring Driver API call such as cuMemAlloc fails with CUDA_ERROR_INVALID_CONTEXT if none is current. A Runtime API call such as cudaMalloc instead initializes and binds the selected/default device’s primary context. Call cudaSetDevice on workers to choose the intended device.

2. Handles do not cross contexts
A cudaMalloc pointer, stream, module, event, or graph is inventory of one context. Same physical GPU, different context → same bit pattern can be invalid. Multi-process default = one private context each (time-sliced) unless MPS enables concurrent scheduling. On Volta and newer, MPS clients keep their own contexts and GPU address spaces.

Helpful background

NVIDIA device files

Identify control, GPU and UVM device nodes and inspect the access a workload needs on hosts and in containers.

Open full page

What a context is

A CUDA context (CUcontext) is the synchronized pair:

SideLives onHolds
Control planeHost (driver + your process)API entry, handle tables, current stack, command submission, flags/limits
Data planeDeviceVA space, allocations, loaded code, schedulable work for this owner

One context ↔ one process × one device association (a process can own many contexts across devices; rare code owns several on one device). The opaque handle is not “a pointer into VRAM” — it names the whole bag.

Control plane · data plane
call
CUdeviceptr
handle on host · bytes on device · hop 1/6
cudaMalloc(&p, n);
CPU control plane
host · driver tables
handlesmodulesstreams

API entry, current stack, handle tables, command submit

ctxResolve current context on this thread
CUcontext
bridge
process × GPU
GPU data plane
device · VA + work
no map yet

VA space, allocations, code pages, SM / DMA work

waiting…

ctxResolve current context on this threadHost keeps the handle; device owns the bytes. Destroy the context and both sides vanish.

One context is the synchronized pair: host tables + device maps for one process on one device. Every CUDA call rides that bridge — flip the call and step the hops.

Inside the context object

When docs say “resources associated with the context,” they mean a concrete inventory. Destroy frees almost all of it. Flip shelves:

Inside the context object
lifetime
VA space
allocations die with the context · 5 entries in focus
CUcontext · live
CUcontext bagprocess × GPU · 6 classic shelves
click a compartment
memory
  • cuMemAlloc / cudaMallocdevice linear allocations
  • managed / UVMcuMemAllocManaged — often still context-scoped
  • pinned host mapsMAP_HOST-style GPU-visible host mem
  • arrays / mipmapsCUarray, CUmipmappedArray
  • external memoryCUexternalMemory imports into this world

Device pointers are addresses in *this* context’s space. Same bits under another context can be invalid.

Click compartments to inspect inventory. Flip to after destroy to see classic shelves empty — while “not in bag” stays outside the wipe.

Memory shelf

  • Device allocations (cuMemAlloc / cudaMalloc, pitched, many managed paths) live in the context’s address space.
  • CUDA arrays / mipmaps for texture paths.
  • External memory / semaphore imports bound into this world.
  • Documented exceptions: some virtual-memory and stream-ordered pool paths have lifetimes that are not simply “die with cuCtxDestroy” — read the destroy notes if you use those APIs.

Code shelf

  • Modules (CUmodule) and functions (CUfunction) from cubin/fatbin loads.
  • Link / library state used during JIT.
  • Device-side dynamic parallelism is gated by context limits (sync depth, pending launch count).

Work shelf

  • Default stream (every context has one; legacy vs per-thread default stream modes change serialization).
  • User streams and events — created under the current context, unusable under another.
  • Graphs and graph execs.
  • cuCtxSynchronize waits for all outstanding work in the (current or named) context — blunter than a single stream sync.

Policy shelf

Flags at create time, limits (cuCtxSetLimit), cache config, stream priority range, and on newer stacks execution affinity (e.g. SM count under MPS). These are properties of the context object, not of a single kernel launch (kernels can still override some cache prefs).

Current context and the stack

CUDA routes almost every API call to the current context of the calling thread.

APIWhat it does
cudaSetDevice(i)Runtime: make device i’s primary context current (retain on first use)
cuCtxGetCurrentRead current (NULL if floating)
cuCtxSetCurrent(ctx)Bind ctx as current; NULL clears current
cuCtxPushCurrent(ctx)Push onto this thread’s stack; top becomes current
cuCtxPopCurrentPop top; previous current restored
cuDevicePrimaryCtxRetainBump primary refcount — does not make it current by itself

Empty stack / NULL current → invalid context for context-requiring Driver API calls. The Runtime can initialize and bind a primary context in this case. Two threads can both hold the same context current — isolation is by context, not by thread.

Current context stack
B
T0 after cuCtxPushCurrent(B)
ok · uses ctx B (top of stack)
call from T0
thread T0call
ctx A · GPU0
ctx B · GPU0current
thread T1
ctx A · GPU0current

Push makes B current without destroying A. Pop restores A. Nested libraries love this.

Driver API: cuCtxPushCurrent / cuCtxPopCurrent · Runtime usually hides this behind the primary context.

Push, pop, set — the film

Libraries that use the Driver API almost always follow: push my context → do work → pop. That restores the caller’s current context. Unbalanced push is how “works until library X returns” bugs ship.

Push · pop · set current
current · ∅
beat 1/7 · —
thread stack (bottom → top = current)
empty · no current
Current-context Driver API example
// thread T0 · empty stackcuCtxGetCurrent(&c);  // c == NULLcuMemAlloc(&p, n);    // INVALID_CONTEXT

No current context: context-requiring Driver calls such as cuMemAlloc fail. Runtime cudaMalloc can instead initialize and bind the selected/default primary context.

Runtime apps mostly call cudaSetDevice (primary current). Push/pop is how Driver API libraries borrow the GPU without permanently stealing the caller’s current context.

c
// Canonical library sandwich
CUcontext caller;
cuCtxGetCurrent(&caller);          // optional: know what you restore
cuCtxPushCurrent(myCtx);
// ... malloc, launch, sync under myCtx ...
cuCtxPopCurrent(NULL);             // restore previous current

// Primary retain pitfall
CUcontext p;
cuDevicePrimaryCtxRetain(&p, dev); // retained, NOT current
cuCtxSetCurrent(p);                // now current — or use cudaSetDevice

cuCtxCreate note: creating a context associates it with the calling thread and makes it current (usage count 1). You must eventually cuCtxDestroy. Destroy pops it from the calling thread if it is current there; other threads that still list it as current see CUDA_ERROR_CONTEXT_IS_DESTROYED on later use.

cuCtxSetCurrent vs push: use push/pop when you must restore the caller. Use set when you own the thread (worker pinned to one context). Pass NULL to float (no current).

Framework code (PyTorch, JAX, etc.) almost always stays on primary contexts and cudaSetDevice. Set the device on every pool worker before its first CUDA call so it selects the intended GPU; an unbound Runtime call can otherwise select the default device.

Primary context vs explicit create

Primary (Runtime)Explicit (Driver cuCtxCreate)
Who createsRuntime on first use per deviceYou
Retain APIcuDevicePrimaryCtxRetain / Runtime first callcreate returns handle
Typical appsNearly all ML / CUDA Runtime codeEngines, multi-ctx tools, plugins
LifetimeRefcounted until last release / process teardowncuCtxDestroy frees inventory
Mental model“This process on GPU i”“This library’s private GPU world”

Primary flags (cuDevicePrimaryCtxSetFlags) must be set before the primary is initialized. After retain, you still need current binding.

Primary vs explicit
primary
one lazy context per device per process
  1. 1
    cudaSetDevice(0)
    initialize primary and make current (CUDA 12+)
  2. 2
    first cudaMalloc / launch
    use current context; bind primary if none is current
  3. 3
    more Runtime calls
    reuse primary · stack usually untouched
  4. 4
    process exit
    driver tears down primary

PyTorch, TensorFlow, and most apps live here. You almost never call cuCtxCreate.

Primary context is retained per device. Fork after CUDA init is unsafe — child inherits driver state it cannot use.

Fork rule: do not fork() after CUDA has initialized. The child inherits host state that is not safe to use. Prefer fork-before-init, or separate processes from the start.

Flags, limits, and sync scope

The context is not only a bag of pointers — it carries scheduling flags, resource limits, and cache preferences. Wrong layer again: SCHED_SPIN will not fix a default-stream serialize; stream priority will not fix a cross-context pointer.

Flags, limits, sync
knob
busy-wait
lower latency · burns a core
lives on CUcontext
CPU
GPU
spin burnyieldsleep / idleGPU work

during a host wait (e.g. device/context sync) — pedagogical tape

CU_CTX_SCHED_SPIN: CPU polls until GPU finishes. Great for latency-sensitive single tenants; multi-tenant boxes hate it.

These knobs sit on the context, not on a stream and not on “the GPU” abstractly. SCHED_SPIN will not fix a null-stream serialize; cuCtxSynchronize will not invent overlap.

c
// Limits (current context)
size_t heap;
cuCtxGetLimit(&heap, CU_LIMIT_MALLOC_HEAP_SIZE);
cuCtxSetLimit(CU_LIMIT_MALLOC_HEAP_SIZE, 128ull << 20);

// Blunt sync
cuCtxSynchronize();              // all work in current context
// Prefer when possible:
cuStreamSynchronize(stream);     // one queue

Isolation wall

Contexts isolate logical GPU worlds on the same silicon (even the same process can hold two contexts on one GPU via the Driver API — rare, but the wall still applies):

  • Device pointers are context virtual addresses, not portable physical tokens. Same GPU ≠ same map.
  • Streams, events, modules, graphs, and classic malloc inventory are handles bound to the creating context.
  • Threads may share one context (each must make it current). Processes default to private contexts and ordinary GPU time slicing (DEFAULT compute mode).
  • IPC maps exported memory into the importer’s context (new local pointer). MPS co-schedules clients for SM packing — it does not make raw pointers portable across processes (use IPC for that). MIG is a hardware partition, not a software context share.
Context isolation
attempt
invalid
VA is context-local · same GPU still
Use d_A while B is current (raw bits)
GPU 0Two contexts · one process · one GPU (Driver API dual-ctx or rare setup)
ctx A
owner · cudaMalloc
d_A = 0xA1…
s0
wall
ctx B
same GPU · now current
kernel<<<…>>>(d_A)
address maps (toy · filled = mapped)
ctx A VA
ctx B VA
d_A is not a portable physical address. Under B it is not a valid mapping — API error, illegal access, or silent corruption. Do not treat “same GPU” as “same pointer.”

A CUDA context is the isolation unit for device virtual addresses. Same device, different context → different maps (Robert Crovella / programming guide: context ≈ process analog).

Pointers, streams, modules, events, and graphs are context inventory. Threads may share a context; processes default to private contexts. IPC maps memory explicitly. MPS co-schedules clients — it does not make raw device pointers portable across processes (use IPC for that).

python
# Sketch: two processes, two contexts, no IPC — ptr is not shareable
# process A
d_a = cuda.malloc(n)
# process B cannot use d_a — different context / address space
# (cudaIpcGetMemHandle / OpenMemHandle if you need a local map)

When many small processes should co-reside kernels, prefer MPS (or consolidate into one process with many streams) — not “create more contexts per process.” Shared buffers across processes still need IPC (or a single process).

Cost: create, warm path, multi-process slice

Context setup is not free. First touch can pull in driver open paths (device files), primary retain, and module load/JIT. Steady-state launches are cheap by comparison.

Ordinary multi-process time slicing pays a different tax: switch between private contexts while small kernels leave SMs empty — the problem MPS targets.

Persistence mode / nvidia-persistenced keeps the device warm between jobs; it does not replace your process’s CUDA context, but it cuts cold-open pain for short tasks.

Context cost tape
~100ms+
order-of-magnitude first context (SKU-dependent)
tax slots in tape
7/12
drv
drv
ctx
ctx
ctx
map
map
ok
▸
▸
▸
▸

amber = driver/context setup · teal = useful GPU work

First CUDA touch pays driver load, device open, primary context retain, and module JIT. Not free.

Numbers are pedagogical. Measure first CUDA call and steady-state launch latency on your driver + SKU. Pair short jobs with nvidia-persistenced.

Numbers on the tape are pedagogical. Measure first cuda call vs nth launch on your driver and SKU.

Context vs stream (and multi-GPU)

Scaling units people confuse:

LayerScalesJob
ProcessOS isolationWho talks to the driver
DevicePhysical GPUWhich chip
ContextOwnership / VA / inventoryIsolation boundary
StreamConcurrency lanesOverlap copy/compute inside a context
Context vs stream hierarchy
devices
contexts / GPU
streams
overlap
4 queues inside one primary context
device = which chipcontext = ownershipstream = order / overlap
devices 1contexts 1stream lanes 4

Streams are concurrency lanes inside the context. Same ownership, more orderings — copies and kernels can overlap.

Dial the layer you care about. More streams → overlap inside one bag. More GPUs → more primary contexts. More contexts on one GPU → isolation tax, not free concurrency. Amber stream bar = legacy default (can serialize peers).

Default mental model for one training process on one GPU:

text
process
  └─ GPU 0
       └─ primary context
            ├─ flags · limits · cache config
            ├─ modules / functions
            ├─ device allocations (VA space)
            ├─ default stream
            ├─ stream 0  (H2D)
            ├─ stream 1  (compute)
            └─ stream 2  (D2H)

Multi-GPU: one primary context per device, switch with cudaSetDevice (or a thread pinned per GPU). Peer access and NCCL sit on top of those separate contexts — they do not merge them into one address space.

Deeper stream mechanics: CUDA streams. Side-by-side compare: context vs streams.

Lifecycle (practical)

  1. Open path — process loads libcuda, opens /dev/nvidiactl + device nodes + often UVM.
  2. Bind device — cudaSetDevice / cuCtxCreate / first Runtime call retains primary.
  3. Optional policy — primary flags before init; limits and cache config after current.
  4. Accumulate — mallocs, modules, streams, graphs attach to that context.
  5. Submit — launches and memcopies use current context on the calling thread.
  6. Library borrow — push foreign context, work, pop (never leave unbalanced).
  7. Teardown — process exit, or cuCtxDestroy / primary release / device reset: inventory goes.
bash
# See processes holding the GPU (contexts behind them)
nvidia-smi
# Compute mode matters for multi-process sharing policies
nvidia-smi | grep -i 'Compute Mode'

Traps

Context traps
no CUDA after fork
sticky fix word
trap
Init CUDA, then fork

Child inherits host tables pointing at parent driver state. Almost every CUDA call in the child is undefined or fails.

do this
fork before cuInit / first Runtime call

Or use spawn / separate processes. Never share a live context across fork.

Most production bugs are current-context and lifetime bugs, not “GPU theory.” Check which thread, which device, which context owns the handle.

SymptomCheck first
Driver INVALID_CONTEXT on workerDid this thread make the intended context current? Runtime calls can bind a primary automatically.
INVALID_CONTEXT after library returnUnbalanced push/pop or destroy?
Weird pointer / illegal addressBuffer allocated under a different context or device?
Works in main, dies after forkCUDA init before fork
Multi-process low SM fillPrivate contexts time-sliced — consider MPS or one process
CONTEXT_IS_DESTROYEDDestroy while another thread still had it current
Retain but still invalidPrimary retain without SetCurrent

What to do

  1. Treat context as ownership inventory and stream as ordering — fix isolation bugs before tuning streams.
  2. Call cudaSetDevice on every thread that will issue CUDA work (or push an explicit context).
  3. Prefer one primary context per device per process; avoid dual contexts on one GPU unless a library requires it.
  4. If you push, pop on every path; libraries should not leave a foreign current.
  5. Never fork after CUDA init; size context lifetime to the process or a clear library enter/exit.
  6. For many small multi-process jobs on one GPU, evaluate MPS; keep devices warm with persistenced when jobs are short.

References

GPU & High-Performance Computing
CUDA Context vs Streams vs MPS: Which Layer Fixes What

Decision map for CUDA: a context is per-process GPU state, a stream is an in-order queue inside a context, and MPS co-schedules work across client contexts on Volta+. Pick the layer that matches the problem.

GPU & High-Performance Computing
CUDA Streams: Asynchronous Execution and Concurrency

A CUDA stream is an in-order queue of GPU ops. Overlap H→D, kernels, and D→H across streams — and avoid the legacy default-stream fences between blocking streams in one context.

GPU & High-Performance Computing
CUDA Multi-Process Service (MPS): Concurrent Client Contexts

Why ordinary multi-process CUDA time slicing can leave SMs idle, how Volta+ MPS co-schedules separate client contexts, execution-resource caps, and when to choose MPS or MIG.

GPU & High-Performance Computing
NVIDIA Device Files in /dev/

How /dev/nvidia*, nvidiactl, nvidia-uvm, and DRI nodes map major/minor numbers to the driver, what CUDA opens first, and the minimum mount set for containers.

GPU & High-Performance Computing
NVIDIA vs AMD for Deep Learning: CUDA vs ROCm and the Datacenter Accelerators

NVIDIA vs AMD for deep learning compared at both layers: the CUDA vs ROCm software moat, the microarchitecture (warp vs wavefront, SM vs CU, Tensor vs Matrix Cores), and the datacenter accelerators (H100/H200/B200 vs MI300X/MI325X).

GPU & High-Performance Computing
Flynn's Classification: Taxonomy of Computer Architectures

Flynn's Classification explained — SISD, SIMD, MISD, MIMD with interactive architecture explorer, SIMD evolution from MMX to AMX, branch divergence visualization, and workload-architecture throughput comparison.

If you found this explanation helpful, consider sharing it with others.

Mastodon