Skip to main content

CUDA Context vs Streams vs MPS: Which Layer Fixes What

Decision map for CUDA: a context is per-process GPU state, a stream is an in-order queue inside a context, and MPS co-schedules work across client contexts on Volta+. Pick the layer that matches the problem.

Three names, three different bugs

Someone says “we need more contexts.” Nsight is serial. Another process on the same GPU is stuck waiting. A pointer from GPU 0 blows up after cudaSetDevice(1).

Those are not one problem with three names. A context is the per-device container of GPU state. A stream is an in-order queue inside that container. MPS lets many processes co-execute instead of ordinary context time-slicing. On Volta+, clients keep separate CUDA contexts and GPU address spaces and submit work directly to hardware.

This page is the decision map. Full machinery lives on the sibling pages.

Two rules

1. Streams nest inside contexts
Adding streams never adds contexts. Concurrency within a process is almost always more streams on the primary context — not a second context on the same GPU.

2. Multi-process is not multi-stream
Two processes each get a private primary context by default. Their streams cannot see each other. Sharing means MPS (or MIG), not cudaStreamCreate in both PIDs.

Three layers

Flip the chip for the question each layer answers — device/state, order/overlap, or multi-process share.

Three layers
what order?
in-order queue inside one context

What runs next, and can independent work overlap?

create
cudaStreamCreate · default stream if you pass 0
how many
Often 2–16 per context for overlap
cost
Cheap (µs bookkeeping)
kernel launchesmemcpyseventswait dependencies

Context owns state. Stream owns order. Volta+ MPS co-schedules client contexts with separate address spaces. Touch the layer that matches the problem — not the one with the coolest API name.

The hierarchy

Process owns devices. Each device gets a context (usually one primary under the runtime API). Context owns the stream table. Streams own the operation queues.

What changes across production setups is how many of each layer you create — not the nesting.

Hierarchy
overlap
process → device → context → stream
proc A
GPU 0
ctx · primary
defaults0s1s2

Concurrency unit is the stream. Same context, more lanes — copies and kernels can overlap.

Amber chips on a stream mark the legacy default stream — it can still serialize blocking streams inside that context; cudaStreamNonBlocking streams are exempt. Teal streams are the overlap lanes you usually want.

Context vs stream, one aspect at a time

Big comparison tables hide the throughline. Pick the aspect you are arguing about.

Context vs stream
stream is the unit
context

Does not introduce concurrency by itself

stream

Yes — unit of asynchrony in CUDA

If the question is multi-GPU or isolation, you are on the context row. If it is overlap or Nsight serialization, you are on the stream row.

Which layer for this symptom?

Start from what you see in the profiler or error log.

Which layer?
streams
layer to touch
why

Need copy/compute overlap or concurrent independent kernels — wrong layer is the context.

do

Non-default streams, async memcpy, record events between producers/consumers.

Faster GPU inside one process → streams. Second GPU → context. Many processes on one chip → MPS (or MIG). Wrong layer wastes weeks.

Mixups people ask in the same breath

Default stream ≠ context. Streams do not own VRAM. Two GPUs mean two contexts first. Create order matters.

Common mixups
no
myth

The default stream is the same thing as the context.

truth

Context is the container. Default stream is one queue inside it. Every context has a default stream so bare API calls still work.

Adding streams never adds contexts. Adding contexts almost never fixes a serialization bug that streams would fix.

Where MPS sits

On Volta and newer, MPS clients retain their own CUDA contexts and GPU address spaces and submit directly to hardware. The server coordinates remaining shared scheduling resources outside the kernel submission path. Control/attach communication uses UNIX domain sockets in the configured MPS directory.

Pre-Volta MPS used a server-owned shared GPU address space and server-mediated work submission. That historical topology is different from the modern client-context model above.

Reach for MPS when: small-batch inference replicas, notebook fleets, concurrent workers whose kernels leave SM capacity available. Persistence addresses device initialization between jobs; MPS targets concurrent scheduling.

Skip MPS when: one job already saturates the GPU, each process already pins its own device, or you need hard isolation (look at MIG instead).

Deep dive: CUDA Multi-Process Service.

Traps

Wrong layer is the expensive trap. Dual contexts for “speed,” streams created on the wrong device, and legacy null-stream serialization are the usual sequels.

Layer traps
match the problem
Tuning the wrong abstraction

Chasing multi-context on one GPU because Nsight looks serial — or adding streams hoping multi-process will overlap.

map symptom → layer

In-process overlap → streams. Multi-GPU → contexts. Multi-process share → MPS/MIG.

This page is the decision map. Deeper dives live on the context, streams, and MPS concept pages — fix the layer first, then read the long form.

What to do

  1. In-process overlap → non-default streams, async copies, events — stay on one primary context per device.
  2. Second GPU → cudaSetDevice, then allocate and create streams on that device.
  3. Many processes, one GPU → MPS (throughput share) or MIG (partition), not more streams in each PID.
  4. Nsight serial → hunt default-stream / implicit sync first; do not invent a second context.
  5. Invalid pointer → match current context/device to the inventory that allocated the handle.

Deeper dives

  • CUDA contexts — primary vs current stack, isolation, create cost
  • CUDA streams — default-stream trap, pipelines, pitfalls
  • CUDA MPS — concurrent clients, SM fill, thread caps

References

GPU & High-Performance Computing
CUDA Contexts: Ownership, Current Stack, Isolation

Deep dive into the CUDA context object: control vs data plane, inventory (memory, modules, streams, events, graphs), push/pop/setCurrent stacks, primary retain/release, flags and limits, isolation, cost, and traps.

GPU & High-Performance Computing
CUDA Streams: Asynchronous Execution and Concurrency

A CUDA stream is an in-order queue of GPU ops. Overlap H→D, kernels, and D→H across streams — and avoid the legacy default-stream fences between blocking streams in one context.

GPU & High-Performance Computing
CUDA Multi-Process Service (MPS): Concurrent Client Contexts

Why ordinary multi-process CUDA time slicing can leave SMs idle, how Volta+ MPS co-schedules separate client contexts, execution-resource caps, and when to choose MPS or MIG.

GPU & High-Performance Computing
NVIDIA Device Files in /dev/

How /dev/nvidia*, nvidiactl, nvidia-uvm, and DRI nodes map major/minor numbers to the driver, what CUDA opens first, and the minimum mount set for containers.

GPU & High-Performance Computing
NVIDIA vs AMD for Deep Learning: CUDA vs ROCm and the Datacenter Accelerators

NVIDIA vs AMD for deep learning compared at both layers: the CUDA vs ROCm software moat, the microarchitecture (warp vs wavefront, SM vs CU, Tensor vs Matrix Cores), and the datacenter accelerators (H100/H200/B200 vs MI300X/MI325X).

GPU & High-Performance Computing
Flynn's Classification: Taxonomy of Computer Architectures

Flynn's Classification explained — SISD, SIMD, MISD, MIMD with interactive architecture explorer, SIMD evolution from MMX to AMX, branch divergence visualization, and workload-architecture throughput comparison.

If you found this explanation helpful, consider sharing it with others.

Mastodon