Three names, three different bugs
Someone says “we need more contexts.” Nsight is serial. Another process on the same GPU is stuck waiting. A pointer from GPU 0 blows up after cudaSetDevice(1).
Those are not one problem with three names. A context is the per-device container of GPU state. A stream is an in-order queue inside that container. MPS lets many processes co-execute instead of ordinary context time-slicing. On Volta+, clients keep separate CUDA contexts and GPU address spaces and submit work directly to hardware.
This page is the decision map. Full machinery lives on the sibling pages.
Two rules
1. Streams nest inside contexts
Adding streams never adds contexts. Concurrency within a process is almost always more streams on the primary context — not a second context on the same GPU.
2. Multi-process is not multi-stream
Two processes each get a private primary context by default. Their streams cannot see each other. Sharing means MPS (or MIG), not cudaStreamCreate in both PIDs.
Three layers
Flip the chip for the question each layer answers — device/state, order/overlap, or multi-process share.
What runs next, and can independent work overlap?
Context owns state. Stream owns order. Volta+ MPS co-schedules client contexts with separate address spaces. Touch the layer that matches the problem — not the one with the coolest API name.
The hierarchy
Process owns devices. Each device gets a context (usually one primary under the runtime API). Context owns the stream table. Streams own the operation queues.
What changes across production setups is how many of each layer you create — not the nesting.
Concurrency unit is the stream. Same context, more lanes — copies and kernels can overlap.
Amber chips on a stream mark the legacy default stream — it can still serialize blocking streams inside that context; cudaStreamNonBlocking streams are exempt. Teal streams are the overlap lanes you usually want.
Context vs stream, one aspect at a time
Big comparison tables hide the throughline. Pick the aspect you are arguing about.
Does not introduce concurrency by itself
Yes — unit of asynchrony in CUDA
If the question is multi-GPU or isolation, you are on the context row. If it is overlap or Nsight serialization, you are on the stream row.
Which layer for this symptom?
Start from what you see in the profiler or error log.
Need copy/compute overlap or concurrent independent kernels — wrong layer is the context.
Non-default streams, async memcpy, record events between producers/consumers.
Faster GPU inside one process → streams. Second GPU → context. Many processes on one chip → MPS (or MIG). Wrong layer wastes weeks.
Mixups people ask in the same breath
Default stream ≠ context. Streams do not own VRAM. Two GPUs mean two contexts first. Create order matters.
The default stream is the same thing as the context.
Context is the container. Default stream is one queue inside it. Every context has a default stream so bare API calls still work.
Adding streams never adds contexts. Adding contexts almost never fixes a serialization bug that streams would fix.
Where MPS sits
On Volta and newer, MPS clients retain their own CUDA contexts and GPU address spaces and submit directly to hardware. The server coordinates remaining shared scheduling resources outside the kernel submission path. Control/attach communication uses UNIX domain sockets in the configured MPS directory.
Pre-Volta MPS used a server-owned shared GPU address space and server-mediated work submission. That historical topology is different from the modern client-context model above.
Reach for MPS when: small-batch inference replicas, notebook fleets, concurrent workers whose kernels leave SM capacity available. Persistence addresses device initialization between jobs; MPS targets concurrent scheduling.
Skip MPS when: one job already saturates the GPU, each process already pins its own device, or you need hard isolation (look at MIG instead).
Deep dive: CUDA Multi-Process Service.
Traps
Wrong layer is the expensive trap. Dual contexts for “speed,” streams created on the wrong device, and legacy null-stream serialization are the usual sequels.
Chasing multi-context on one GPU because Nsight looks serial — or adding streams hoping multi-process will overlap.
In-process overlap → streams. Multi-GPU → contexts. Multi-process share → MPS/MIG.
This page is the decision map. Deeper dives live on the context, streams, and MPS concept pages — fix the layer first, then read the long form.
What to do
- In-process overlap → non-default streams, async copies, events — stay on one primary context per device.
- Second GPU →
cudaSetDevice, then allocate and create streams on that device. - Many processes, one GPU → MPS (throughput share) or MIG (partition), not more streams in each PID.
- Nsight serial → hunt default-stream / implicit sync first; do not invent a second context.
- Invalid pointer → match current context/device to the inventory that allocated the handle.
Deeper dives
- CUDA contexts — primary vs current stack, isolation, create cost
- CUDA streams — default-stream trap, pipelines, pitfalls
- CUDA MPS — concurrent clients, SM fill, thread caps
References
- NVIDIA MPS architecture — modern client ownership and submission
- NVIDIA Runtime/Driver interoperability — current-context binding
- NVIDIA cudaSetDevice — changing devices does not synchronize pending work
Related concepts
Deep dive into the CUDA context object: control vs data plane, inventory (memory, modules, streams, events, graphs), push/pop/setCurrent stacks, primary retain/release, flags and limits, isolation, cost, and traps.
A CUDA stream is an in-order queue of GPU ops. Overlap H→D, kernels, and D→H across streams — and avoid the legacy default-stream fences between blocking streams in one context.
Why ordinary multi-process CUDA time slicing can leave SMs idle, how Volta+ MPS co-schedules separate client contexts, execution-resource caps, and when to choose MPS or MIG.
How /dev/nvidia*, nvidiactl, nvidia-uvm, and DRI nodes map major/minor numbers to the driver, what CUDA opens first, and the minimum mount set for containers.
NVIDIA vs AMD for deep learning compared at both layers: the CUDA vs ROCm software moat, the microarchitecture (warp vs wavefront, SM vs CU, Tensor vs Matrix Cores), and the datacenter accelerators (H100/H200/B200 vs MI300X/MI325X).
Flynn's Classification explained — SISD, SIMD, MISD, MIMD with interactive architecture explorer, SIMD evolution from MMX to AMX, branch divergence visualization, and workload-architecture throughput comparison.
