Skip to main content

CUDA Streams: Asynchronous Execution and Concurrency

A CUDA stream is an in-order queue of GPU ops. Overlap H→D, kernels, and D→H across streams — and avoid the legacy default-stream fences between blocking streams in one context.

Kernels look fine. The timeline doesn’t.

Nsight shows H→D, then a kernel, then D→H, then the same three bars again. SMs sit idle while the copy engine works. The copy engine sits idle while SMs work. Nothing is “wrong” with the kernel — the schedule never stacked independent engines.

A CUDA stream is an in-order queue of GPU work: kernels, memcpys, events. Same stream → serial. Different streams → the hardware may run phases together if they need different engines (or spare SM capacity).

Two numbers

1. Serial · wall ≈ 3N for N balanced chunks
One stream (or everything on the default stream): each chunk pays H→D + kernel + D→H end-to-end. Engines take turns.

2. Pipelined · wall ≈ N + 2 in steady state
With enough non-default streams, chunk k’s kernel overlaps chunk k+1’s H→D and chunk k−1’s D→H. Three engines busy; wall time drops toward the longest phase, not the sum.

Flip the tapes until that feels mechanical — not magical.

Helpful background

CUDA contexts

Follow resource ownership, current-context binding, primary contexts and lifecycle boundaries.

Open full page

Pipeline: three engines, N streams

Same four chunks. Change stream count and watch H→D · kernel · D→H fill in.

Stream pipeline
12u
wall time · 4 chunks · serial baseline
engines 33%
t0 · now 1/3
Hardware engines (not stream lanes) · hollow = idle engines while one phase runs
H→D
kernel
D→H
Stream pipeline example
cudaMemcpy(d, h, n, H2D);        // one queuemyKernel<<<g,b>>>(...);          // default streamcudaMemcpy(out, d, n, D2H);      // then next chunk

One stream: only one engine can run this chunk’s next phase. The other two sit hollow until the FIFO advances.

One in-order queue means phases of chunk N+1 cannot start until chunk N finishes. Copy engines and SMs take turns — the authored model fills one of three engine slots at a time.

C++
// Async API + explicit stream — the building blocks
cudaMemcpyAsync(dst, src, bytes, cudaMemcpyHostToDevice, stream);
myKernel<<<grid, block, 0, stream>>>(args);
cudaMemcpyAsync(host, dst, bytes, cudaMemcpyDeviceToHost, stream);

Kernel launch is already async w.r.t. the host. Overlap lives or dies on which stream you pass and whether the host buffer is pinned.

The default stream is a trap

Each CUDA context has a legacy default stream (stream 0 in legacy mode). Bare cudaMemcpy and myKernel<<<g,b>>> use it. It waits for previously queued work in blocking streams in that same context; later work in those streams waits for the legacy operation. Streams created with cudaStreamNonBlocking are exempt. Other contexts/devices are outside this fence.

You can wire four beautiful non-default streams and still open a gap among blocking streams in one context with one forgotten sync memcpy.

Default stream trap · same-context blocking streams
4u stall
units frozen by one default-stream op
issuing
t0
Stream activity · amber = default-stream work (peers idle)
stream 0
stream 1
stream 2
default
Default-stream example
kernel<<<g,b,0,s0>>>(...);   // non-default OKcudaMemcpy(d, h, n, H2D);    // DEFAULT — trap// s0/s1 wait until default drains

Streams look healthy… until one host call without a stream lands on the default queue.

The legacy default stream implicitly synchronizes with blocking streams in the same context. A legacy cudaMemcpy (or any default-stream kernel) orders work across those blocking streams. cudaStreamNonBlocking streams and other contexts are exempt.

Synchronization without the sledgehammer

Eventually the host needs “is it done?” Use the lightest tool that fits:

CallBlocks
cudaDeviceSynchronize()All streams in the calling thread’s current context on the device
cudaStreamSynchronize(s)One stream’s queue
cudaStreamQuery(s)Nothing (poll)
Events + cudaStreamWaitEventOne stream waits on another — host free
C++
cudaEvent_t done;
cudaEventCreate(&done);

myKernel<<<grid, block, 0, streamA>>>(d_data);
cudaEventRecord(done, streamA);

// streamB waits on GPU work, not on the host thread
cudaStreamWaitEvent(streamB, done, 0);
consumerKernel<<<grid, block, 0, streamB>>>(d_data);

Events also measure GPU time: record before/after, then cudaEventElapsedTime — no CPU-clock skew.

Pitfalls that invent serialization

Same streams idea. Four ways people accidentally kill concurrency.

Stream pitfalls
Trap
Pageable host buffer

cudaMemcpyAsync quietly stages through a driver pin. Overlap dies; CPU pays the copy.

sync
Fix
cudaMallocHost / pin

True async H→D/D→H. DMA can run while kernels and the host keep going.

Streams only buy concurrency the hardware can honor. Pageable “async”, default-stream fences, and sync allocs are the three ways people invent serialization without meaning to.

Stream priorities

C++
int lo, hi;
cudaDeviceGetStreamPriorityRange(&lo, &hi);
cudaStream_t hot;
cudaStreamCreateWithPriority(&hot, cudaStreamNonBlocking, hi);

High priority is not preemption mid-kernel — it wins at the next launch boundary. Useful for a latency-sensitive inference path beside a bulk training stream.

PyTorch

python
import torch

s1 = torch.cuda.Stream()
s2 = torch.cuda.Stream()

with torch.cuda.stream(s1):
    a = torch.randn(1024, 1024, device='cuda')
    b = a @ a.T

s2.wait_stream(s1)
with torch.cuda.stream(s2):
    c = b.softmax(dim=-1)
    b.record_stream(s2)  # protect b storage until this consumer finishes

torch.cuda.synchronize()

On non-default streams, you establish producer/consumer order with wait_stream or events. Here s2.wait_stream(s1) orders the softmax after production of b. Allocation lifetime is separate: b.record_stream(s2) prevents allocator reuse while the consumer is pending; retaining the storage until an explicit completion wait is an alternative. The allocator does not infer arbitrary data dependencies, and record_stream does not create an execution dependency. Custom CUDA ops must use the current PyTorch stream or establish equivalent explicit dependencies.

What to do

  1. Never leave critical path work on the legacy default stream — explicit streams (or per-thread default) everywhere you care about overlap.
  2. Pin host memory for cudaMemcpyAsync — pageable “async” is often a silent sync tax.
  3. Pipeline H→D / kernel / D→H with 2–4 streams first; measure before inventing sixteen.
  4. Prefer events and stream waits over cudaDeviceSynchronize in production loops.
  5. Keep allocs out of the hot path — pool or cudaMallocAsync; setup-time malloc only.
C++
cudaStream_t streams[4];
for (int i = 0; i < 4; i++)
  cudaStreamCreateWithFlags(&streams[i], cudaStreamNonBlocking);

for (int i = 0; i < nChunks; i++) {
  auto s = streams[i % 4];
  cudaMemcpyAsync(d[i], h[i], bytes, cudaMemcpyHostToDevice, s);
  kernel<<<grid, block, 0, s>>>(d[i]);
  cudaMemcpyAsync(out[i], d[i], bytes, cudaMemcpyDeviceToHost, s);
}
for (int i = 0; i < 4; i++) cudaStreamSynchronize(streams[i]);

When streams are the wrong tool

  • Multi-process GPU sharing → CUDA MPS, not more streams in one process.
  • Cross-device work → NCCL / peer memcpy; streams do not cross devices.
  • One long kernel, nothing to overlap → fix the kernel; streams only hide independent latency.

Streams keep the GPU fed. Get the compute right first, then stop wasting engines between phases.

Further reading

GPU & High-Performance Computing
CUDA Contexts: Ownership, Current Stack, Isolation

Deep dive into the CUDA context object: control vs data plane, inventory (memory, modules, streams, events, graphs), push/pop/setCurrent stacks, primary retain/release, flags and limits, isolation, cost, and traps.

GPU & High-Performance Computing
CUDA Context vs Streams vs MPS: Which Layer Fixes What

Decision map for CUDA: a context is per-process GPU state, a stream is an in-order queue inside a context, and MPS co-schedules work across client contexts on Volta+. Pick the layer that matches the problem.

GPU & High-Performance Computing
CUDA Multi-Process Service (MPS): Concurrent Client Contexts

Why ordinary multi-process CUDA time slicing can leave SMs idle, how Volta+ MPS co-schedules separate client contexts, execution-resource caps, and when to choose MPS or MIG.

GPU & High-Performance Computing
NVIDIA Device Files in /dev/

How /dev/nvidia*, nvidiactl, nvidia-uvm, and DRI nodes map major/minor numbers to the driver, what CUDA opens first, and the minimum mount set for containers.

GPU & High-Performance Computing
NVIDIA vs AMD for Deep Learning: CUDA vs ROCm and the Datacenter Accelerators

NVIDIA vs AMD for deep learning compared at both layers: the CUDA vs ROCm software moat, the microarchitecture (warp vs wavefront, SM vs CU, Tensor vs Matrix Cores), and the datacenter accelerators (H100/H200/B200 vs MI300X/MI325X).

GPU & High-Performance Computing
Flynn's Classification: Taxonomy of Computer Architectures

Flynn's Classification explained — SISD, SIMD, MISD, MIMD with interactive architecture explorer, SIMD evolution from MMX to AMX, branch divergence visualization, and workload-architecture throughput comparison.

If you found this explanation helpful, consider sharing it with others.

Mastodon