Skip to main content

CUDA Multi-Process Service (MPS): Concurrent Client Contexts

Why ordinary multi-process CUDA time slicing can leave SMs idle, how Volta+ MPS co-schedules separate client contexts, execution-resource caps, and when to choose MPS or MIG.

Three services, one mostly empty GPU

You run three small inference processes on one card. Each kernel only needs about a fifth of the SMs. nvidia-smi shows the GPU “in use,” but the chip spends most of its time either idle or swapping CUDA contexts between processes.

That is the ordinary multi-process story in DEFAULT compute mode: private contexts take turns on the device. CUDA Multi-Process Service (MPS) lets the hardware co-schedule kernels from different clients when resources allow. On Volta and newer, clients retain their own CUDA contexts and GPU address spaces and submit directly to hardware. This page uses that modern architecture; older pre-Volta MPS is called out separately.

No app rewrite. Daemon + attach/control communication + compute mode. The win only shows up when clients are GPU-light and concurrent.

Two numbers

1. One small client ≈ its SM footprint only
In this toy model, each client occupies 4 of 20 SMs. Ordinary time slicing runs one client at a time: 20% instantaneous SM fill while it runs.

2. Concurrent clients can stack footprints

Three co-resident 4-SM clients occupy 12 of 20 SMs: 60% instantaneous SM fill. These are authored spatial illustrations, not measured GPU-Util values or throughput claims. nvidia-smi GPU-Util reports the fraction of sampled time with any kernel executing; a continuous 4-SM kernel can already show nearly 100%. Measure throughput, latency and timelines separately on your workload.

Time-slice tax vs concurrent fill

Without MPS, the driver runs one process’s context, then another. Context switches show up as latency jitter; empty SMs show up as wasted silicon.

Time-slice vs MPS
~20%
instantaneous SM fill while one runs · pedagogical
client Aclient Bclient C
t →amber = context switch · teal band = one client alone on the GPU

One CUDA context on the device at a time. Small kernels leave most SMs empty; switches cost wall time.

Each client wants 4 of 20 SMs in this authored illustration. Ordinary time slicing runs one at a time; MPS can stack them. This is spatial SM fill, not nvidia-smi GPU-Util: a continuous kernel on four SMs can already report nearly 100% GPU-Util. Tape slots and switch gaps are illustrative.

SM fill

Same spatial model on a grid: ordinary time slicing paints one client’s band; MPS can paint several.

SM fill
20%
SM busy · 20-slot toy grid
idle SMs
16

Three processes time-slice. Instantaneous fill is still one client’s 20% — others wait for the context.

MPS does not invent free SMs. It can co-schedule clients that would otherwise take turns, filling spare SM capacity when resource requirements permit.

Call path: client contexts and server coordination

On Volta+, each client CUDA context manages its own GPU address space and most hardware resources. Its work goes directly to hardware, outside the server’s critical execution path. The MPS server mediates remaining shared scheduling resources.

Client attach and control communication use UNIX domain sockets in the configured CUDA_MPS_PIPE_DIRECTORY; that directory name does not imply that every kernel launch takes an IPC hop. Pre-Volta MPS instead used a server-owned context and shared GPU address space and server-mediated work submission.

Call path
direct
GPU context ownership
hop 1/3
client process
client process: your binary · standard CUDA API

Each Volta+ client owns its context/address space and submits directly. Pre-Volta used a shared GPU address space and server-mediated submission.

Attach/control path (separate from work submission)

Client ↔ UNIX domain sockets in CUDA_MPS_PIPE_DIRECTORY ↔ control daemon / MPS server. The server coordinates remaining shared resources; it is outside the critical kernel submission path.

Apps need no code change. Share the control directory with containers. EXCLUSIVE_PROCESS is recommended: the server owns the GPU exclusively while its MPS clients share it.

PieceBinary / pathRole
Controlnvidia-cuda-mps-controlstart/stop server, config, client attach
Servernvidia-cuda-mps-servercoordinate shared resources for concurrent client scheduling
Control directoryCUDA_MPS_PIPE_DIRECTORY (default /tmp/nvidia-mps)UNIX domain sockets for client attach/control
ClientMPS-aware libcuda.soown context/address space, direct hardware submission on Volta+

For MPS, NVIDIA recommends EXCLUSIVE_PROCESS: the MPS server is the exclusive GPU owner, and its clients can still share through MPS. DEFAULT also permits MPS but allows ordinary clients to bypass it. Set the mode before starting the server, with no competing GPU owner:

bash
sudo nvidia-smi -i 0 -c EXCLUSIVE_PROCESS
nvidia-smi | grep -i 'Compute Mode'
bash
# Host: established MPS control interface (documented as legacy v2)
# Verify support and setup for your installed driver/MPS release.
export CUDA_VISIBLE_DEVICES=0
export CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps
export CUDA_MPS_LOG_DIRECTORY=/var/log/nvidia-mps
nvidia-cuda-mps-control -d

# Clients inherit the pipe directory (containers must mount it)
# echo quit | nvidia-cuda-mps-control   # stop

Cap greedy clients

CUDA_MPS_ACTIVE_THREAD_PERCENTAGE (or set_default_active_thread_percentage) bounds how much of the device’s thread capacity one client may claim. Default 100% each means a single fat launch can still dominate.

Thread percentage
33%
ACTIVE_THREAD_PERCENTAGE / client
sum of caps
99%
client 1
33%
client 2
33%
client 3
33%
export CUDA_MPS_ACTIVE_THREAD_PERCENTAGE=33
# or: echo "set_default_active_thread_percentage 33" | nvidia-cuda-mps-control

set_default_active_thread_percentage 33 (or env). Each client is capped near a third of device threads.

Caps bound thread/SM appetite per client. They do not partition VRAM. Memory oversubscription still OOMs the process (and can hurt neighbors pre-Volta).

bash
export CUDA_MPS_ACTIVE_THREAD_PERCENTAGE=50
# or interactive:
# nvidia-cuda-mps-control
# set_default_active_thread_percentage 50

Memory is separate: two processes can still exhaust VRAM. Thread caps are not MIG-style memory walls.

Time slicing · MPS · MIG

Pick by packing need and isolation need. “Time slicing” below means ordinary private contexts in DEFAULT mode, not the EXCLUSIVE_PROCESS compute-mode setting.

Time slicing · MPS · MIG
MPS
sticky word for this aspect
  • time-sliced

    Small multi-process work time-slices. Instantaneous SM fill reflects one client; GPU-Util can still be near 100% if kernels run continuously.

  • MPS

    Co-resident kernels fill SMs. Best when each client is GPU-light.

  • MIG

    Fixed slices. Good inside a slice; no borrowing idle SMs from a neighbor slice.

MPS enables concurrent scheduling; MIG provides hardware partitions. Ordinary time slicing here means DEFAULT compute mode. EXCLUSIVE_PROCESS is a separate setting recommended for an MPS server.

Ordinary time slicingMPS (Volta+)MIG
Co-reside kernelsno (time-slice)yeswithin a slice
Isolationprocesssoft (Volta+ better)hardware
Best forone big ownermany small trustedmulti-tenant QoS

Production sketch

systemd (after persistenced if you use it):

ini
# /etc/systemd/system/nvidia-mps.service
[Unit]
Description=NVIDIA CUDA MPS Control Daemon
After=nvidia-persistenced.service

[Service]
Type=forking
Environment=CUDA_VISIBLE_DEVICES=0
Environment=CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps
Environment=CUDA_MPS_LOG_DIRECTORY=/var/log/nvidia-mps
ExecStartPre=/bin/mkdir -p /var/log/nvidia-mps /tmp/nvidia-mps
ExecStart=/usr/bin/nvidia-cuda-mps-control -d
ExecStop=/bin/bash -c 'echo quit | /usr/bin/nvidia-cuda-mps-control'
Restart=on-failure

[Install]
WantedBy=multi-user.target

Containers must see the same control directory:

bash
docker run --gpus all \
  -v /tmp/nvidia-mps:/tmp/nvidia-mps \
  -e CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps \
  my-inference-server

Traps

MPS traps
mount pipes
sticky fix word
trap
Daemon on host, empty /tmp in pod

Client lib looks for CUDA_MPS_PIPE_DIRECTORY and finds nothing. Looks like “CUDA broken.”

do this
shared volume + env

Mount host pipe dir into containers; set CUDA_MPS_PIPE_DIRECTORY on both sides.

Check compute mode, pipe directory, and whether the workload is actually multi-small before rewriting application code.

Volta+ added address-space isolation between MPS clients — still not a hard multi-tenant boundary. Pre-Volta: trusted workloads only. Untrusted partitions → MIG on Ampere/Hopper-class hardware.

What to do

  1. Measure whether multi-process load is SM-light and concurrent — if one job fills the GPU, skip MPS.
  2. Start nvidia-cuda-mps-control -d; mount CUDA_MPS_PIPE_DIRECTORY into every client container.
  3. Set thread percentage when clients are unequal or bursty.
  4. Prefer compute mode EXCLUSIVE_PROCESS for MPS; pair busy nodes with persistence.
  5. Prefer MIG when isolation/QoS beats packing efficiency.

References

GPU & High-Performance Computing
CUDA Contexts: Ownership, Current Stack, Isolation

Deep dive into the CUDA context object: control vs data plane, inventory (memory, modules, streams, events, graphs), push/pop/setCurrent stacks, primary retain/release, flags and limits, isolation, cost, and traps.

GPU & High-Performance Computing
CUDA Context vs Streams vs MPS: Which Layer Fixes What

Decision map for CUDA: a context is per-process GPU state, a stream is an in-order queue inside a context, and MPS co-schedules work across client contexts on Volta+. Pick the layer that matches the problem.

GPU & High-Performance Computing
CUDA Streams: Asynchronous Execution and Concurrency

A CUDA stream is an in-order queue of GPU ops. Overlap H→D, kernels, and D→H across streams — and avoid the legacy default-stream fences between blocking streams in one context.

GPU & High-Performance Computing
NVIDIA Device Files in /dev/

How /dev/nvidia*, nvidiactl, nvidia-uvm, and DRI nodes map major/minor numbers to the driver, what CUDA opens first, and the minimum mount set for containers.

GPU & High-Performance Computing
NVIDIA vs AMD for Deep Learning: CUDA vs ROCm and the Datacenter Accelerators

NVIDIA vs AMD for deep learning compared at both layers: the CUDA vs ROCm software moat, the microarchitecture (warp vs wavefront, SM vs CU, Tensor vs Matrix Cores), and the datacenter accelerators (H100/H200/B200 vs MI300X/MI325X).

GPU & High-Performance Computing
Flynn's Classification: Taxonomy of Computer Architectures

Flynn's Classification explained — SISD, SIMD, MISD, MIMD with interactive architecture explorer, SIMD evolution from MMX to AMX, branch divergence visualization, and workload-architecture throughput comparison.

If you found this explanation helpful, consider sharing it with others.

Mastodon