Three services, one mostly empty GPU
You run three small inference processes on one card. Each kernel only needs about a fifth of the SMs. nvidia-smi shows the GPU “in use,” but the chip spends most of its time either idle or swapping CUDA contexts between processes.
That is the ordinary multi-process story in DEFAULT compute mode: private contexts take turns on the device. CUDA Multi-Process Service (MPS) lets the hardware co-schedule kernels from different clients when resources allow. On Volta and newer, clients retain their own CUDA contexts and GPU address spaces and submit directly to hardware. This page uses that modern architecture; older pre-Volta MPS is called out separately.
No app rewrite. Daemon + attach/control communication + compute mode. The win only shows up when clients are GPU-light and concurrent.
Two numbers
1. One small client ≈ its SM footprint only
In this toy model, each client occupies 4 of 20 SMs. Ordinary time slicing runs one client at a time: 20% instantaneous SM fill while it runs.
2. Concurrent clients can stack footprints
Three co-resident 4-SM clients occupy 12 of 20 SMs: 60% instantaneous SM fill. These are authored spatial illustrations, not measured GPU-Util values or throughput claims. nvidia-smi GPU-Util reports the fraction of sampled time with any kernel executing; a continuous 4-SM kernel can already show nearly 100%. Measure throughput, latency and timelines separately on your workload.
Time-slice tax vs concurrent fill
Without MPS, the driver runs one process’s context, then another. Context switches show up as latency jitter; empty SMs show up as wasted silicon.
One CUDA context on the device at a time. Small kernels leave most SMs empty; switches cost wall time.
Each client wants 4 of 20 SMs in this authored illustration. Ordinary time slicing runs one at a time; MPS can stack them. This is spatial SM fill, not nvidia-smi GPU-Util: a continuous kernel on four SMs can already report nearly 100% GPU-Util. Tape slots and switch gaps are illustrative.
SM fill
Same spatial model on a grid: ordinary time slicing paints one client’s band; MPS can paint several.
Three processes time-slice. Instantaneous fill is still one client’s 20% — others wait for the context.
MPS does not invent free SMs. It can co-schedule clients that would otherwise take turns, filling spare SM capacity when resource requirements permit.
Call path: client contexts and server coordination
On Volta+, each client CUDA context manages its own GPU address space and most hardware resources. Its work goes directly to hardware, outside the server’s critical execution path. The MPS server mediates remaining shared scheduling resources.
Client attach and control communication use UNIX domain sockets in the configured CUDA_MPS_PIPE_DIRECTORY; that directory name does not imply that every kernel launch takes an IPC hop. Pre-Volta MPS instead used a server-owned context and shared GPU address space and server-mediated work submission.
Each Volta+ client owns its context/address space and submits directly. Pre-Volta used a shared GPU address space and server-mediated submission.
Client ↔ UNIX domain sockets in CUDA_MPS_PIPE_DIRECTORY ↔ control daemon / MPS server. The server coordinates remaining shared resources; it is outside the critical kernel submission path.
Apps need no code change. Share the control directory with containers. EXCLUSIVE_PROCESS is recommended: the server owns the GPU exclusively while its MPS clients share it.
| Piece | Binary / path | Role |
|---|---|---|
| Control | nvidia-cuda-mps-control | start/stop server, config, client attach |
| Server | nvidia-cuda-mps-server | coordinate shared resources for concurrent client scheduling |
| Control directory | CUDA_MPS_PIPE_DIRECTORY (default /tmp/nvidia-mps) | UNIX domain sockets for client attach/control |
| Client | MPS-aware libcuda.so | own context/address space, direct hardware submission on Volta+ |
For MPS, NVIDIA recommends EXCLUSIVE_PROCESS: the MPS server is the exclusive GPU owner, and its clients can still share through MPS. DEFAULT also permits MPS but allows ordinary clients to bypass it. Set the mode before starting the server, with no competing GPU owner:
sudo nvidia-smi -i 0 -c EXCLUSIVE_PROCESS
nvidia-smi | grep -i 'Compute Mode'
# Host: established MPS control interface (documented as legacy v2)
# Verify support and setup for your installed driver/MPS release.
export CUDA_VISIBLE_DEVICES=0
export CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps
export CUDA_MPS_LOG_DIRECTORY=/var/log/nvidia-mps
nvidia-cuda-mps-control -d
# Clients inherit the pipe directory (containers must mount it)
# echo quit | nvidia-cuda-mps-control # stop
Cap greedy clients
CUDA_MPS_ACTIVE_THREAD_PERCENTAGE (or set_default_active_thread_percentage) bounds how much of the device’s thread capacity one client may claim. Default 100% each means a single fat launch can still dominate.
set_default_active_thread_percentage 33 (or env). Each client is capped near a third of device threads.
Caps bound thread/SM appetite per client. They do not partition VRAM. Memory oversubscription still OOMs the process (and can hurt neighbors pre-Volta).
export CUDA_MPS_ACTIVE_THREAD_PERCENTAGE=50
# or interactive:
# nvidia-cuda-mps-control
# set_default_active_thread_percentage 50
Memory is separate: two processes can still exhaust VRAM. Thread caps are not MIG-style memory walls.
Time slicing · MPS · MIG
Pick by packing need and isolation need. “Time slicing” below means ordinary private contexts in DEFAULT mode, not the EXCLUSIVE_PROCESS compute-mode setting.
- time-sliced
Small multi-process work time-slices. Instantaneous SM fill reflects one client; GPU-Util can still be near 100% if kernels run continuously.
- MPS
Co-resident kernels fill SMs. Best when each client is GPU-light.
- MIG
Fixed slices. Good inside a slice; no borrowing idle SMs from a neighbor slice.
MPS enables concurrent scheduling; MIG provides hardware partitions. Ordinary time slicing here means DEFAULT compute mode. EXCLUSIVE_PROCESS is a separate setting recommended for an MPS server.
| Ordinary time slicing | MPS (Volta+) | MIG | |
|---|---|---|---|
| Co-reside kernels | no (time-slice) | yes | within a slice |
| Isolation | process | soft (Volta+ better) | hardware |
| Best for | one big owner | many small trusted | multi-tenant QoS |
Production sketch
systemd (after persistenced if you use it):
# /etc/systemd/system/nvidia-mps.service
[Unit]
Description=NVIDIA CUDA MPS Control Daemon
After=nvidia-persistenced.service
[Service]
Type=forking
Environment=CUDA_VISIBLE_DEVICES=0
Environment=CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps
Environment=CUDA_MPS_LOG_DIRECTORY=/var/log/nvidia-mps
ExecStartPre=/bin/mkdir -p /var/log/nvidia-mps /tmp/nvidia-mps
ExecStart=/usr/bin/nvidia-cuda-mps-control -d
ExecStop=/bin/bash -c 'echo quit | /usr/bin/nvidia-cuda-mps-control'
Restart=on-failure
[Install]
WantedBy=multi-user.target
Containers must see the same control directory:
docker run --gpus all \
-v /tmp/nvidia-mps:/tmp/nvidia-mps \
-e CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps \
my-inference-server
Traps
Client lib looks for CUDA_MPS_PIPE_DIRECTORY and finds nothing. Looks like “CUDA broken.”
Mount host pipe dir into containers; set CUDA_MPS_PIPE_DIRECTORY on both sides.
Check compute mode, pipe directory, and whether the workload is actually multi-small before rewriting application code.
Volta+ added address-space isolation between MPS clients — still not a hard multi-tenant boundary. Pre-Volta: trusted workloads only. Untrusted partitions → MIG on Ampere/Hopper-class hardware.
What to do
- Measure whether multi-process load is SM-light and concurrent — if one job fills the GPU, skip MPS.
- Start
nvidia-cuda-mps-control -d; mountCUDA_MPS_PIPE_DIRECTORYinto every client container. - Set thread percentage when clients are unequal or bursty.
- Prefer compute mode EXCLUSIVE_PROCESS for MPS; pair busy nodes with persistence.
- Prefer MIG when isolation/QoS beats packing efficiency.
References
- NVIDIA MPS architecture — client contexts, direct submission and control communication
- NVIDIA MPS compute modes — EXCLUSIVE_PROCESS recommendation
- NVIDIA MPS common tasks — commands and deployment setup
- NVIDIA System Management Interface — GPU-Util metric definition
Related concepts
Deep dive into the CUDA context object: control vs data plane, inventory (memory, modules, streams, events, graphs), push/pop/setCurrent stacks, primary retain/release, flags and limits, isolation, cost, and traps.
Decision map for CUDA: a context is per-process GPU state, a stream is an in-order queue inside a context, and MPS co-schedules work across client contexts on Volta+. Pick the layer that matches the problem.
A CUDA stream is an in-order queue of GPU ops. Overlap H→D, kernels, and D→H across streams — and avoid the legacy default-stream fences between blocking streams in one context.
How /dev/nvidia*, nvidiactl, nvidia-uvm, and DRI nodes map major/minor numbers to the driver, what CUDA opens first, and the minimum mount set for containers.
NVIDIA vs AMD for deep learning compared at both layers: the CUDA vs ROCm software moat, the microarchitecture (warp vs wavefront, SM vs CU, Tensor vs Matrix Cores), and the datacenter accelerators (H100/H200/B200 vs MI300X/MI325X).
Flynn's Classification explained — SISD, SIMD, MISD, MIMD with interactive architecture explorer, SIMD evolution from MMX to AMX, branch divergence visualization, and workload-architecture throughput comparison.
