CUDA Multi-Process Service (MPS): Sharing One GPU Context

Three services, one mostly empty GPU

You run three small inference processes on one card. Each kernel only needs about a fifth of the SMs. nvidia-smi shows the GPU “in use,” but the chip spends most of its time either idle or swapping CUDA contexts between processes.

That is the default multi-process story: exclusive contexts, time-sliced onto the device. CUDA Multi-Process Service (MPS) exists so those processes can submit work through one shared GPU context and let the hardware scheduler co-reside kernels when space allows.

No app rewrite. Daemon + pipes + compute mode. The win only shows up when clients are GPU-light and concurrent.

Two numbers

1. One small client ≈ its SM footprint only
Three 20% jobs in exclusive mode still peak near 20% at any instant — the others wait.

2. Shared context can stack footprints
Same three clients under MPS can approach ~60% on a toy model (until SM or VRAM limits). Numbers on this page are pedagogical; measure on your SKU.

Time-slice tax vs concurrent fill

Without MPS, the driver runs one process’s context, then another. Context switches show up as latency jitter; empty SMs show up as wasted silicon.

SM fill

Same claim on a grid: exclusive paints one client’s band; MPS paints several.

Call path: who owns the context

Under MPS, libcuda.so detects the control daemon and routes API traffic over named pipes. The MPS server owns the shared CUDA context; clients do not each create a private one on the device.

# Host: start control daemon
export CUDA_VISIBLE_DEVICES=0
export CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps
export CUDA_MPS_LOG_DIRECTORY=/var/log/nvidia-mps
nvidia-cuda-mps-control -d

# Clients inherit the pipe directory (containers must mount it)
# echo quit | nvidia-cuda-mps-control   # stop

Cap greedy clients

CUDA_MPS_ACTIVE_THREAD_PERCENTAGE (or set_default_active_thread_percentage) bounds how much of the device’s thread capacity one client may claim. Default 100% each means a single fat launch can still dominate.

export CUDA_MPS_ACTIVE_THREAD_PERCENTAGE=50
# or interactive:
# nvidia-cuda-mps-control
# set_default_active_thread_percentage 50

Memory is separate: two processes can still exhaust VRAM. Thread caps are not MIG-style memory walls.

Exclusive · MPS · MIG

Pick by utilization need vs isolation need.

Production sketch

systemd (after persistenced if you use it):

# /etc/systemd/system/nvidia-mps.service
[Unit]
Description=NVIDIA CUDA MPS Control Daemon
After=nvidia-persistenced.service

[Service]
Type=forking
Environment=CUDA_VISIBLE_DEVICES=0
Environment=CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps
Environment=CUDA_MPS_LOG_DIRECTORY=/var/log/nvidia-mps
ExecStartPre=/bin/mkdir -p /var/log/nvidia-mps /tmp/nvidia-mps
ExecStart=/usr/bin/nvidia-cuda-mps-control -d
ExecStop=/bin/bash -c 'echo quit | /usr/bin/nvidia-cuda-mps-control'
Restart=on-failure

[Install]
WantedBy=multi-user.target

Containers must see the same pipe directory:

docker run --gpus all \
  -v /tmp/nvidia-mps:/tmp/nvidia-mps \
  -e CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps \
  my-inference-server

Confirm compute mode is not exclusive:

nvidia-smi -c DEFAULT
nvidia-smi | grep -i 'Compute Mode'

Traps

Volta+ added address-space isolation between MPS clients — still not a hard multi-tenant boundary. Pre-Volta: trusted workloads only. Untrusted partitions → MIG on Ampere/Hopper-class hardware.

What to do

1. Measure whether multi-process load is SM-light and concurrent — if one job fills the GPU, skip MPS.
2. Start nvidia-cuda-mps-control -d; mount CUDA_MPS_PIPE_DIRECTORY into every client container.
3. Set thread percentage when clients are unequal or bursty.
4. Keep compute mode DEFAULT; pair busy nodes with persistence.
5. Prefer MIG when isolation/QoS beats packing efficiency.

Related concepts