NVIDIA Persistence Daemon: Killing GPU Cold Starts

Summary: Why the first nvidia-smi or CUDA open costs seconds, what that cold path rebuilds, how nvidia-persistenced holds device FDs on the host, and when persistence pays off for pods, CI, and batch jobs.

The three-second tax

You ship a one-line health check. The container is up. The job is “ready.” Then the first nvidia-smi or CUDA open sits for about three seconds on a quiet node — firmware cache, clocks, PCIe link, power state — before anything useful runs. Run it again immediately and it returns in tens of milliseconds. Wait long enough with no process holding the GPU open, and the tax comes back.

That is not a slow kernel. That is cold driver state. nvidia-persistenced exists so production nodes stop paying it on every short job.

On a training box that runs one process for twelve hours, three seconds is a footnote. On a node that starts hundreds of GPU pods a day, three seconds is the product.

Two numbers

1. Cold open ≈ seconds · warm open ≈ tens of ms
Illustrative: ~3.2 s first call vs ~90 ms while the driver stays loaded. SKU, driver branch, and whether the display stack already owns the card all move the absolute numbers. The ratio is the lesson.

2. One open FD keeps the driver up
As long as something holds /dev/nvidia* (and usually /dev/nvidiactl), the module keeps initialized state. The daemon is that something when your apps come and go.

Flip the instruments until both feel mechanical — not “another flag on a checklist.”

Every job pays again

Without persistence, the lifecycle is: open → cold init → work → exit → unload → next open cold again. Short inference workers, CI GPU tests, and pod churn make the amber bar the product.

code

sudo nvidia-smi -pm 0
nvidia-smi --query-compute-apps=pid --format=csv,noheader
# empty

time nvidia-smi > /dev/null    # cold: often multi-second
time nvidia-smi > /dev/null    # warm: tens of ms
sleep 20
time nvidia-smi > /dev/null    # may cold again if nothing holds FDs

What’s inside those seconds

The wall clock is not one mysterious “GPU is slow” blob. Cold open rebuilds a stack of driver and device scaffolding the kernel is happy to drop when refcount hits zero. Warm open mostly reuses what is already resident.

Click a phase. Flip cold ↔ warm.

This is why “we already use a big instance” does not fix cold starts: you are not short on FLOPs. You are re-paying init work that is not your model.

Persistence also does not replace:

First CUDA context creation costs (real, but usually smaller than multi-second cold driver bring-up)
Loading weights into VRAM
Image pull, Python import graphs, or torch init

If a job is still multi-second after persistence_mode is On and time nvidia-smi is snappy, profile above the driver.

Hold a quiet client open

The mechanism is almost boring — and that is why it works. The daemon opens each GPU device node and sleeps. Kernel refcount stays non-zero. Driver state stays initialized. No CUDA context, no model, no intentional VRAM hog — just open handles (a few MB of process RSS).

nvidia-smi -pm 1 asks the driver to stay in persistence mode without you writing a daemon. End state looks the same in nvidia-smi’s Persistence-M column. Production still prefers nvidia-persistenced as a host unit: it survives job churn, restarts cleanly, and shows up in journalctl when something is wrong.

What stays · what does not

Kept warm	Still destroyed on app exit
Firmware / VBIOS cache	CUDA contexts
Memory controller + ECC setup	Device allocations / model weights
PCIe link training + AER scaffolding	In-flight kernels
Power / P-state scaffolding	App-specific register state
Display engine config (if relevant)	Your process’s streams and events

Persistence is not “leave my model on the GPU.” It is “do not re-initialize the card from zero every process.” For resident weights you want a long-lived server (or a warm pool of workers), not the daemon alone.

code

// Mental model — not the real source tree
for each GPU:
    fd = open("/dev/nvidiaN", O_RDWR);
    // optionally ioctl / set persistence mode
// sleep forever; FDs stay open → refcount ≥ 1 → state stays warm

Daemon vs `nvidia-smi -pm`

	`nvidia-smi -pm 1`	`nvidia-persistenced`
How	Driver flag	Userspace process holds FDs
Survives reboot	Only if something re-applies it	With `systemctl enable`
Multi-tenant ops	Easy to flip by accident	Unit + logs
Containers	Host only still	Host only still
When to use	Debug, laptop, one-shot	Always-on GPU nodes

Both can show Enabled. Prefer the unit in production; keep -pm as a hammer when the unit is missing and you need a warm card now.

Burst math: short jobs win

One 12-hour train pays cold start once. A hundred 400 ms inference processes pay it a hundred times unless the driver stays warm. CI that launches a fresh Python per test case is the same shape.

Where it matters

Batch / serverless-style inference — process per request or per batch
Kubernetes GPU pods — rolling updates, scale-out, preemption, canaries
CI GPU tests — suite after suite on the same node
Interactive scripts — python train.py every few minutes while you iterate
Multi-process frameworks that restart workers between runs

Where it barely matters

One long-running trainer or model server — the app already holds the device
Desktop display path — X/Wayland (or the compositor) already keeps the driver busy
One GPU job per day — three seconds is noise next to data movement
True power-down policies on rarely used hardware — you may want cold teardown

Pods: ready is not “container started”

Kubernetes will mark a container running long before the first CUDA open finishes. If your readiness probe hits GPU code (or your main process blocks on first device open), cold init sits on the critical path for Ready.

The daemon must run on the node, not in each pod. GPU Operator and similar stacks can install and manage it for you — still host-scoped.

code

# On the node (VM / bare metal / GPU node image)
sudo systemctl enable --now nvidia-persistenced
nvidia-smi --query-gpu=persistence_mode --format=csv,noheader

# From a pod — you should see Enabled; you should not run the daemon here
kubectl run gpu-check --rm -it --restart=Never \
  --image=nvidia/cuda:12.6.0-base-ubuntu22.04 \
  --limits='nvidia.com/gpu=1' \
  -- nvidia-smi --query-gpu=persistence_mode --format=csv,noheader

Docker is the same story: --gpus all sees host driver state. Warm the host once.

Cost, power, and multi-GPU

RSS: on the order of a few MB per process — often quoted ~2–4 MB class, not a training footprint.

VRAM: the daemon is not your model. You should not see multi-GB device allocations from persistenced itself.

Power: keeping the driver warm can mean the GPU is less eager to sit in the deepest idle. On always-on training nodes that is usually the right trade. On a laptop you open twice a week, leaving persistence off can be fine. Measure with your power tools if joules matter more than cold starts.

Multi-GPU / MIG: the daemon registers devices the driver exposes. After GPU hot-add, MIG reconfigure, or driver reload, confirm FDs and persistence_mode again — do not assume a unit that started at boot still matches the current topology.

Ops pitfalls

Most “we turned it on and nothing got faster” tickets are placement or expectation bugs.

What to do

Install and enable on the host — systemd unit or GPU Operator path, not a sidecar in every pod.
Verify three ways — persistence_mode, timed nvidia-smi after idle, and lsof on the daemon’s /dev/nvidia* FDs.
Leave it on multi-tenant / high-churn nodes — cost is quiet process + small RSS.
Do not expect VRAM or contexts to survive — only driver init state.
If still slow after warm driver — profile model load, first context, and container start; persistence already did its job.
Pair with sane device nodes — see NVIDIA device files.

code

# Debian/Ubuntu
sudo apt-get install nvidia-persistenced
sudo systemctl enable --now nvidia-persistenced
sudo systemctl status nvidia-persistenced --no-pager

nvidia-smi --query-gpu=persistence_mode --format=csv,noheader
# Enabled

# After a quiet minute with no other clients
time nvidia-smi > /dev/null
# real ~0.1s class when warm

sudo lsof -p "$(pgrep -n nvidia-persistenced)" | grep /dev/nvidia
# nvidiactl + nvidia0… open

journalctl -u nvidia-persistenced -n 30 --no-pager

code

# One-shot without the unit (debug / laptop)
sudo nvidia-smi -pm 1
sudo nvidia-smi -pm 0   # restore cold teardown policy

Quick failure checklist

Symptom	Check
Persistence-M Off	unit active? `-pm` flipped? permission to enable?
On but still multi-second `nvidia-smi`	wrong GPU visible? driver reload mid-flight?
Fast `nvidia-smi`, slow app	model load / first CUDA context / Python imports
Works on host, slow in pod	runtime not injecting devices; not a persistence bug
Was fine, now cold after MIG change	restart unit / re-enable; re-verify FDs