Skip to main content

NVIDIA Persistence Daemon: Killing GPU Cold Starts

Why the first nvidia-smi or CUDA open costs seconds, what that cold path rebuilds, how nvidia-persistenced holds device FDs on the host, and when persistence pays off for pods, CI, and batch jobs.

The three-second tax

You ship a one-line health check. The container is up. The job is “ready.” Then the first nvidia-smi or CUDA open sits for about three seconds on a quiet node — firmware cache, clocks, PCIe link, power state — before anything useful runs. Run it again immediately and it returns in tens of milliseconds. Wait long enough with no process holding the GPU open, and the tax comes back.

That is not a slow kernel. That is cold driver state. nvidia-persistenced exists so production nodes stop paying it on every short job.

On a training box that runs one process for twelve hours, three seconds is a footnote. On a node that starts hundreds of GPU pods a day, three seconds is the product.

Two numbers

1. Cold open ≈ seconds · warm open ≈ tens of ms
Illustrative: ~3.2 s first call vs ~90 ms while the driver stays loaded. SKU, driver branch, and whether the display stack already owns the card all move the absolute numbers. The ratio is the lesson.

2. One open FD keeps the driver up
As long as something holds /dev/nvidia* (and usually /dev/nvidiactl), the module keeps initialized state. The daemon is that something when your apps come and go.

Flip the instruments until both feel mechanical — not “another flag on a checklist.”

Every job pays again

Without persistence, the lifecycle is: open → cold init → work → exit → unload → next open cold again. Short inference workers, CI GPU tests, and pod churn make the amber bar the product.

Cold start tape
3200ms
first useful call · pedagogical
4 jobs on tape
×36 tax vs warm
driver
app
cold init useful work held warm idle / unloaded

Job 1: driver cold. Firmware, clocks, PCIe link — ~3200 ms before any CUDA work.

Without a client holding the device open, the NVIDIA module drops driver state after the last process exits. The next job reinitializes — often seconds, not milliseconds. Persistence keeps one quiet client alive so cold start is paid once, not per pod.

bash
sudo nvidia-smi -pm 0
nvidia-smi --query-compute-apps=pid --format=csv,noheader
# empty

time nvidia-smi > /dev/null    # cold: often multi-second
time nvidia-smi > /dev/null    # warm: tens of ms
sleep 20
time nvidia-smi > /dev/null    # may cold again if nothing holds FDs

What’s inside those seconds

The wall clock is not one mysterious “GPU is slow” blob. Cold open rebuilds a stack of driver and device scaffolding the kernel is happy to drop when refcount hits zero. Warm open mostly reuses what is already resident.

Click a phase. Flip cold ↔ warm.

Inside the cold open
3200ms
init only · pedagogical phases
cold 3200 → warm 90
~36× less init tax

firmware / VBIOS: ~900 ms (~28% of cold open). ROM reads and structure cache the module rebuilds when cold.

Persistence does not skip “loading your model.” It skips redoing firmware, memory controller, PCIe, and power scaffolding on every process. Click a phase; flip cold ↔ warm and watch the amber stack collapse.

This is why “we already use a big instance” does not fix cold starts: you are not short on FLOPs. You are re-paying init work that is not your model.

Persistence also does not replace:

  • First CUDA context creation costs (real, but usually smaller than multi-second cold driver bring-up)
  • Loading weights into VRAM
  • Image pull, Python import graphs, or torch init

If a job is still multi-second after persistence_mode is On and time nvidia-smi is snappy, profile above the driver.

Hold a quiet client open

The mechanism is almost boring — and that is why it works. The daemon opens each GPU device node and sleeps. Kernel refcount stays non-zero. Driver state stays initialized. No CUDA context, no model, no intentional VRAM hog — just open handles (a few MB of process RSS).

nvidia-smi -pm 1 asks the driver to stay in persistence mode without you writing a daemon. End state looks the same in nvidia-smi’s Persistence-M column. Production still prefers nvidia-persistenced as a host unit: it survives job churn, restarts cleanly, and shows up in journalctl when something is wrong.

Who holds the device
UP
driver state · pedagogical
refcount ≥ 1
init state retained
userspace
nvidia-persistenced
sleep + open FDs
your CUDA app
not holding
device nodes
  • /dev/nvidiactlopen
  • /dev/nvidia0open
  • /dev/nvidia1open
nvidia.ko state
initialized
  • firmware / VBIOS cache
  • memory controller config
  • PCIe link training
  • power / P-state setup
persistence keeps
firmware / VBIOS cache · memory controller config · PCIe link training · power / P-state setup
never keeps
CUDA contexts · device allocations · in-flight kernels

nvidia-persistenced keeps FDs open and sleeps. No VRAM, no CUDA context — just refcounts that keep the driver warm.

Same outcome as nvidia-smi -pm 1: driver stays ready. The daemon path is a long-lived process with open handles — preferred on multi-tenant hosts and across restarts of individual jobs.

What stays · what does not

Kept warmStill destroyed on app exit
Firmware / VBIOS cacheCUDA contexts
Memory controller + ECC setupDevice allocations / model weights
PCIe link training + AER scaffoldingIn-flight kernels
Power / P-state scaffoldingApp-specific register state
Display engine config (if relevant)Your process’s streams and events

Persistence is not “leave my model on the GPU.” It is “do not re-initialize the card from zero every process.” For resident weights you want a long-lived server (or a warm pool of workers), not the daemon alone.

c
// Mental model — not the real source tree
for each GPU:
    fd = open("/dev/nvidiaN", O_RDWR);
    // optionally ioctl / set persistence mode
// sleep forever; FDs stay open → refcount ≥ 1 → state stays warm

Daemon vs nvidia-smi -pm

nvidia-smi -pm 1nvidia-persistenced
HowDriver flagUserspace process holds FDs
Survives rebootOnly if something re-applies itWith systemctl enable
Multi-tenant opsEasy to flip by accidentUnit + logs
ContainersHost only stillHost only still
When to useDebug, laptop, one-shotAlways-on GPU nodes

Both can show Enabled. Prefer the unit in production; keep -pm as a hammer when the unit is missing and you need a warm card now.

Burst math: short jobs win

One 12-hour train pays cold start once. A hundred 400 ms inference processes pay it a hundred times unless the driver stays warm. CI that launches a fresh Python per test case is the same shape.

Job burst wall clock
160s
total init tax · pedagogical
wall 180 s
3.2 s × 50
persist off180 s(init 160.0 + work 20)
persist on25 s(init 4.5 + work 20)
active mode wall180.0 s
inituseful work (fixed)cold bar width 100% · warm 14% of cold

50 short jobs × 3.2 s cold init ≈ 160 s of pure tax before useful work (20 s).

Batch inference, CI GPU tests, and Kubernetes rollouts multiply the tax. A long single training job pays cold start once — persistence barely moves the needle. Short, frequent processes are the whole point.

Where it matters

  • Batch / serverless-style inference — process per request or per batch
  • Kubernetes GPU pods — rolling updates, scale-out, preemption, canaries
  • CI GPU tests — suite after suite on the same node
  • Interactive scripts — python train.py every few minutes while you iterate
  • Multi-process frameworks that restart workers between runs

Where it barely matters

  • One long-running trainer or model server — the app already holds the device
  • Desktop display path — X/Wayland (or the compositor) already keeps the driver busy
  • One GPU job per day — three seconds is noise next to data movement
  • True power-down policies on rarely used hardware — you may want cold teardown

Pods: ready is not “container started”

Kubernetes will mark a container running long before the first CUDA open finishes. If your readiness probe hits GPU code (or your main process blocks on first device open), cold init sits on the critical path for Ready.

The daemon must run on the node, not in each pod. GPU Operator and similar stacks can install and manage it for you — still host-scoped.

Pod time-to-ready
5.5s
time-to-ready · pedagogical
GPU 3.2 s
cold 5.5 s → warm 2.4 s
schedule / pull / app cold GPU open ready

GPU open owns ~3.2 s of ready time. Container was “running”; probes still wait on cold driver init.

Autoscalers and rolling updates multiply pod starts. Warming the node with persistenced cuts the GPU slice; image pull and app boot still own the rest of the bar.

bash
# On the node (VM / bare metal / GPU node image)
sudo systemctl enable --now nvidia-persistenced
nvidia-smi --query-gpu=persistence_mode --format=csv,noheader

# From a pod — you should see Enabled; you should not run the daemon here
kubectl run gpu-check --rm -it --restart=Never \
  --image=nvidia/cuda:12.6.0-base-ubuntu22.04 \
  --limits='nvidia.com/gpu=1' \
  -- nvidia-smi --query-gpu=persistence_mode --format=csv,noheader

Docker is the same story: --gpus all sees host driver state. Warm the host once.

Cost, power, and multi-GPU

RSS: on the order of a few MB per process — often quoted ~2–4 MB class, not a training footprint.

VRAM: the daemon is not your model. You should not see multi-GB device allocations from persistenced itself.

Power: keeping the driver warm can mean the GPU is less eager to sit in the deepest idle. On always-on training nodes that is usually the right trade. On a laptop you open twice a week, leaving persistence off can be fine. Measure with your power tools if joules matter more than cold starts.

Multi-GPU / MIG: the daemon registers devices the driver exposes. After GPU hot-add, MIG reconfigure, or driver reload, confirm FDs and persistence_mode again — do not assume a unit that started at boot still matches the current topology.

Ops pitfalls

Most “we turned it on and nothing got faster” tickets are placement or expectation bugs.

Ops pitfalls
host
one failure mode at a time
Trap
Daemon inside every pod

Each container fights for device nodes, dies with the pod, and never warms the host for the next tenant.

→
Fix
systemd on the node

One nvidia-persistenced (or GPU Operator equivalent) on the host. Pods inherit warm driver state.

Most “persistence didn’t help” tickets are placement or expectation bugs: daemon not on the host, weights expected in VRAM, or a different cold layer (first CUDA context, model load, container pull).

What to do

  1. Install and enable on the host — systemd unit or GPU Operator path, not a sidecar in every pod.
  2. Verify three ways — persistence_mode, timed nvidia-smi after idle, and lsof on the daemon’s /dev/nvidia* FDs.
  3. Leave it on multi-tenant / high-churn nodes — cost is quiet process + small RSS.
  4. Do not expect VRAM or contexts to survive — only driver init state.
  5. If still slow after warm driver — profile model load, first context, and container start; persistence already did its job.
  6. Pair with sane device nodes — see NVIDIA device files.
bash
# Debian/Ubuntu
sudo apt-get install nvidia-persistenced
sudo systemctl enable --now nvidia-persistenced
sudo systemctl status nvidia-persistenced --no-pager

nvidia-smi --query-gpu=persistence_mode --format=csv,noheader
# Enabled

# After a quiet minute with no other clients
time nvidia-smi > /dev/null
# real ~0.1s class when warm

sudo lsof -p "$(pgrep -n nvidia-persistenced)" | grep /dev/nvidia
# nvidiactl + nvidia0… open

journalctl -u nvidia-persistenced -n 30 --no-pager
bash
# One-shot without the unit (debug / laptop)
sudo nvidia-smi -pm 1
sudo nvidia-smi -pm 0   # restore cold teardown policy

Quick failure checklist

SymptomCheck
Persistence-M Offunit active? -pm flipped? permission to enable?
On but still multi-second nvidia-smiwrong GPU visible? driver reload mid-flight?
Fast nvidia-smi, slow appmodel load / first CUDA context / Python imports
Works on host, slow in podruntime not injecting devices; not a persistence bug
Was fine, now cold after MIG changerestart unit / re-enable; re-verify FDs

Further reading

If you found this explanation helpful, consider sharing it with others.

Mastodon