The three paths that were not there
The image has CUDA. The host has an A100. nvidia-smi on the node is fine. Inside the pod:
CUDA error: CUDA_ERROR_NOT_INITIALIZED
or a blunt openat(.../dev/nvidiactl) = -1 ENOENT. Nobody rewrote your kernel. The character device nodes the driver publishes under /dev/ never made it into the container — or the open order hit a missing node first.
These files are not decoration. They are the only userspace door into nvidia.ko, nvidia-uvm.ko, and friends. Persistence is about keeping those doors warm; this page is about what the doors are.
Two numbers
1. Three nodes for single-GPU CUDA
/dev/nvidia0 · /dev/nvidiactl · /dev/nvidia-uvm. Forget ctl or uvm and modern stacks look “broken” while ls still shows a GPU device.
2. Major picks the driver · minor picks the instance
Character device 195, 0 is not “file size.” It is “NVIDIA major, GPU 0 minor.” Same major family for nvidiactl (195, 255); UVM has a dynamically assigned major (237 is only one possible example).
Flip the instruments until both feel mechanical.
Character devices, not disk files
The leading c in crw-rw-rw- means character device: unbuffered, ioctl-heavy, driver-backed. The pair after owner/group is major, minor.
$ ls -la /dev/nvidia*
crw-rw-rw- 1 root root 195, 0 /dev/nvidia0
crw-rw-rw- 1 root root 195, 1 /dev/nvidia1
crw-rw-rw- 1 root root 195, 255 /dev/nvidiactl
crw-rw-rw- 1 root root 195, 254 /dev/nvidia-modeset
crw-rw-rw- 1 root root 237, 0 /dev/nvidia-uvm
This is example output, not a fixed UVM device-number recipe. Inspect ls -l /dev/nvidia-uvm or stat /dev/nvidia-uvm, and check cat /proc/devices for the assigned nvidia-uvm major before setting device/cgroup rules.
Major 195 → classic NVIDIA control/compute nodes. The example UVM major 237 is dynamic; discover the actual value on your host. Major 226 → DRM (/dev/dri/*). Chip through the decoder:
Major picks the driver. Minor picks which node inside that driver. Same UVM dynamically receives its major; never hard-code 237 in device rules. Major 195 for nvidia0 and nvidiactl — different minors, same module family.
Open order: ctl first
Libraries do not start at /dev/nvidia0. They open /dev/nvidiactl to talk to the driver, enumerate GPUs, then open the per-GPU node, then (for modern paths) UVM.
- 1okopen(/dev/nvidiactl)fd held · continue
- 2okopen(/dev/nvidia0)fd held · continue
- 3okopen(/dev/nvidia-uvm)fd held · continue
Minimum character path for a single-GPU CUDA process. nvidia-container-toolkit mounts this set for you.
nvidiactl is not optional for discovery. A lone /dev/nvidia0 looks helpful and still fails at step 1.
Typical userspace shape (illustrative ioctl names — real IOCTLs are private):
int ctl = open("/dev/nvidiactl", O_RDWR);
// enum + driver queries on ctl
int gpu = open("/dev/nvidia0", O_RDWR);
int uvm = open("/dev/nvidia-uvm", O_RDWR);
// then cudaSetDevice / contexts / allocs
UVM is on the floor, not the balcony
cudaMallocManaged, page migration, and many framework init paths need /dev/nvidia-uvm. Mounting only nvidia0 is a classic incomplete hand---device list.
Container has nvidia0 + ctl. First real CUDA init or cudaMallocManaged fails hard.
nvidia-uvm-tools is optional (profiling). nvidia-uvm itself is not — treat it as part of the CUDA floor, not a debug extra.
/dev/nvidia-uvm-tools (minor 1) is optional — profilers and stats. Do not confuse it with the required UVM node.
Container: mount the set for the workload
Prefer nvidia-container-toolkit (--gpus) or the Kubernetes device plugin over a permanent hand-curated --device list. When you must reason about the list, match the workload:
- /dev/nvidia0required
- /dev/nvidiactlrequired
- /dev/nvidia-uvmrequired
- /dev/nvidia-uvm-toolsoptional
- /dev/nvidia-modesetskip
- /dev/dri/card0skip
- /dev/dri/renderD128skip
- /dev/nvidia-caps/*skip
Single-GPU CUDA / PyTorch: three character nodes. Toolkit fills the rest of nvidia*.
Three paths cover most CUDA pods: nvidia0 + nvidiactl + nvidia-uvm. Hand-mounting and forgetting ctl or uvm is the classic container trap.
# Prefer toolkit
docker run --rm --gpus all nvidia/cuda:12.6.0-base nvidia-smi
# Hand mount (debug only) — single GPU CUDA floor
docker run --rm \
--device=/dev/nvidia0 \
--device=/dev/nvidiactl \
--device=/dev/nvidia-uvm \
nvidia/cuda:12.6.0-base nvidia-smi
MIG and fabric add /dev/nvidia-caps/*. Display stacks add modeset and /dev/dri/*.
DRI: card vs render
DRM nodes are a parallel Linux interface (major 226), not a rename of nvidia0. Pure CUDA often never opens them. Desktop and headless GL do.
Compositor and modeset need card0 (and nvidia-modeset). Render alone is not enough for a session.
GPU index and render node index are related but not the same namespace. GPU 0 → often renderD128; GPU 1 → renderD129. cardN tracks DRM minors separately.
| Node | Typical use | Group |
|---|---|---|
/dev/dri/card0 | Display + modeset | video |
/dev/dri/renderD128 | Headless render / compute via DRM | render |
/dev/nvidia-modeset | NVIDIA KMS | varies |
Permissions and traps
Default 0666 on some installs hides access bugs until you harden to 0660 + groups. Missing nodes, incomplete mounts, and wrong GPU indices show up as “CUDA is broken.”
Forgot ctl or uvm. Symptom looks like “CUDA broken,” not “mount list incomplete.”
nvidia-container-toolkit (or device plugin) mounts the required set. Hand-mount only when debugging the set itself.
strace -e openat on a failing python -c "import torch" usually names the first missing path. Fix that path before rewriting CUDA code.
# See the first failing open
strace -e openat python -c "import torch" 2>&1 | grep nvidia
# Host nodes absent?
sudo modprobe -a nvidia nvidia_uvm
sudo nvidia-modprobe -u
# Groups (after hardening MODE=0660)
sudo usermod -aG video,render $USER
# re-login or newgrp
Udev sketch (adjust to site policy):
# /etc/udev/rules.d/70-nvidia.rules
KERNEL=="nvidia*", MODE="0660", GROUP="video"
KERNEL=="nvidia_uvm*", MODE="0660", GROUP="video"
Manual mknod is last-resort recovery — wrong major/minor or no module behind the node is worse than an empty ls.
What to do
- On the host, confirm
/dev/nvidiactl,/dev/nvidiaN,/dev/nvidia-uvmexist before debugging the app. - For CUDA containers, mount ctl + gpu + uvm (or use
--gpus/ the device plugin). - On
NOT_INITIALIZED/ weird init,stracethe firstopenatof an nvidia path. - Do not assume
nvidia0is the card in the top slot — check PCI mapping. - Pair with persistence on busy nodes so cold init is not paid every short job.
Related articles
- Solving GPU Boot Errors — driver load, initramfs, modprobe
- NVIDIA Persistence Daemon — keep
/dev/nvidia*warm between jobs - Unified Memory — why UVM exists
References
- NVIDIA UVM device registration source —
alloc_chrdev_regiondynamically assigns the UVM major - kmod modprobe manual —
--allloads every module named on the command line
Related concepts
Understand how containerized processes access GPU hardware through device files, bind mounts, and the NVIDIA container runtime. Learn the kernel driver vs user-space library distinction.
Deep dive into the CUDA context object: control vs data plane, inventory (memory, modules, streams, events, graphs), push/pop/setCurrent stacks, primary retain/release, flags and limits, isolation, cost, and traps.
Decision map for CUDA: a context is per-process GPU state, a stream is an in-order queue inside a context, and MPS co-schedules work across client contexts on Volta+. Pick the layer that matches the problem.
Why ordinary multi-process CUDA time slicing can leave SMs idle, how Volta+ MPS co-schedules separate client contexts, execution-resource caps, and when to choose MPS or MIG.
A CUDA stream is an in-order queue of GPU ops. Overlap H→D, kernels, and D→H across streams — and avoid the legacy default-stream fences between blocking streams in one context.
NVIDIA vs AMD for deep learning compared at both layers: the CUDA vs ROCm software moat, the microarchitecture (warp vs wavefront, SM vs CU, Tensor vs Matrix Cores), and the datacenter accelerators (H100/H200/B200 vs MI300X/MI325X).
