Skip to main content

NVIDIA Device Files in /dev/

How /dev/nvidia*, nvidiactl, nvidia-uvm, and DRI nodes map major/minor numbers to the driver, what CUDA opens first, and the minimum mount set for containers.

The three paths that were not there

The image has CUDA. The host has an A100. nvidia-smi on the node is fine. Inside the pod:

text
CUDA error: CUDA_ERROR_NOT_INITIALIZED

or a blunt openat(.../dev/nvidiactl) = -1 ENOENT. Nobody rewrote your kernel. The character device nodes the driver publishes under /dev/ never made it into the container — or the open order hit a missing node first.

These files are not decoration. They are the only userspace door into nvidia.ko, nvidia-uvm.ko, and friends. Persistence is about keeping those doors warm; this page is about what the doors are.

Two numbers

1. Three nodes for single-GPU CUDA
/dev/nvidia0 · /dev/nvidiactl · /dev/nvidia-uvm. Forget ctl or uvm and modern stacks look “broken” while ls still shows a GPU device.

2. Major picks the driver · minor picks the instance
Character device 195, 0 is not “file size.” It is “NVIDIA major, GPU 0 minor.” Same major family for nvidiactl (195, 255); UVM has a dynamically assigned major (237 is only one possible example).

Flip the instruments until both feel mechanical.

Character devices, not disk files

The leading c in crw-rw-rw- means character device: unbuffered, ioctl-heavy, driver-backed. The pair after owner/group is major, minor.

bash
$ ls -la /dev/nvidia*
crw-rw-rw- 1 root root 195,   0  /dev/nvidia0
crw-rw-rw- 1 root root 195,   1  /dev/nvidia1
crw-rw-rw- 1 root root 195, 255  /dev/nvidiactl
crw-rw-rw- 1 root root 195, 254  /dev/nvidia-modeset
crw-rw-rw- 1 root root 237,   0  /dev/nvidia-uvm

This is example output, not a fixed UVM device-number recipe. Inspect ls -l /dev/nvidia-uvm or stat /dev/nvidia-uvm, and check cat /proc/devices for the assigned nvidia-uvm major before setting device/cgroup rules.

Major 195 → classic NVIDIA control/compute nodes. The example UVM major 237 is dynamic; discover the actual value on your host. Major 226 → DRM (/dev/dri/*). Chip through the decoder:

Character device decode
195, 255
major, minor · character device
nvidia.ko
kernel module
$ ls -la /dev/nvidiactl
crw-rw-rw- 1 root root 195 255 /dev/nvidiactl
c = char195 = driver255 = instance
Driver-wide control. Open first to enumerate GPUs and init.

Major picks the driver. Minor picks which node inside that driver. Same UVM dynamically receives its major; never hard-code 237 in device rules. Major 195 for nvidia0 and nvidiactl — different minors, same module family.

Open order: ctl first

Libraries do not start at /dev/nvidia0. They open /dev/nvidiactl to talk to the driver, enumerate GPUs, then open the per-GPU node, then (for modern paths) UVM.

Open path
OK
init · pedagogical
typical order: ctl → gpuN → uvm
  1. 1
    open(/dev/nvidiactl)
    fd held · continue
    ok
  2. 2
    open(/dev/nvidia0)
    fd held · continue
    ok
  3. 3
    open(/dev/nvidia-uvm)
    fd held · continue
    ok

Minimum character path for a single-GPU CUDA process. nvidia-container-toolkit mounts this set for you.

nvidiactl is not optional for discovery. A lone /dev/nvidia0 looks helpful and still fails at step 1.

Typical userspace shape (illustrative ioctl names — real IOCTLs are private):

c
int ctl = open("/dev/nvidiactl", O_RDWR);
// enum + driver queries on ctl
int gpu = open("/dev/nvidia0", O_RDWR);
int uvm = open("/dev/nvidia-uvm", O_RDWR);
// then cudaSetDevice / contexts / allocs

UVM is on the floor, not the balcony

cudaMallocManaged, page migration, and many framework init paths need /dev/nvidia-uvm. Mounting only nvidia0 is a classic incomplete hand---device list.

UVM gate
INIT ERR
/dev/nvidia-uvm · major assigned dynamically
runtime blockedmanaged no
/dev/nvidia-uvm
node absent in namespace · page migration driver unreachable
cudaSetDevice(0);
// needs nvidia-uvm for modern paths
cudaMallocManaged(&p, n);
// → CUDA_ERROR_NOT_INITIALIZED

Container has nvidia0 + ctl. First real CUDA init or cudaMallocManaged fails hard.

nvidia-uvm-tools is optional (profiling). nvidia-uvm itself is not — treat it as part of the CUDA floor, not a debug extra.

/dev/nvidia-uvm-tools (minor 1) is optional — profilers and stats. Do not confuse it with the required UVM node.

Container: mount the set for the workload

Prefer nvidia-container-toolkit (--gpus) or the Kubernetes device plugin over a permanent hand-curated --device list. When you must reason about the list, match the workload:

Container mount set
3
required · 3 paths lit
prefer --gpus / device plugin
not hand --device forever
  • /dev/nvidia0required
  • /dev/nvidiactlrequired
  • /dev/nvidia-uvmrequired
  • /dev/nvidia-uvm-toolsoptional
  • /dev/nvidia-modesetskip
  • /dev/dri/card0skip
  • /dev/dri/renderD128skip
  • /dev/nvidia-caps/*skip

Single-GPU CUDA / PyTorch: three character nodes. Toolkit fills the rest of nvidia*.

Three paths cover most CUDA pods: nvidia0 + nvidiactl + nvidia-uvm. Hand-mounting and forgetting ctl or uvm is the classic container trap.

bash
# Prefer toolkit
docker run --rm --gpus all nvidia/cuda:12.6.0-base nvidia-smi

# Hand mount (debug only) — single GPU CUDA floor
docker run --rm \
  --device=/dev/nvidia0 \
  --device=/dev/nvidiactl \
  --device=/dev/nvidia-uvm \
  nvidia/cuda:12.6.0-base nvidia-smi

MIG and fabric add /dev/nvidia-caps/*. Display stacks add modeset and /dev/dri/*.

DRI: card vs render

DRM nodes are a parallel Linux interface (major 226), not a rename of nvidia0. Pure CUDA often never opens them. Desktop and headless GL do.

DRI: card vs render
card0
major 226 · nvidia-drm
linux groups
video + render
/dev/dri/card0
226, 0
primary · modeset + display
in path
/dev/dri/renderD128
226, 128
render · headless GPU
in path
/dev/nvidia-modeset
195, 254
NVIDIA KMS helper
in path

Compositor and modeset need card0 (and nvidia-modeset). Render alone is not enough for a session.

GPU index and render node index are related but not the same namespace. GPU 0 → often renderD128; GPU 1 → renderD129. cardN tracks DRM minors separately.

NodeTypical useGroup
/dev/dri/card0Display + modesetvideo
/dev/dri/renderD128Headless render / compute via DRMrender
/dev/nvidia-modesetNVIDIA KMSvaries

Permissions and traps

Default 0666 on some installs hides access bugs until you harden to 0660 + groups. Missing nodes, incomplete mounts, and wrong GPU indices show up as “CUDA is broken.”

Device traps
toolkit
sticky fix word
trap
docker --device nvidia0 only

Forgot ctl or uvm. Symptom looks like “CUDA broken,” not “mount list incomplete.”

do this
--gpus / CDI / plugin

nvidia-container-toolkit (or device plugin) mounts the required set. Hand-mount only when debugging the set itself.

strace -e openat on a failing python -c "import torch" usually names the first missing path. Fix that path before rewriting CUDA code.

bash
# See the first failing open
strace -e openat python -c "import torch" 2>&1 | grep nvidia

# Host nodes absent?
sudo modprobe -a nvidia nvidia_uvm
sudo nvidia-modprobe -u

# Groups (after hardening MODE=0660)
sudo usermod -aG video,render $USER
# re-login or newgrp

Udev sketch (adjust to site policy):

bash
# /etc/udev/rules.d/70-nvidia.rules
KERNEL=="nvidia*", MODE="0660", GROUP="video"
KERNEL=="nvidia_uvm*", MODE="0660", GROUP="video"

Manual mknod is last-resort recovery — wrong major/minor or no module behind the node is worse than an empty ls.

What to do

  1. On the host, confirm /dev/nvidiactl, /dev/nvidiaN, /dev/nvidia-uvm exist before debugging the app.
  2. For CUDA containers, mount ctl + gpu + uvm (or use --gpus / the device plugin).
  3. On NOT_INITIALIZED / weird init, strace the first openat of an nvidia path.
  4. Do not assume nvidia0 is the card in the top slot — check PCI mapping.
  5. Pair with persistence on busy nodes so cold init is not paid every short job.

References

Systems & Architecture
How Docker Works with GPUs: Device Files, Bind Mounts, and Driver Stacks

Understand how containerized processes access GPU hardware through device files, bind mounts, and the NVIDIA container runtime. Learn the kernel driver vs user-space library distinction.

GPU & High-Performance Computing
CUDA Contexts: Ownership, Current Stack, Isolation

Deep dive into the CUDA context object: control vs data plane, inventory (memory, modules, streams, events, graphs), push/pop/setCurrent stacks, primary retain/release, flags and limits, isolation, cost, and traps.

GPU & High-Performance Computing
CUDA Context vs Streams vs MPS: Which Layer Fixes What

Decision map for CUDA: a context is per-process GPU state, a stream is an in-order queue inside a context, and MPS co-schedules work across client contexts on Volta+. Pick the layer that matches the problem.

GPU & High-Performance Computing
CUDA Multi-Process Service (MPS): Concurrent Client Contexts

Why ordinary multi-process CUDA time slicing can leave SMs idle, how Volta+ MPS co-schedules separate client contexts, execution-resource caps, and when to choose MPS or MIG.

GPU & High-Performance Computing
CUDA Streams: Asynchronous Execution and Concurrency

A CUDA stream is an in-order queue of GPU ops. Overlap H→D, kernels, and D→H across streams — and avoid the legacy default-stream fences between blocking streams in one context.

GPU & High-Performance Computing
NVIDIA vs AMD for Deep Learning: CUDA vs ROCm and the Datacenter Accelerators

NVIDIA vs AMD for deep learning compared at both layers: the CUDA vs ROCm software moat, the microarchitecture (warp vs wavefront, SM vs CU, Tensor vs Matrix Cores), and the datacenter accelerators (H100/H200/B200 vs MI300X/MI325X).

If you found this explanation helpful, consider sharing it with others.

Mastodon