How Docker Works with GPUs: Device Files, Bind Mounts, and Driver Stacks

Your ML training container needs GPU access. But containers are supposed to be isolated — they have their own filesystem, their own process tree, their own network. How does a containerized process talk to physical GPU hardware?

The answer is surprisingly simple once you understand mount namespaces. GPU access is fundamentally a filesystem problem: applications talk to GPUs through device files, and the container runtime makes those files visible by bind-mounting them into the container’s mount namespace.

How Linux Exposes GPUs

Device Files Are the Interface

GPUs don’t have a special API. They appear to userspace as device files in /dev/, just like disks, terminals, and random number generators. The NVIDIA kernel driver (nvidia.ko) creates these character devices when it loads:

$ ls -la /dev/nvidia*
crw-rw-rw- 1 root root 195,   0 Mar 14 10:00 /dev/nvidia0
crw-rw-rw- 1 root root 195,   1 Mar 14 10:00 /dev/nvidia1
crw-rw-rw- 1 root root 195, 255 Mar 14 10:00 /dev/nvidiactl
crw-rw-rw- 1 root root 510,   0 Mar 14 10:00 /dev/nvidia-uvm
crw-rw-rw- 1 root root 510,   1 Mar 14 10:00 /dev/nvidia-uvm-tools

When a CUDA program runs, it doesn’t talk to the GPU directly. It opens /dev/nvidia0 (or whichever GPU) and issues ioctl() syscalls through that file descriptor. The kernel routes those calls to the registered NVIDIA kernel driver, which actually communicates with the hardware. So GPU access = file access.

What NVIDIA Container Runtime Does

A bare container has no GPU access. Its /dev/ directory contains only standard devices (null, zero, pts). The NVIDIA container runtime (nvidia-container-runtime) solves this by injecting three things into the container before the application starts:

1. Device nodes — bind mounts /dev/nvidia0, /dev/nvidiactl, /dev/nvidia-uvm from host
2. Driver libraries — bind mounts libcuda.so, libnvidia-ml.so and other driver-matched libraries
3. Device permissions — configures the cgroup device controller to allow access to NVIDIA device major numbers

All three are required. Device files alone aren’t enough (CUDA needs libcuda.so). Libraries alone aren’t enough (they need the device files to issue ioctl() calls). And even with both, the cgroup device controller must explicitly allow access.

The Two Things Called “NVIDIA Driver”

This is where most confusion happens. People say “NVIDIA driver” to mean two completely different things.

Kernel Driver (nvidia.ko)

The real driver. It’s a kernel module loaded on the host via modprobe nvidia. It runs in kernel space, talks directly to the GPU over PCIe, manages hardware resources, and handles memory allocation. Only one version can exist per kernel — you can’t run two different versions of nvidia.ko simultaneously.

$ lsmod | grep nvidia
nvidia_uvm           1503232  0
nvidia_drm             77824  0
nvidia_modeset       1306624  1 nvidia_drm
nvidia              56446976  2 nvidia_uvm,nvidia_modeset

All containers share this kernel driver because all containers share the host kernel. This is a fundamental property of Linux containers.

User-Space Libraries (libcuda.so)

These are not drivers. They’re client libraries that talk to the kernel driver through device files. libcuda.so is the CUDA driver API — it opens /dev/nvidia0 and issues ioctl() calls. libcudart.so is the CUDA runtime API that most applications use. libcublas.so, libcudnn.so, and others are higher-level libraries built on top.

Different containers can ship different versions of these libraries. The full call stack looks like:

PyTorch
  ↓
libcudart.so (CUDA Runtime — lives in container)
  ↓
libcuda.so (CUDA Driver API — bind-mounted from host)
  ↓
/dev/nvidia0 (device file — bind-mounted from host)
  ↓ ioctl() syscall — crosses user/kernel boundary
nvidia.ko (kernel driver — host kernel)
  ↓
GPU Hardware

CUDA Version Compatibility

Because the kernel driver is shared but CUDA libraries can differ per container, version compatibility matters. The rule is forward compatibility: newer host drivers support older CUDA toolkits, but not the reverse.

A host running NVIDIA driver 550 can serve containers using CUDA 11.8, 12.0, 12.1, or 12.4. But a host running driver 535 cannot serve a container using CUDA 12.4 — the container’s libcuda.so would try to call kernel driver APIs that don’t exist in the older driver.

GPU Access Methods Compared

There are several ways to give containers GPU access. They differ significantly in security and production-readiness.

Common Pitfalls

nvidia-smi works but CUDA fails

nvidia-smi uses libnvidia-ml.so to query the management interface. CUDA uses libcuda.so to submit compute work. They’re different code paths. If nvidia-smi shows your GPU but CUDA programs fail, the CUDA libraries are missing or mismatched — check that libcuda.so is properly mounted and its version matches the host kernel driver.

“No CUDA-capable device is detected”

The device files aren’t visible in the container. Either the runtime didn’t inject them (missing --gpus flag) or the cgroup device controller is blocking access. Check ls /dev/nvidia* inside the container.

Driver version mismatch

The container expects a newer CUDA version than the host driver supports. Check nvidia-smi on the host for the driver version, then verify it meets the minimum requirement for the container’s CUDA toolkit.

Container can’t see GPU after host driver update

After updating the host kernel driver, existing containers still have the old libcuda.so bind-mounted. Restart the container to pick up the new library version.

Related Concepts

• Linux Namespaces: Mount namespaces enable the bind-mount mechanism that exposes GPUs to containers
• Containers Under the Hood: How namespaces + cgroups combine to create containers
• Kernel Modules: How nvidia.ko is loaded and managed
• cgroups: Device controller that permits or blocks GPU access