An NVIDIA Xid 31 MMU fault is a GPU page-table-walk failure that crashes the active CUDA context. This article explains what triggers Xid 31, how to decode every field in the kernel-log error message, and how we eliminated 28 daily Xid 31 crashes in a production video pipeline processing 7,000+ videos.
Your GPU inference pipeline is processing thousands of videos per day. Then you see this in journalctl:
NVRM: Xid (PCI:0000:c1:00): 31, pid=2646416, name=python, channel 0x0000001a, intr 00000000. MMU Fault: ENGINE GRAPHICS GPC10 GPCCLIENT_T1_5 faulted @ 0x725f_fb800000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ
The CUDA context is dead. Your pipeline resets. And you have no idea why.
This article decodes every field in that error message, explains the GPU virtual memory system that produced it, and walks through the multi-allocator race condition that causes these faults in production video pipelines. We went from 28 crashes per day to zero — and this is exactly how.
The Error Message
Every field in an Xid 31 error carries diagnostic information. The challenge is knowing how to read it. Click any highlighted field below to see what it means and how to use it for debugging.
Select a highlighted field. The log is from the reported incident.
NVRM: Xid (PCI:0000:c1:00): 31, , , intr 00000000. MMU Fault: faulted @ . Fault is of type Part 1: The GPU’s Virtual Memory System
Why GPUs Have Virtual Memory
NVIDIA GPUs use a hardware Memory Management Unit (MMU) to translate virtual addresses, much like a CPU. Pascal supports a 49-bit virtual address space, with a capacity of 512 TiB. This is an architectural address capacity, not the amount of installed memory. GPU unified memory builds on virtual-memory support to make data accessible across processors.
A kernel uses virtual addresses for its buffers. Translation resolves those addresses to backing memory; cached translations can avoid repeating a full page-table walk.
The Page Table Walk
The GPU MMU uses a multi-level page table. The figure uses NVIDIA’s documented Pascal MMU format for a 4 KiB page: PD3, PD2, PD1, PD0, then the page table. Their index fields have 2, 9, 9, 8, and 9 bits, followed by a 12-bit page offset. Table formats and supported page sizes vary by architecture.
A directory entry leads to a lower table or, where supported, a larger-page mapping. At the leaf, the PTE resolves the physical page. Step through a successful walk, then select a missing PDE; the failing level is illustrative because the log does not specify it.
Virtual address 0x725f_fb800000
FAULT_PDE vs FAULT_PTE
FAULT_PDE identifies a failure at a page-directory entry. FAULT_PTE identifies a failure at the leaf page-table entry. Neither label by itself tells us the allocation size, whether the address was previously valid, or which allocator owned it.
Key Diagnostic Insight
A missing mapping is the observation; premature release is one possible explanation. Check pointer bounds, allocation records, ownership, and stream ordering before attributing the fault to cache cleanup.
Part 2: Decoding the Fault Address
The 49-Bit Virtual Address Space
A 49-bit byte-address space runs from 0x0 through 0x1_FFFF_FFFF_FFFF. The CUDA driver manages mappings for allocations within it. Frameworks do not each have a fixed address partition, and nearby address prefixes do not imply that live allocations overlap.
Our fault address is 0x725f_fb800000. It is 8 MiB aligned, but alignment alone does not establish the allocation’s size or owner. Select an allocator below to see its responsibilities and the evidence needed to identify a real allocation.
Schematic allocations within one virtual address space. Rows do not indicate fixed ranges, sizes, or overlapping live allocations.
Fault address 0x725f_fb800000Ownership cannot be inferred from this address alone.
Who Lives at These High Addresses?
In a multi-component GPU pipeline, several components manage allocations backed by CUDA and driver mappings:
| Allocator | What It Allocates |
|---|---|
| CUDA driver internals | Page tables, channel descriptors |
| PyTorch caching allocator | Tensors, model weights, intermediates |
| CuPy memory pool | Kernel outputs, NV12→RGB results |
| TensorRT workspace | Engine workspace, activation memory |
| NVDEC mapped surfaces | Mapped, postprocessed decoded output |
To identify the owner of 0x725f_fb800000, correlate it with pointer ranges, allocation sizes, and release events from the same process. The prefix alone cannot distinguish PyTorch, CuPy, TensorRT, or NVDEC.
Part 3: Who Faulted and Why
ENGINE GRAPHICS GPC10 GPCCLIENT_T1_5
ENGINE GRAPHICS identifies the reporting graphics/compute engine. GPC10 identifies a graphics processing cluster, while GPCCLIENT_T1_5 is an architecture-specific client identifier. These fields help locate the failing access, but do not identify who produced or released the buffer.
Use the full log alongside the GPU model, driver version, and kernel trace. The next field tells us the direction of the failed access.
ACCESS_TYPE_VIRT_READ
VIRT_READ means the operation attempted to read through a virtual address; VIRT_WRITE means it attempted to write. Neither direction identifies the engine or proves use-after-free. Invalid pointers, out-of-bounds accesses, and lifetime errors all need investigation.
The Timeline of a Use-After-Free Fault
One possible sequence: an allocation is valid, a consumer retains its address, and the backing mapping is released before that consumer finishes. A later access through the stale address can then fault. This is an illustration of a lifetime violation, not a diagnosis from the log alone.
Part 4: Why This Happens in GPU Video Pipelines
The Multi-Allocator Problem
This video pipeline combines four components with different memory-lifetime responsibilities:
- PyTorch caching allocator — manages tensor memory with lazy deallocation
- CuPy memory pool — manages CuPy array memory separately from PyTorch
- TensorRT workspace — pre-allocated scratch space for inference kernels
- NVDEC surfaces — decoder-managed pictures and mapped output buffers
Each allocator has its own view of what memory is “in use” vs “free.” The danger arises when allocators share memory across boundaries without coordinating lifecycle — especially through DLPack zero-copy transfers.
The DLPack Zero-Copy Trap
DLPack enables zero-copy sharing between frameworks: torch.from_dlpack(cupy_array) can create a tensor that shares the CuPy array’s storage. The DLPack specification includes managed lifetime and stream coordination, rather than merely passing an unowned pointer.
- The producer provides the shared storage and a managed tensor.
- The consumer assumes responsibility for calling the managed deleter when finished.
- Producer and consumer streams must be ordered so data is ready before use.
- External surface wrappers must also retain their underlying mapping until all consumers finish.
CuPy documents free_all_blocks() as releasing free blocks. It should not release an allocation still live through correctly managed sharing. A premature release therefore requires an ownership, wrapper, or synchronization failure; it is not an inevitable consequence of DLPack.
NVDEC Surface Mapping Lifecycle
The NVDEC programming guide distinguishes internal decode surfaces from postprocessed output surfaces returned by cuvidMapVideoFrame(). At most ulNumOutputSurfaces outputs may remain mapped simultaneously. Holding them indefinitely can exhaust that mapping limit; unmapping before consumers finish can invalidate their source.
Compare the configurations below. For a copied path, keep each source valid until its copy completes, then release it according to the wrapper or mapping API’s ownership contract.
- 1 · NVDECDecodeH.264 frames
- 2 · NVDECOutput formatRGB · 3 B/pixel
- 3 · NVDECDecoded surfaceRetained output
- 4 · CuPyConversion / copyShared view
- 5 · CuPyConsumer bufferProducer-owned
- 6 · PyTorchComputeRead RGB tensor
Authored pipeline. Mapped output surfaces and the decoder’s internal picture buffer (DPB) have separate limits. Byte counts exclude pitch and padding.
The DLPack Trap in Practice
Correct zero-copy sharing can be safe. Copying is an alternative that separates lifetimes, provided the copy completes before source release. Neither approach removes the need for stream ordering and a correct surface wrapper.
Part 5: How We Fixed It
The following counts and addresses are observations from our two-day investigation. The suspected mechanisms summarize that investigation; the log fields alone do not establish them.
| Day | Count | Type | Address Range | Suspected Mechanism |
|---|---|---|---|---|
| Day 0 | 28 | VIRT_WRITE | Various | Surface retention and output-memory pressure with RGB (3 B/px) |
| Day 1 | 2 | VIRT_READ | Prefixes (0x7b77, 0x0_48) | Lifetime or ordering issue while reading mapped NVDEC output |
| Day 1 | 3 | VIRT_READ | Prefixes (0x7f91, 0x7f3c, 0x7e64) | Conversion-buffer lifetime or ordering issue around cleanup |
| Day 2 | 1 | VIRT_READ | Prefixes (0x725f) | Residual allocator race — same mechanism as Day 1 batch 3 |
Reported case-study observations · Xid 31 faults/day
Final reported run: 7,000+ videos, approximately 630 videos/minute, 4 workers. This figure does not simulate or independently reproduce the incident.
Fix 1: NV12 Output Mode (28 → 2 Xid/day)
We switched the pipeline output from packed 8-bit RGB (3 bytes/pixel) to NV12 (1.5 bytes/pixel before pitch and padding). This halves ideal output storage, not necessarily the decoder’s internal DPB footprint. Our observed count fell from 28 to 2, and the remaining faults were reads; storage arithmetic alone does not prove the cause or a throughput improvement.
Fix 2: Two-Phase Clone (2 → 3 Xid/day, different type)
We copied mapped NV12 output into PyTorch-owned storage before the CuPy conversion kernel, retaining the mapped source until copying completed. The observed count rose from 2 to 3 as the investigation shifted to a different suspected mechanism around RGB conversion and cleanup. This was a changed pipeline, not a controlled comparison of clone overhead or reliability.
Fix 3: Clone RGB Output + Remove free_all_blocks() (3 → 0 Xid/day)
Two further changes preceded the zero-fault run:
- Clone CuPy RGB output into PyTorch memory — retain the source and establish conversion/copy completion before releasing it.
- Remove
free_all_blocks()from chunk cleanup — leave free blocks available for pool reuse and simplify cleanup timing while investigating ownership and stream ordering.
These changes separated buffer lifetimes. They do not establish that CuPy normally frees live allocations.
Results
After all three fixes, we observed 0 Xid 31 faults across 7,000+ videos processed, sustaining ~630 videos/minute on 4 inference workers. The pipeline copies decoded NV12 and converted RGB into consumer-owned storage. Each source remains valid until its copy completes. These are results from this production run, not a general guarantee or a benchmark reproduced by the figures.
Quick Reference for Xid 31 Diagnosis
Error Message Field Reference
| Field | Meaning |
|---|---|
PCI:0000:c1:00 | GPU PCI bus address |
31 | Xid error code — MMU fault |
pid=2646416 | Linux process ID that triggered the fault |
channel 0x1a | GPU hardware channel (execution queue) |
ENGINE GRAPHICS | Fault from compute/graphics engine (not COPY or NVDEC) |
GPC10 | Graphics Processing Cluster #10 |
GPCCLIENT_T1_5 | Architecture-specific client identifier within GPC10 |
0x725f_fb800000 | Virtual address that failed translation |
FAULT_PDE | Missing or invalid page-directory mapping |
ACCESS_TYPE_VIRT_READ | A kernel tried to read from the address |
Fault Type Comparison
| Fault | Reported Translation Level | What It Does Not Establish |
|---|---|---|
| FAULT_PDE | Page-directory entry | Allocation size, owner, or whether memory was freed |
| FAULT_PTE | Leaf page-table entry | Why the mapping is absent or invalid |
Access Type Comparison
| Access | Meaning | Additional Evidence Needed |
|---|---|---|
| VIRT_WRITE | Write through a virtual address | Reporting engine, kernel, pointer bounds and lifetime |
| VIRT_READ | Read through a virtual address | Reporting engine, kernel, pointer bounds and lifetime |
Address Range Heuristics
Address prefixes are useful correlation keys, not framework signatures.
| Evidence | What It Can Establish |
|---|---|
| Pointer plus allocation size | Whether the fault lies inside a recorded allocation |
| Allocation and release timestamps | Whether that allocation was live at the access |
| Owner and stream records | Which component retained the buffer and how work was ordered |
Sources
- NVIDIA Xid Error Documentation
- Analyzing Xid Errors with the Xid Catalog
- CUDA Virtual Memory Management
- NVDEC Video Decoder API Programming Guide
- Introducing Low-Level GPU Virtual Memory Management
- Pascal MMU Format Changes
- NVIDIA Developer Forums: FAULT_PDE Discussion
- Optimizing Video Memory with NVDECODE API
