Skip to main content

Xid 31 MMU Faults: What Causes Them and How to Fix Production GPU Crashes

Deep dive into NVIDIA GPU Xid 31 MMU faults: how GPU virtual memory works, what causes page table walk failures, and how we eliminated 28 daily crashes in a production video pipeline processing 7,000+ videos.

An NVIDIA Xid 31 MMU fault is a GPU page-table-walk failure that crashes the active CUDA context. This article explains what triggers Xid 31, how to decode every field in the kernel-log error message, and how we eliminated 28 daily Xid 31 crashes in a production video pipeline processing 7,000+ videos.

Your GPU inference pipeline is processing thousands of videos per day. Then you see this in journalctl:

code
NVRM: Xid (PCI:0000:c1:00): 31, pid=2646416, name=python, channel 0x0000001a, intr 00000000. MMU Fault: ENGINE GRAPHICS GPC10 GPCCLIENT_T1_5 faulted @ 0x725f_fb800000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ

The CUDA context is dead. Your pipeline resets. And you have no idea why.

This article decodes every field in that error message, explains the GPU virtual memory system that produced it, and walks through the multi-allocator race condition that causes these faults in production video pipelines. We went from 28 crashes per day to zero — and this is exactly how.

The Error Message

Every field in an Xid 31 error carries diagnostic information. The challenge is knowing how to read it. Click any highlighted field below to see what it means and how to use it for debugging.

Decode an Xid 31 log

Select a highlighted field. The log is from the reported incident.

Kernel log
NVRM: Xid (PCI:0000:c1:00): 31, ,  , intr 00000000.  MMU Fault:    faulted @ .  Fault is of type  

Part 1: The GPU’s Virtual Memory System

Why GPUs Have Virtual Memory

NVIDIA GPUs use a hardware Memory Management Unit (MMU) to translate virtual addresses, much like a CPU. Pascal supports a 49-bit virtual address space, with a capacity of 512 TiB. This is an architectural address capacity, not the amount of installed memory. GPU unified memory builds on virtual-memory support to make data accessible across processors.

A kernel uses virtual addresses for its buffers. Translation resolves those addresses to backing memory; cached translations can avoid repeating a full page-table walk.

The Page Table Walk

The GPU MMU uses a multi-level page table. The figure uses NVIDIA’s documented Pascal MMU format for a 4 KiB page: PD3, PD2, PD1, PD0, then the page table. Their index fields have 2, 9, 9, 8, and 9 bits, followed by a 12-bit page offset. Table formats and supported page sizes vary by architecture.

A directory entry leads to a lower table or, where supported, a larger-page mapping. At the leaf, the PTE resolves the physical page. Step through a successful walk, then select a missing PDE; the failing level is illustrative because the log does not specify it.

GPU page-table walk
Translation

Virtual address 0x725f_fb800000

48:470x0PD3
46:380x1c9PD2
37:290xffPD1
28:210xdcPD0
20:120x0PT
11:00x0Offset
PD3Not visited
PD2Not visited
PD1Not visited
PD0Not visited
PTNot visited
DRAMNot visited
Resolved pathNot visitedMissing entry
Step 1 / 7

FAULT_PDE vs FAULT_PTE

FAULT_PDE identifies a failure at a page-directory entry. FAULT_PTE identifies a failure at the leaf page-table entry. Neither label by itself tells us the allocation size, whether the address was previously valid, or which allocator owned it.

Key Diagnostic Insight

A missing mapping is the observation; premature release is one possible explanation. Check pointer bounds, allocation records, ownership, and stream ordering before attributing the fault to cache cleanup.

Part 2: Decoding the Fault Address

The 49-Bit Virtual Address Space

A 49-bit byte-address space runs from 0x0 through 0x1_FFFF_FFFF_FFFF. The CUDA driver manages mappings for allocations within it. Frameworks do not each have a fixed address partition, and nearby address prefixes do not imply that live allocations overlap.

Our fault address is 0x725f_fb800000. It is 8 MiB aligned, but alignment alone does not establish the allocation’s size or owner. Select an allocator below to see its responsibilities and the evidence needed to identify a real allocation.

Who owns this GPU allocation?

Schematic allocations within one virtual address space. Rows do not indicate fixed ranges, sizes, or overlapping live allocations.

Fault address 0x725f_fb800000Ownership cannot be inferred from this address alone.

Who Lives at These High Addresses?

In a multi-component GPU pipeline, several components manage allocations backed by CUDA and driver mappings:

AllocatorWhat It Allocates
CUDA driver internalsPage tables, channel descriptors
PyTorch caching allocatorTensors, model weights, intermediates
CuPy memory poolKernel outputs, NV12→RGB results
TensorRT workspaceEngine workspace, activation memory
NVDEC mapped surfacesMapped, postprocessed decoded output

To identify the owner of 0x725f_fb800000, correlate it with pointer ranges, allocation sizes, and release events from the same process. The prefix alone cannot distinguish PyTorch, CuPy, TensorRT, or NVDEC.

Part 3: Who Faulted and Why

ENGINE GRAPHICS GPC10 GPCCLIENT_T1_5

ENGINE GRAPHICS identifies the reporting graphics/compute engine. GPC10 identifies a graphics processing cluster, while GPCCLIENT_T1_5 is an architecture-specific client identifier. These fields help locate the failing access, but do not identify who produced or released the buffer.

Use the full log alongside the GPU model, driver version, and kernel trace. The next field tells us the direction of the failed access.

ACCESS_TYPE_VIRT_READ

VIRT_READ means the operation attempted to read through a virtual address; VIRT_WRITE means it attempted to write. Neither direction identifies the engine or proves use-after-free. Invalid pointers, out-of-bounds accesses, and lifetime errors all need investigation.

The Timeline of a Use-After-Free Fault

One possible sequence: an allocation is valid, a consumer retains its address, and the backing mapping is released before that consumer finishes. A later access through the stale address can then fault. This is an illustration of a lifetime violation, not a diagnosis from the log alone.

Part 4: Why This Happens in GPU Video Pipelines

The Multi-Allocator Problem

This video pipeline combines four components with different memory-lifetime responsibilities:

  1. PyTorch caching allocator — manages tensor memory with lazy deallocation
  2. CuPy memory pool — manages CuPy array memory separately from PyTorch
  3. TensorRT workspace — pre-allocated scratch space for inference kernels
  4. NVDEC surfaces — decoder-managed pictures and mapped output buffers

Each allocator has its own view of what memory is “in use” vs “free.” The danger arises when allocators share memory across boundaries without coordinating lifecycle — especially through DLPack zero-copy transfers.

Buffer lifetime across allocators
Lifetime
1Allocate
2Share
3Queue
4Release
5Read
6Result
NVDEC
Decoded
·
·
·
·
·
CuPy
Allocate
·
·
·
·
·
PyTorch
—
·
·
·
·
·
Mapping
Mapped
·
·
·
·
·
CuPy bufferPyTorch consumerNVDEC source
Step 1 / 6

The DLPack Zero-Copy Trap

DLPack enables zero-copy sharing between frameworks: torch.from_dlpack(cupy_array) can create a tensor that shares the CuPy array’s storage. The DLPack specification includes managed lifetime and stream coordination, rather than merely passing an unowned pointer.

  • The producer provides the shared storage and a managed tensor.
  • The consumer assumes responsibility for calling the managed deleter when finished.
  • Producer and consumer streams must be ordered so data is ready before use.
  • External surface wrappers must also retain their underlying mapping until all consumers finish.

CuPy documents free_all_blocks() as releasing free blocks. It should not release an allocation still live through correctly managed sharing. A premature release therefore requires an ownership, wrapper, or synchronization failure; it is not an inevitable consequence of DLPack.

NVDEC Surface Mapping Lifecycle

The NVDEC programming guide distinguishes internal decode surfaces from postprocessed output surfaces returned by cuvidMapVideoFrame(). At most ulNumOutputSurfaces outputs may remain mapped simultaneously. Holding them indefinitely can exhaust that mapping limit; unmapping before consumers finish can invalidate their source.

Compare the configurations below. For a copied path, keep each source valid until its copy completes, then release it according to the wrapper or mapping API’s ownership contract.

Decode, hand off, compute
Configuration
  1. 1 · NVDECDecodeH.264 frames
  2. 2 · NVDECOutput formatRGB · 3 B/pixel
  3. 3 · NVDECDecoded surfaceRetained output
  4. 4 · CuPyConversion / copyShared view
  5. 5 · CuPyConsumer bufferProducer-owned
  6. 6 · PyTorchComputeRead RGB tensor

Authored pipeline. Mapped output surfaces and the decoder’s internal picture buffer (DPB) have separate limits. Byte counts exclude pitch and padding.

The DLPack Trap in Practice

Correct zero-copy sharing can be safe. Copying is an alternative that separates lifetimes, provided the copy completes before source release. Neither approach removes the need for stream ordering and a correct surface wrapper.

Part 5: How We Fixed It

The following counts and addresses are observations from our two-day investigation. The suspected mechanisms summarize that investigation; the log fields alone do not establish them.

DayCountTypeAddress RangeSuspected Mechanism
Day 028VIRT_WRITEVariousSurface retention and output-memory pressure with RGB (3 B/px)
Day 12VIRT_READPrefixes (0x7b77, 0x0_48)Lifetime or ordering issue while reading mapped NVDEC output
Day 13VIRT_READPrefixes (0x7f91, 0x7f3c, 0x7e64)Conversion-buffer lifetime or ordering issue around cleanup
Day 21VIRT_READPrefixes (0x725f)Residual allocator race — same mechanism as Day 1 batch 3
What changed across the fixes?

Reported case-study observations · Xid 31 faults/day

Final reported run: 7,000+ videos, approximately 630 videos/minute, 4 workers. This figure does not simulate or independently reproduce the incident.

Fix 1: NV12 Output Mode (28 → 2 Xid/day)

We switched the pipeline output from packed 8-bit RGB (3 bytes/pixel) to NV12 (1.5 bytes/pixel before pitch and padding). This halves ideal output storage, not necessarily the decoder’s internal DPB footprint. Our observed count fell from 28 to 2, and the remaining faults were reads; storage arithmetic alone does not prove the cause or a throughput improvement.

Fix 2: Two-Phase Clone (2 → 3 Xid/day, different type)

We copied mapped NV12 output into PyTorch-owned storage before the CuPy conversion kernel, retaining the mapped source until copying completed. The observed count rose from 2 to 3 as the investigation shifted to a different suspected mechanism around RGB conversion and cleanup. This was a changed pipeline, not a controlled comparison of clone overhead or reliability.

Fix 3: Clone RGB Output + Remove free_all_blocks() (3 → 0 Xid/day)

Two further changes preceded the zero-fault run:

  1. Clone CuPy RGB output into PyTorch memory — retain the source and establish conversion/copy completion before releasing it.
  2. Remove free_all_blocks() from chunk cleanup — leave free blocks available for pool reuse and simplify cleanup timing while investigating ownership and stream ordering.

These changes separated buffer lifetimes. They do not establish that CuPy normally frees live allocations.

Results

After all three fixes, we observed 0 Xid 31 faults across 7,000+ videos processed, sustaining ~630 videos/minute on 4 inference workers. The pipeline copies decoded NV12 and converted RGB into consumer-owned storage. Each source remains valid until its copy completes. These are results from this production run, not a general guarantee or a benchmark reproduced by the figures.

Quick Reference for Xid 31 Diagnosis

Error Message Field Reference

FieldMeaning
PCI:0000:c1:00GPU PCI bus address
31Xid error code — MMU fault
pid=2646416Linux process ID that triggered the fault
channel 0x1aGPU hardware channel (execution queue)
ENGINE GRAPHICSFault from compute/graphics engine (not COPY or NVDEC)
GPC10Graphics Processing Cluster #10
GPCCLIENT_T1_5Architecture-specific client identifier within GPC10
0x725f_fb800000Virtual address that failed translation
FAULT_PDEMissing or invalid page-directory mapping
ACCESS_TYPE_VIRT_READA kernel tried to read from the address

Fault Type Comparison

FaultReported Translation LevelWhat It Does Not Establish
FAULT_PDEPage-directory entryAllocation size, owner, or whether memory was freed
FAULT_PTELeaf page-table entryWhy the mapping is absent or invalid

Access Type Comparison

AccessMeaningAdditional Evidence Needed
VIRT_WRITEWrite through a virtual addressReporting engine, kernel, pointer bounds and lifetime
VIRT_READRead through a virtual addressReporting engine, kernel, pointer bounds and lifetime

Address Range Heuristics

Address prefixes are useful correlation keys, not framework signatures.

EvidenceWhat It Can Establish
Pointer plus allocation sizeWhether the fault lies inside a recorded allocation
Allocation and release timestampsWhether that allocation was live at the access
Owner and stream recordsWhich component retained the buffer and how work was ordered

Sources

Abhik Sarkar

Abhik Sarkar

Machine Learning Consultant specializing in Computer Vision and Deep Learning. Leading ML teams and building innovative solutions.

Share this article

If you found this article helpful, consider sharing it with your network

Mastodon