You see NVRM: Xid in the kernel log after a GPU job fails. Perhaps one worker crashed while the others kept running. Perhaps nvidia-smi can no longer reach the device. The code is your first clue to what happened and what needs to recover.
This guide explains how to read that message, judge its impact, and investigate the error families you encounter in production. We will work through memory faults, application failures, and communication problems before deciding what the evidence says about the hardware.
Handling an incident now? Jump to triage or look up a code.
What Are Xid Errors?
The NVIDIA driver's kernel module runs on the host and reports GPU-related events to the kernel log. These messages begin with NVRM, the NVIDIA Resource Manager prefix, followed by an Xid number and details about the event. The GPU's own firmware and engines can contribute to those reports.
A simplified, authored log example looks like this:
NVRM: Xid (PCI:0000:65:00): 31, pid=1234, name=worker
The PCI address identifies the device, 31 identifies the event, and the PID/name help correlate the workload. Additional fields can identify an engine, faulting address, channel, or recovery state. Fields vary by driver and event; do not expect every line to contain all of them.
Three questions keep interpretation disciplined:
- What was reported? A memory access failed, an ECC error was contained, or a device became inaccessible.
- What must recover? An application, a GPU, a peer-to-peer workload, or a node.
- What caused it? That requires additional evidence: a reproduction, sanitizer report, hardware diagnostics, or correlated host telemetry.
The Xid catalog supplies the code definitions and recovery workflows. To apply them to your incident, first establish what failed and which workloads are affected.
Start Here: Triage Your Error
Start with the symptoms. Did one worker fail while others continued, or did several workloads lose access to the same GPU? Can nvidia-smi still reach it? Capture that state before changing the driver, resetting a GPU, or clearing counters. If the node is unstable, collect what remains available and record what could not be retrieved.
- Identify the resource. Record the hostname, time and timezone, GPU UUID and PCI bus address, driver version, and affected job or process. GPU indices can change after a reboot; correlate by UUID and PCI address.
- Preserve the sequence. Save the host kernel log around the failure, including the messages before the first Xid. Keep application logs and return codes alongside it. In a containerized environment, collect the host's kernel log.
- Determine the impact. Establish whether one process failed, several workloads on one GPU failed, or the device became inaccessible. Stop automatic retries if they keep returning work to a resource awaiting recovery.
- Read the recovery requirement. On drivers that report Xid 154, read its action text for the same GPU and incident. It can request peer-to-peer draining, draining and reset, GPU reset, or node reboot. It describes recovery for another event.
- Recover, validate, then investigate recurrence. Coordinate disruptive actions with the GPU's other users. After recovery, check device visibility, health flags, diagnostics, and a representative workload before reopening scheduling.
Here is a read-only starting point. Replace the example time window with the actual incident and retain the complete log; filtering is for inspection, not the only evidence you save.
# Host kernel messages with timestamps from the current boot. sudo journalctl -k -b --since "30 minutes ago" --no-pager > kernel-incident.log # Inspect Xids and nearby PCIe/AER messages in the saved log. grep -E 'NVRM: Xid|PCIe|AER' kernel-incident.log # Record identity, driver, and available health information. nvidia-smi --query-gpu=uuid,pci.bus_id,name,driver_version --format=csv nvidia-smi -q > nvidia-smi-incident.txt
Collect the application's first error, OS/kernel versions, recent changes, and the output of nvidia-bug-report.sh for escalation. HGX incidents may also need Fabric Manager logs. A command that hangs or returns no device is itself an observation; do not let one missing diagnostic block recovery. These evidence requirements follow NVIDIA's node-triage guidance.
Keep log lines in chronological order. Include the Xid 154 line if present. This helper reads code fields and recovery text; it does not diagnose a root cause. Text stays in this page.
Detected: Xid 94, Xid 45
Restart affected applicationPreserve the failure evidence and restart the failed process. Investigate recurrence before retrying it repeatedly. Other Xids or a reported recovery action can require wider recovery.
Xid 94Contained memory error
Restart the affected application and schedule reset when convenient for pending repair.
Interpretation: Contained means the impact was isolated. It does not mean the error was corrected or the application continued.
Memory and ECC · Vendor reference
Xid 45Preemptive channel cleanup
Check for process termination or a preceding error. Check Fabric Manager if it appears alone on a relevant platform.
Interpretation: Ctrl-C, process termination, and other recovery events can trigger cleanup. The GPU was not necessarily removed.
Cleanup and recovery reports · Vendor reference
Check reset prerequisites and platform-specific recovery before taking action.
A reset is a coordinated recovery operation
GPU reset requires supported hardware and platform configuration, appropriate privileges, and no active clients on the devices being reset. NVLink topology, Fabric Manager, MIG, and virtualization can affect the recovery procedure. Follow the platform runbook and the reported recovery action; a failed process restart is not permission to reset a shared GPU immediately.
The Severity Spectrum
Severity matters because it changes what you can safely restart. An application failure may leave other work running normally. An uncontained memory error can require GPU recovery before applications restart. If the device has become inaccessible, recovery may extend to the node. A cleanup message, meanwhile, can simply follow a process that already exited.
Keep the scope of that disruption separate from the cause. A reset requirement tells you how to recover; establishing permanent hardware damage requires further evidence.
The explorer below therefore groups codes by response. Read its entry conditions before acting. Follow the latest Xid 154 action for the same GPU and incident: “None” requests no GPU recovery, but an application that already failed still needs a new process. Recovery-state definitions.
Editorial groups for the initial response. Read each code's conditions and any Xid 154 action; these are not fixed severity ratings.
Restart affected application
GPU reset required
Node recovery / reboot
Review and schedule maintenance
Read the surrounding events
Check the vendor recovery guidance
Xid 94Contained memory error
Restart the affected application and schedule reset when convenient for pending repair.
Interpretation: Contained means the impact was isolated. It does not mean the error was corrected or the application continued.
Memory and ECC · Vendor reference
The memory reports below show why the recovery scope depends on the whole incident.
Category 1: Memory Errors
Memory-related Xids describe different parts of error handling. One report can identify an uncorrectable error, another can describe its containment, and a third can record a repair. Read them together before deciding which workloads can continue.
Xid 48: Double Bit ECC Error
Xid 48 reports an uncorrectable ECC error. Inspect the affected memory type and accompanying repair messages. NVIDIA's node-triage workflow distinguishes an isolated 48 from 48 followed by 63/64; the latter uses drain-and-reset handling. If the same incident also reports containment with Xid 94, check the current recovery flags and platform guidance before choosing the scope of recovery. Preserve evidence and follow diagnostics rather than treating one event as proof of permanently failing DRAM. Recovery workflow, containment recovery flags.
Xid 63: ECC Page Retirement
This older label needs an architecture qualification. On legacy GPUs, Xid 63 records page retirement; on supported Ampere-and-later GPUs it records row-remapping activity. Row remapping substitutes spare hardware rows and requires reset to activate pending changes. It is distinct from software page offlining and the legacy retirement limit. Page retirement versus row remapping.
Xid 94 and Xid 95: Contained vs Uncontained ECC
For Xid 94, restart the affected application. Other applications can remain unaffected. Check pending repair and schedule a convenient reset where the platform guidance permits it. For Xid 95, the error was uncontained and application restart alone is insufficient: recover the GPU before restarting applications. Follow any more specific node-reboot requirement, including Xid 154 text and the applicable MIG/platform procedure. NVIDIA containment guidance.
Containment describes how far the error's impact reached. It does not mean the error was corrected while the application continued. Xid 92 is the separate high-correctable-ECC-rate report; 140 indicates an unrecovered ECC condition that interrupted normal handling. Inspect the exact code and repair state.
Example: Recovering from a Contained Memory Error
Authored scenario: one job fails and the log contains Xid 94, a memory repair event, and cleanup messages.
Inspect the supported memory reports:
nvidia-smi -q -d ECC # Legacy page-retirement information, where supported. nvidia-smi -q -d PAGE_RETIREMENT # Ampere-and-later row-remapper information, where supported. nvidia-smi -q -d ROW_REMAPPER
N/A means that a field is unavailable or unsupported in this environment; it is not a zero-error reading. Distinguish volatile counters from aggregate history, and DRAM errors from SRAM errors. Record pending repairs and failure flags before recovery so you can compare afterward. The selectors above follow the nvidia-smi command reference.
A repair-recording event can accompany the failure without representing an additional independent incident. After the scheduled reset, inspect whether the pending repair was applied, whether failure flags remain, and whether diagnostics and the workload pass. “The process restarted” is only one part of recovery verification.
Xid 64: Memory Repair Recording Failure
Xid 64 concerns failed retirement/remapping recording, not a combined ECC/power diagnosis. The correct response depends on the platform: the general catalog calls for GPU reset, while NVIDIA's A100 node-triage guidance specifies immediate node reboot. Follow the applicable guidance and escalate recurrence. Platform recovery details.
Category 2: MMU Faults
A memory-access fault raises a different question: was the GPU allowed to access that address? The Memory Management Unit (MMU) checks address translation and access permissions. Its report helps you locate the failing access, while the memory-health reports above describe ECC and repair state.
Xid 31: GPU Page Table Fault
Read the fault type, engine, virtual address, and access direction in the message. Missing mappings and access permissions are possible investigation paths. FAULT_PDE identifies a directory-level translation problem, but does not establish that a memory pool was torn down. Read/write direction alone cannot establish use-after-free or identify NVDEC.
Example: Debugging a Worker with Xid 31
Authored debugging scenario: an inference worker fails while another workload continues on the same GPU. The host records Xid 31 followed by Xid 45. These are teaching examples, not recorded results from a production incident.
Start by correlating the PCI address and PID with the failed worker. Keep the MMU details with the reproduction so you can compare the failing access between runs. Those fields do not identify the allocator that owned the address or prove that memory was freed.
Preserve evidence and restart the affected application if the remaining messages and device state allow application-level recovery. A lost CUDA context should be recovered by restarting its owning process, rather than continuing to issue CUDA work through it.
To investigate the failure, reduce the input until the failure is reproducible. For a native CUDA reproduction, build with line information and run it under Compute Sanitizer:
# Use your reproduction source and its required build flags. nvcc -lineinfo repro.cu -o repro compute-sanitizer --tool memcheck --error-exitcode 1 ./repro
For a Python entry point, instrument the actual failing command:
compute-sanitizer --tool memcheck --error-exitcode 1 python repro.py # Separate localization experiment; this changes asynchronous execution. CUDA_LAUNCH_BLOCKING=1 python repro.py
A sanitizer report of an invalid read at a reproducible source location is useful evidence. For example, this illustrative kernel fragment needs a bounds check because the launch can round up the thread count:
// Before: excess threads access beyond the logical array length. int i = blockIdx.x * blockDim.x + threadIdx.x; out[i] = in[i]; // After: guard both accesses with the allocation's element count. if (i < n) { out[i] = in[i]; }
This fragment explains one possible defect; it does not promise a particular Xid on every GPU. Re-run the reproduction and sanitizer after the fix, then validate the real workload. If CUDA_LAUNCH_BLOCKING=1 changes behavior, investigate timing and error-reporting location; that change alone does not prove a stream race. Compute Sanitizer documentation.
For cross-framework buffers, check ownership and execution ordering. DLPack sharing includes managed lifetime and stream-handshake rules, and CuPy's free_all_blocks() releases free pool blocks. Test any suspected lifetime violation in a minimal reproduction, including the wrapper that transfers ownership, before attributing a live tensor's failure to pool cleanup. Copying can simplify ownership, but the source still has to stay valid until the copy completes. DLPack specification, CuPy memory-pool API.
Record the original failing input, the code or dependency change, and the result of rerunning the same test. If sanitizers are inconclusive or failures persist across controlled comparisons, keep driver and hardware causes open and escalate with the evidence bundle.
Category 3: Interconnect and Recovery
Communication faults shift the investigation toward the path between devices. PCIe and NVLink reports help identify that path; application and cleanup reports show what happened to the workloads using it.
Xid 32: PBDMA Error
Despite its older “invalid push buffer” description, this is a DMA-controller communication error, not an MMU page fault. Investigate the PCIe communication path and surrounding host logs. It should not inherit the Xid 31 allocator checklist merely because the numbers are adjacent. NVIDIA's PBDMA explanation.
Xid 79: Fallen Off the Bus
Xid 79 means the driver can no longer access the GPU over PCIe. The node may need a reboot even though the evidence has not yet isolated a GPU, host, riser, power, or driver fault. Preserve the available evidence before that recovery.
Example: Investigating an Inaccessible GPU
Authored scenario: a node reports Xid 79 and nvidia-smi can no longer access the affected GPU. Stop assigning new work to that node or device and preserve the available host/BMC logs before coordinated node recovery.
Inspect the PCIe path with the actual device address:
# Replace this example PCI address with the affected device. lspci -nn -s 0000:65:00.0 sudo lspci -vv -s 0000:65:00.0 sudo journalctl -k -b --since "30 minutes ago" --no-pager
Look for PCIe/AER messages and correlate them with power or thermal events recorded by the host. lspci may still enumerate a device that the driver cannot use. A low negotiated link speed at idle can also be power management; assess capabilities, width, load state, and error telemetry together before calling the connection degraded.
If the problem returns after recovery, use a controlled maintenance procedure to compare the suspect GPU, slot/riser, host, and power path. Change one factor at a time. A repeatable failure that follows the GPU strengthens a device-specific hypothesis; it does not replace the vendor's diagnostics or RMA process. NVIDIA's Xid 79 description.
Xid 74: NVLink Error
Inspect both endpoints and NVSwitch logs when applicable. NVLink is not limited to removable desktop bridges, and a remote GPU failure can make a healthy peer report a link error. Newer architectures have additional NVLink codes and subcodes; use the appropriate catalog for that GPU.
# Status and error counters are separate queries. nvidia-smi nvlink -s nvidia-smi nvlink -e
Do not reseat hardware based on the number alone. Apply the platform's maintenance procedure once the logs and topology identify a connection worth investigating. NVLink investigation, command meanings.
Xid 43: GPU Stopped Processing
Xid 43 can accompany a software-induced application failure while the GPU remains healthy. Do not use it as proof that a universal watchdog expired, and do not prescribe an unspecified NVreg_RegistryDwords timeout change. Investigate the failed application and neighboring messages. Xid 43 semantics.
Xid 45: Preemptive Channel Cleanup
Application termination and earlier faults can trigger this cleanup report. It does not mean the driver removed a physically failing GPU. A 43 → 45 → 79 sequence needs correlation, not an assumed progression toward hardware death. On relevant A100 systems, an isolated 45 also warrants checking Fabric Manager. Cleanup and platform context.
Category 4: Engine & Software Errors
Other reports identify an execution engine or a firmware component. The failing workload, API errors, and driver or firmware version become useful context here. Start with the component named in the report, then reproduce the operation that failed.
Xid 13: Graphics Engine Exception
A kernel or graphics workload encountered an execution exception. Bounds errors are a common debugging target, but the code is not proof that hardware and drivers are fault-free. Use Compute Sanitizer or cuda-gdb, and capture a small reproduction rather than assuming a particular instruction caused it.
Xid 61 and Xid 62: Internal Microcontroller Errors
Xid 61 reports a microcontroller breakpoint; Xid 62 reports a halt. Both need evidence capture and the documented reset/report workflow. A breakpoint report should not be dismissed as a harmless warning. Record driver and firmware versions; firmware changes should follow the platform support process. Modern 119/120 reports concern GSP timeout/error conditions and need their own recovery investigation. Microcontroller recovery guidance.
Xid 68: Video Processor Exception
In the current catalog, 68 identifies NVDEC0. It is not a generic NVENC encoder error. Save the failing input, codec/profile, decoder configuration, API errors, and driver version. A corrupt stream or an API/resource problem is a hypothesis to test, not a cause established by this code.
Xid 69: Graphics Engine Class Error
Investigate the graphics/compute API path and application behavior. Xid 69 is not a thermal-event code. Temperature and clock-throttling telemetry belong in the environmental checks below; they should not be inferred from this number.
Category 5: Power & Thermal
The device also depends on the host's power and cooling. When failures coincide with load, correlate temperature, power, clock-event reasons, and BMC telemetry at the incident timestamp. Use the device's reported limits rather than a universal 85°C threshold, and distinguish configured power limiting from a faulty power supply.
These environmental checks can help explain an incident, but Xid 64 and 69 have the specific memory-repair and engine meanings described above. Neither code identifies a power or thermal problem by itself.
nvidia-smi -q -d TEMPERATURE,POWER,PERFORMANCE # Continuous telemetry: power/temperature, clocks, ECC/PCIe replay counters. nvidia-smi dmon -s pce -d 60
Reading the Error Patterns
An incident can produce several of the codes above. A memory-error report, a repair event, and application cleanup may all describe the same failure. Reconstruct that sequence before counting events. Group by host, GPU identity, boot, and time window. Check which messages describe the initiating failure, which describe its consequences, and which specify recovery. Even the earliest available message may be downstream of an event that was not logged.
The manual timeline below shows how each additional observation changes interpretation. Its scenarios are authored illustrations. A burst or rising count alone cannot identify a dying GPU, a cosmic-ray event, or a driver defect.
Authored examples, not production telemetry. Event order and accompanying detail matter; these are not frequency-based predictions.
A worker reports Xid 31 and fails.
The MMU reported a memory-access fault. The code alone does not identify the invalid instruction or its cause.
Seen so far: 31
Restart affected applicationPreserve the failure evidence and restart the failed process. Investigate recurrence before retrying it repeatedly. Other Xids or a reported recovery action can require wider recovery.
Is It Hardware or Software?
The sequence tells you what to test next. To distinguish an application defect from a device or system problem, compare the failing workload and GPU under controlled conditions. A workload-specific failure can involve an application defect, a shared driver path, or hardware stressed by that access pattern. A failure across workloads can still involve shared software or infrastructure. A successful stress test establishes that the tested path passed under those conditions.
Keep a short experiment record: hypothesis, changed variable, fixed variables, observed result, and remaining alternatives. This turns “it worked once” into evidence another engineer can assess.
- Possible explanations
- An application defect, a driver path used by that workload, or hardware stressed by its access pattern could explain this.
- Next controlled test
- Reduce the failing input and run Compute Sanitizer. Compare the same reproduction on a known-good GPU while holding software versions fixed.
- What the result establishes
- An invalid access reported at a reproducible source location supports an application investigation. A clean run does not clear every driver or hardware path.
Authored investigation prompts, informed by NVIDIA's diagnostic workflow. No observation here automatically establishes RMA eligibility.
Quick Reference
Use this lookup when you have a code and need its meaning or recovery guidance. Search by number, subsystem, or keyword, then expand the entry for conditions and source links.
The reference covers 26 selected codes, checked on November 6, 2026. NVIDIA's full current catalog and legacy reference cover the wider code set. Match the guidance to your GPU, driver, virtualization mode, and platform runbook.
26 selected codes, reviewed 2026-11-06. Applicability varies by GPU and driver. Expand a result for context. Full NVIDIA catalog · Legacy reference
26 matching codes
Xid 8GPU stopped processing
Record the workload and driver version; escalate repeated failures.
Interpretation: The code alone does not identify a particular hung instruction.
Application and MMU · Vendor reference
Xid 13Graphics engine exception
Reproduce under Compute Sanitizer or cuda-gdb and inspect invalid accesses.
Interpretation: Application defects are common; driver and hardware failures remain possible.
Application and MMU · Vendor reference
Xid 31GPU memory access fault
Inspect the fault type, engine, address, and failing access in a minimal reproduction.
Interpretation: An MMU fault does not prove use-after-free, allocator conflict, or physical memory damage.
Application and MMU · Vendor reference
Xid 32PBDMA communication error
Correlate PCIe/AER and system logs with the failure, then follow platform diagnostics.
Interpretation: Do not classify this as an ordinary CUDA allocator bug; PCIe communication is a key suspect.
PCIe and NVLink · Vendor reference
Xid 43Application stopped processing
Inspect the application failure and preceding messages; restart the failed application.
Interpretation: The GPU may remain healthy. This is not a universal kernel-watchdog timeout.
Application and MMU · Vendor reference
Xid 45Preemptive channel cleanup
Check for process termination or a preceding error. Check Fabric Manager if it appears alone on a relevant platform.
Interpretation: Ctrl-C, process termination, and other recovery events can trigger cleanup. The GPU was not necessarily removed.
Cleanup and recovery reports · Vendor reference
Xid 48Uncorrectable ECC error
Check accompanying Xid 63/64, DRAM versus SRAM details, and repair flags; arrange diagnostics.
Interpretation: The 48 plus 63/64 workflow calls for drain and reset. If containment is also reported, check the current recovery flags and platform guidance. One event is not a universal RMA verdict.
Memory and ECC · Vendor reference
Xid 61Microcontroller breakpoint
Collect a bug report and firmware/driver versions; report the issue.
Interpretation: Do not dismiss a breakpoint as a harmless warning.
Firmware and engines · Vendor reference
Xid 62Microcontroller halt
Collect a bug report and coordinate recovery; escalate persistent failures.
Interpretation: The message does not by itself prove a failed chip or require a VBIOS flash.
Firmware and engines · Vendor reference
Xid 63Memory repair recording event
Check whether repair is pending and whether 48 or 94 accompanies it.
Interpretation: Legacy page retirement differs from Ampere-and-later row remapping. Pending remaps require reset to take effect.
Memory and ECC · Vendor reference
Xid 64Memory repair recording failure
Take the affected resource out of service and follow its reset/reboot requirements; escalate recurrence.
Interpretation: This is not a power-supply error code. A100 node-triage guidance calls for immediate reboot.
Memory and ECC · Vendor reference
Xid 68NVDEC0 exception
Save a small failing bitstream, decoder configuration, return codes, and driver version.
Interpretation: This identifies a decode engine; it does not establish corrupt input or a session-limit cause.
Firmware and engines · Vendor reference
Xid 69Graphics engine class error
Investigate the application/API path and capture a reproducible failure.
Interpretation: This is not a thermal event. Use temperature and clock-event telemetry for thermal diagnosis.
Application and MMU · Vendor reference
Xid 74NVLink connection error
Correlate both link endpoints, NVSwitch events where applicable, and link counters.
Interpretation: A remote device failure can produce this event. Do not assume a removable bridge is the cause.
PCIe and NVLink · Vendor reference
Xid 79GPU inaccessible over PCIe
Capture kernel PCIe/AER and BMC logs, then investigate the GPU, slot, riser, power, and driver.
Interpretation: Inaccessibility does not guarantee disappearance from lspci or prove the GPU itself is defective.
PCIe and NVLink · Vendor reference
Xid 92High correctable ECC error rate
Record error trends and arrange vendor diagnostics.
Interpretation: An elevated correctable-error rate needs investigation; it is different from Xid 94.
Memory and ECC · Vendor reference
Xid 93InfoROM wear limit reached
Check the provisioned wear-protection setting with the system vendor.
Interpretation: The setting may intentionally prevent writes; do not infer a failed flash device.
Firmware and engines · Vendor reference
Xid 94Contained memory error
Restart the affected application and schedule reset when convenient for pending repair.
Interpretation: Contained means the impact was isolated. It does not mean the error was corrected or the application continued.
Memory and ECC · Vendor reference
Xid 95Uncontained memory error
Reset before restarting applications; follow platform-specific reboot/MIG guidance.
Interpretation: Restarting processes alone is insufficient. Scope recovery using the platform and accompanying messages.
Memory and ECC · Vendor reference
Xid 109Context-switch timeout
Capture driver logs and the workload configuration; escalate repeated timeouts.
Interpretation: A timeout is an observation, not proof of an infinite loop in your CUDA kernel.
Firmware and engines · Vendor reference
Xid 119GSP RPC timeout
Capture the bug report and driver/GSP versions; investigate recurrence after recovery.
Interpretation: A timeout can involve firmware or communication. A power cycle may be needed if recovery fails.
Firmware and engines · Vendor reference
Xid 120GSP error
Capture firmware and driver diagnostics and follow the reported recovery action.
Interpretation: Do not equate a GSP failure with an automatic hardware replacement.
Firmware and engines · Vendor reference
Xid 121Corrected C2C link error
On a relevant Grace platform, record recurrence and review firmware guidance.
Interpretation: This is platform-specific; do not apply C2C recovery advice to an unrelated PCIe GPU.
PCIe and NVLink · Vendor reference
Xid 137NVLink peer access fault
Inspect peer-memory permissions and accesses with Compute Sanitizer or cuda-gdb.
Interpretation: A remote MMU fault is not the same as a broken NVLink cable.
Application and MMU · Vendor reference
Xid 140Unrecovered ECC error
Reset and contact the vendor if the error persists.
Interpretation: Normal page-offlining or row-remapping handling could not complete.
Memory and ECC · Vendor reference
Xid 154GPU recovery action changed
Read the action text and apply it to this GPU alongside its earlier errors.
Interpretation: This is recovery guidance about another event, not the initiating failure.
Cleanup and recovery reports · Vendor reference
Understanding the Categories
If you know which operation failed but are still narrowing down the relevant codes, browse by subsystem. These groups point you toward the diagnostic tools discussed above. Keep the surrounding messages: a fault can originate elsewhere and propagate to the subsystem that reports it.
These groups organize investigation; the reporting subsystem does not necessarily contain the original defect.
Xid 48Uncorrectable ECC error
Check accompanying Xid 63/64, DRAM versus SRAM details, and repair flags; arrange diagnostics.
Interpretation: The 48 plus 63/64 workflow calls for drain and reset. If containment is also reported, check the current recovery flags and platform guidance. One event is not a universal RMA verdict.
Memory and ECC · Vendor reference
Xid 63Memory repair recording event
Check whether repair is pending and whether 48 or 94 accompanies it.
Interpretation: Legacy page retirement differs from Ampere-and-later row remapping. Pending remaps require reset to take effect.
Memory and ECC · Vendor reference
Xid 64Memory repair recording failure
Take the affected resource out of service and follow its reset/reboot requirements; escalate recurrence.
Interpretation: This is not a power-supply error code. A100 node-triage guidance calls for immediate reboot.
Memory and ECC · Vendor reference
Xid 92High correctable ECC error rate
Record error trends and arrange vendor diagnostics.
Interpretation: An elevated correctable-error rate needs investigation; it is different from Xid 94.
Memory and ECC · Vendor reference
Xid 94Contained memory error
Restart the affected application and schedule reset when convenient for pending repair.
Interpretation: Contained means the impact was isolated. It does not mean the error was corrected or the application continued.
Memory and ECC · Vendor reference
Xid 95Uncontained memory error
Reset before restarting applications; follow platform-specific reboot/MIG guidance.
Interpretation: Restarting processes alone is insufficient. Scope recovery using the platform and accompanying messages.
Memory and ECC · Vendor reference
Xid 140Unrecovered ECC error
Reset and contact the vendor if the error persists.
Interpretation: Normal page-offlining or row-remapping handling could not complete.
Memory and ECC · Vendor reference
When to RMA
When the investigation points toward hardware, use reproducible evidence and vendor diagnostics to decide whether to escalate for replacement. “Xid 48 twice in a week,” “32 retired pages,” and “some ECC errors” are not universal replacement rules.
Legacy Tesla page-retirement documentation distinguishes a 64-page capacity from its RMA criteria: 60 retired pages, or evaluation at 15 or more with continuing retirement at least weekly. Those legacy thresholds must not be applied to current row-remapping GPUs. Legacy RMA criteria.
For GPUs using row remapping, inspect the failure flag and follow NVIDIA's applicable field-diagnostic process. Bank-level exhaustion, failures on previously remapped rows, and architecture-specific repair behavior matter; a single global counter is insufficient. SRAM has separate criteria. Memory RMA policy.
Send support the GPU identity, software versions, incident timestamps, full logs, bug report, diagnostic results, repair flags, and controlled comparisons already attempted. “Suspected hardware issue” is an evidence-based escalation state; it does not need to pretend that every alternative has been eliminated.
Monitoring Setup
Capture the same evidence automatically for the next incident. Build alerts around required recovery and workload impact. An application restart, a GPU reset, and an inaccessible node should route differently. Attach the preceding Xids, recovery text, GPU identity, job identity, and current health flags to the alert so the responder can reconstruct the incident.
Use host log collection for Xids and DCGM/your metrics system for health and utilization. Group related reports into incidents without discarding the raw messages. Track recurrence per GPU and workload over a stated time window; a count of cleanup messages is not a count of independent hardware failures.
# Live inspection; retain persistent host logs in your logging system too. sudo journalctl -k -f | grep --line-buffered 'NVRM: Xid'
Separate three outcomes in the incident record: service restored, cause established, and follow-up complete. A successful restart establishes the first. A reproducible fix or diagnostic finding supports the second. Verified health, applied repairs, and sustained workload checks close the third.
