NVIDIA H100 SXM5 80GB Error XID 31: H100 SXM5 XID 31 Fatal Memory Error
An uncorrectable multi-bit ECC error occurred in the High Bandwidth Memory (HBM3) stack, triggering NVIDIA driver event XID 31. The GPU driver attempted dynamic memory page retirement to isolate the failing physical address, but the retirement sequence failed or the bad page table exhausted its capacity, causing the GPU to fall off the bus and trigger a fatal kernel freeze.
Hardware Diagnostic Matrix
Spec Ref #f55b3680Required Diagnostic Tools & Instrumentation
Multimeter Probing Targets & Circuit Tolerances
Bench Calibrated ±5%| Test Point / Circuit | Target Nominal Spec | Expected State | Fault Condition |
|---|---|---|---|
| Main Logic DC Bus | 3.3V / 5.0V DC (±0.1V) | Stable DC Potential | <2.9V (Brownout / Sag) |
| Thermistor / NTC Sensor | 10kΩ - 100kΩ @ 25°C | Smooth Thermal Curve | 0Ω (Short) or ∞ (Open) |
| Safety Ground / PE Continuity | < 0.5Ω Resistance | Solid Earth Bond | > 1.0Ω (Floating Chassis) |
Common Symptoms & Fault Manifestations
- Kernel dmesg log displays 'NVRM: Xid (PCI:0000:xx:00): 31, GPU page retirement failed'
- CUDA application terminates immediately with 'CUDA error: uncorrectable ECC error encountered'
- nvidia-smi reports GPU state as 'ERR!' or becomes unresponsive due to fallen off the bus
- Host system hangs or crashes during multi-GPU NCCL all-reduce collective communication operations
Root Causes & System Failure Modes
Physical silicon degradation or microbump solder fracture within one of the HBM3 DRAM dies or silicon interposer
Persistent memory page retirement table saturation (>64 pending retired pages threshold exceeded)
Transient VDD_HBM rail voltage drop below 1.1V under high-current dI/dt transient loads during deep learning bursts
Thermal throttling failure with HBM3 junction temperature spiking above 95°C causing fatal bit flips
High-Voltage & Mechanical Safety Isolation
Prior to opening access panels, disconnecting terminals, or probing wiring, perform complete lock-out / tag-out (LOTO). Disconnect all AC mains, DC string isolators, and lithium battery banks. Allow internal smoothing capacitors at least 3 to 5 minutes to dissipate stored residual energy below safe threshold (<30VDC).
Step-by-Step Diagnostic & Resolution Procedure
Safe Node Drain and Isolation
Drain the node from the compute cluster (Kubernetes/Slurm). Terminate all CUDA workloads and issue 'nvidia-smi -i <GPU_INDEX> -pm 0' to release the driver handle before initiating diagnostic isolation.
Diagnostic Log Triage and Page Retirement Query
Execute 'nvidia-smi -q -d ECC,PAGE_RETIREMENT' to inspect uncorrectable error counts and retired page addresses. Check BMC system event logs (SEL) for corresponding PCIe AER (Advanced Error Reporting) fatal events.
Hardware Diagnostics and Forced Dynamic Retirement
Run NVIDIA System Validation Suite: 'nvvs -t golden_memory_test'. If NVVS reports persistent double-bit failures or dynamic page retirement exhaustion, attempt pending page clearance using 'nvidia-smi --clear-retired-pages' only if permitted by support guidelines, followed by a full AC power cycle.
SXM5 Interconnect Inspection & RMA Replacement
If XID 31 persists after power cycling, the HBM3 stack has suffered irreversible hardware damage. Disassemble the node chassis, inspect SXM5 mezzanine connector pins for physical debris, verify thermal interface material (TIM) coverage, and replace the failing H100 SXM5 GPU module via NVIDIA enterprise RMA.
Hardware Reset & Fault Clear Sequence
Controller ProcedurePrevention & Routine Maintenance Guidelines
- Maintain SXM5 cold-plate coolant inlet temperatures strictly below 35°C to limit HBM3 junction thermal stress.
- Implement proactive monitoring via DCGM (Data Center GPU Manager) alerts for XID 63 (Single-bit ECC) before escalation to XID 31.
- Ensure host firmware, HGX baseboard CPLD, and NVIDIA GPU drivers are updated to certified enterprise branches.
Frequently Asked Diagnostic Questions
What does error code XID 31 indicate on NVIDIA H100 SXM5 80GB?
An uncorrectable multi-bit ECC error occurred in the High Bandwidth Memory (HBM3) stack, triggering NVIDIA driver event XID 31. The GPU driver attempted dynamic memory page retirement to isolate the failing physical address, but the retirement sequence failed or the bad page table exhausted its capacity, causing the GPU to fall off the bus and trigger a fatal kernel freeze. In engineering telemetry, it indicates h100 sxm5 xid 31 fatal memory error requiring circuit verification.
What are the primary triggers causing error XID 31?
Diagnostic bench telemetry isolates the following common root causes: Physical silicon degradation or microbump solder fracture within one of the HBM3 DRAM dies or silicon interposer; Persistent memory page retirement table saturation (>64 pending retired pages threshold exceeded); Transient VDD_HBM rail voltage drop below 1.1V under high-current dI/dt transient loads during deep learning bursts; Thermal throttling failure with HBM3 junction temperature spiking above 95°C causing fatal bit flips.
How do I reset and clear NVIDIA error XID 31?
Execute a full host cold chassis power cycle via BMC CLI ('ipmitool power cycle' or Redfish system reset) to power-cycle the HGX baseboard VRMs, reload the GPU SPI-ROM firmware, and reset the HBM3 training registers. If the error status repeats immediately after clearing, discontinue clearing and test circuit continuity.
Can I safely resolve error XID 31 without specialized shop tooling?
This fault is graded as Advanced difficulty with Critical severity. Standard diagnostics require an insulated digital multimeter and appropriate hand tools. Always observe high-voltage isolation procedures.