NVIDIA HGX H200 SXM5 Error XID 79: H200 GPU Fallen Off Bus
NVIDIA driver logged XID 79 indicating the GPU has fallen off the PCIe/NVLink bus. This is triggered when an uncorrectable double-bit ECC memory error (DBE) or link-layer parity collapse isolates the HBM3e subsystem, causing the ASIC microcontroller to halt execution and sever host MMIO communication.
Hardware Diagnostic Matrix
Spec Ref #520f2512Required Diagnostic Tools & Instrumentation
Multimeter Probing Targets & Circuit Tolerances
Bench Calibrated ±5%| Test Point / Circuit | Target Nominal Spec | Expected State | Fault Condition |
|---|---|---|---|
| Main Logic DC Bus | 3.3V / 5.0V DC (±0.1V) | Stable DC Potential | <2.9V (Brownout / Sag) |
| Thermistor / NTC Sensor | 10kΩ - 100kΩ @ 25°C | Smooth Thermal Curve | 0Ω (Short) or ∞ (Open) |
| Safety Ground / PE Continuity | < 0.5Ω Resistance | Solid Earth Bond | > 1.0Ω (Floating Chassis) |
Common Symptoms & Fault Manifestations
- nvidia-smi returns 'Unable to determine the device handle for GPU' or GPU listed as ERR!
- Host kernel dmesg logs 'NVRM: Xid (PCI:0000:xx:00): 79, GPU has fallen off the bus'
- Distributed NCCL training halts immediately with unhandled CUDA illegal instruction or bus error
- Baseboard Management Controller (BMC) reports SXM5 autonomous power gating or fatal PCIe AER
Root Causes & System Failure Modes
Uncorrectable HBM3e Multi-Bit ECC error triggering internal safety trip and bus detachment
NVLink 4th Gen mesh link integrity failure due to SXM5 mezzanine connector debris or uncalibrated torque
Transient rail voltage sag below nominal threshold (e.g., 54V intermediate bus to 0.8V VDD core VRM phase failure)
Thermal throttling induced silicon shutdown caused by cold plate microchannel particulate blockage
High-Voltage & Mechanical Safety Isolation
Prior to opening access panels, disconnecting terminals, or probing wiring, perform complete lock-out / tag-out (LOTO). Disconnect all AC mains, DC string isolators, and lithium battery banks. Allow internal smoothing capacitors at least 3 to 5 minutes to dissipate stored residual energy below safe threshold (<30VDC).
Step-by-Step Diagnostic & Resolution Procedure
System Isolation and Cold Reset
Halt all workload pods via orchestration (Kubernetes/Slurm). Perform a full AC auxiliary power cycle rather than a warm reboot: execute 'ipmitool chassis power cycle' or physically remove mains PDU feeds for 120 seconds to discharge the auxiliary rails and reset SXM5 volatile registers.
NVLink and PCIe AER Diagnostics
Boot to host OS in single-user mode. Inspect PCI Advanced Error Reporting and NVLink health using 'lspci -vvv -s <pci_id>' and 'nvidia-smi nvlink --status'. Check for fatal Uncorrectable Errors (UE) such as Flow Control Protocol Violation or Receiver Error on NVLink links.
SXM5 Socket & Mechanical Assembly Service
If the fault reoccurs, disconnect power, ground to ESD chassis, and unmount the liquid cold plate. Carefully dismount the faulty H200 module. Inspect the high-density SXM5 board-to-board connector under magnification for bent pins or debris. Clean with 99.9% IPA, reseat module, and tighten diagonally in cross pattern to manufacturer spec (0.9 Nm).
Burn-In Stress Verification and ECC Clearing
Power up system. Clear volatile and persistent ECC logs via 'nvidia-smi --clear-ecc-error-flags=all'. Execute NVIDIA System Management (NVVS) level 4 stress diagnostic: 'nvsm test run --level 4' alongside an unconstrained 30-minute NCCL all-reduce stress benchmark to verify link stability under thermal saturation.
Hardware Reset & Fault Clear Sequence
Controller ProcedurePrevention & Routine Maintenance Guidelines
- Ensure coolant chemistry and flow rates strictly match HGX H200 thermal loop specifications (minimum 1.5 LPM per module) to prevent VRM thermal spikes
- Schedule automated cron audits evaluating pending single-bit ECC (SBE) rates using nvidia-smi dmon to predict HBM degradation before DBE XID 79 events occur
Frequently Asked Diagnostic Questions
What does error code XID 79 indicate on NVIDIA HGX H200 SXM5?
NVIDIA driver logged XID 79 indicating the GPU has fallen off the PCIe/NVLink bus. This is triggered when an uncorrectable double-bit ECC memory error (DBE) or link-layer parity collapse isolates the HBM3e subsystem, causing the ASIC microcontroller to halt execution and sever host MMIO communication. In engineering telemetry, it indicates h200 gpu fallen off bus requiring circuit verification.
What are the primary triggers causing error XID 79?
Diagnostic bench telemetry isolates the following common root causes: Uncorrectable HBM3e Multi-Bit ECC error triggering internal safety trip and bus detachment; NVLink 4th Gen mesh link integrity failure due to SXM5 mezzanine connector debris or uncalibrated torque; Transient rail voltage sag below nominal threshold (e.g., 54V intermediate bus to 0.8V VDD core VRM phase failure); Thermal throttling induced silicon shutdown caused by cold plate microchannel particulate blockage.
How do I reset and clear NVIDIA error XID 79?
Drain intermediate bus capacitors via complete chassis AC power withdrawal for 2 minutes, then clear hardware fatal error logs in host BMC and reset driver state via: sudo systemctl restart nvidia-persistenced && sudo nvidia-smi --gpu-reset -i <GPU_INDEX>. If the error status repeats immediately after clearing, discontinue clearing and test circuit continuity.
Can I safely resolve error XID 79 without specialized shop tooling?
This fault is graded as Advanced difficulty with Critical severity. Standard diagnostics require an insulated digital multimeter and appropriate hand tools. Always observe high-voltage isolation procedures.