NVIDIA HGX H200 8-GPU Board Error XID 119: GSP-RM RPC Timeout Failure
An XID 119 event indicates an RPC communication failure between the host kernel driver and the GPU System Processor (GSP) firmware on one or more Hopper SXM5 modules, leading to NVLink interconnect desynchronization, NVSwitch fabric stalls, and dropped NCCL collective communication rings.
Hardware Diagnostic Matrix
Spec Ref #2af54330Required Diagnostic Tools & Instrumentation
Multimeter Probing Targets & Circuit Tolerances
Bench Calibrated ±5%| Test Point / Circuit | Target Nominal Spec | Expected State | Fault Condition |
|---|---|---|---|
| Main Logic DC Bus | 3.3V / 5.0V DC (±0.1V) | Stable DC Potential | <2.9V (Brownout / Sag) |
| Thermistor / NTC Sensor | 10kΩ - 100kΩ @ 25°C | Smooth Thermal Curve | 0Ω (Short) or ∞ (Open) |
| Safety Ground / PE Continuity | < 0.5Ω Resistance | Solid Earth Bond | > 1.0Ω (Floating Chassis) |
Common Symptoms & Fault Manifestations
- Kernel log reports 'NVRM: Xid (PCI:0000:xx:00): 119, pid=xxxx, RPC to GSP timed out'
- Distributed multi-GPU training jobs crash instantly with 'NCCL WARN Call to connect returned Connection refused / Network failure'
- nvidia-smi hangs indefinitely in D-state or reports GPUs in 'ERR!' / 'Unresponsive' state
Root Causes & System Failure Modes
GSP RISC-V firmware execution hang or internal deadlock triggered by memory parity error in high-bandwidth memory (HBM3e)
SXM5 connector pin degradation or localized 12V-to-0.8V VDD core VRM transient undervoltage trip under heavy NVSwitch line-rate traffic
PCIe Gen5 link degradation causing uncorrectable replay timer timeouts (TLP timeouts) between CPU root complex, PCIe switch, and SXM5 baseboard
NVSwitch fabric ASIC firmware desynchronization or thermals exceeding 85°C throttle limit
High-Voltage & Mechanical Safety Isolation
Prior to opening access panels, disconnecting terminals, or probing wiring, perform complete lock-out / tag-out (LOTO). Disconnect all AC mains, DC string isolators, and lithium battery banks. Allow internal smoothing capacitors at least 3 to 5 minutes to dissipate stored residual energy below safe threshold (<30VDC).
Step-by-Step Diagnostic & Resolution Procedure
Diagnostic Triage & Log Isolation
Capture 'nvidia-bug-report.sh' and query BMC SEL logs for SXM5 power supply fault asserts (PE_HOT, THERM_ALERT). Inspect 'dmesg -T' to identify the specific PCI BDF triggering the GSP RPC timeout.
Subsystem Power Cycle & Fabric Reset
A standard warm reboot fails to reinitialize the GSP. Execute an AC power cycle or complete chassis-level chassis cold boot via BMC IPMI/Redfish (`ipmitool chassis power cycle`) to cold-reset GSP-RM, SXM5 module regulators, and NVSwitch ASICs.
Firmware & Fabric Manager Alignment
Verify driver parity with the NVIDIA Fabric Manager daemon (`nvsm`, `nvidia-fabricmanager`). Ensure H200 driver version matches exactly with the firmware payload on SXM5 baseboard (minimum driver branch 535.161.08 / 550.54.15 or newer with GSP timeout hardening).
Physical Interconnect & Cold Re-seat
If XID 119 repeats consistently on the same BDF address, disconnect chassis AC mains, pull the tray, unseat the offending SXM5 module, inspect high-density mezzanine multi-pin arrays for bent/carbonized contacts, and re-torque to 1.2 Nm in an alternating X-pattern.
NCCL & Fabric Health Verification
Run NVIDIA System Management (`nvsm dump health`), execute Level 4 NVVS diagnostics (`nvsm test --level 4`), and run an all-reduce NCCL benchmark (`nccl-tests/build/all_reduce_perf -b 8M -e 8G -f 2 -g 8`) to verify 900 GB/s inter-GPU line-rate stability without RPC drops.
Hardware Reset & Fault Clear Sequence
Controller ProcedurePrevention & Routine Maintenance Guidelines
- Lock Driver and Fabric Manager versions via pinned repository packages to prevent mismatched user-space and kernel RPC APIs.
- Maintain delta-T chassis inlet temperatures below 25°C to avoid localized NVSwitch core hot-spots and transient power supply ripple.
Frequently Asked Diagnostic Questions
What does error code XID 119 indicate on NVIDIA HGX H200 8-GPU Board?
An XID 119 event indicates an RPC communication failure between the host kernel driver and the GPU System Processor (GSP) firmware on one or more Hopper SXM5 modules, leading to NVLink interconnect desynchronization, NVSwitch fabric stalls, and dropped NCCL collective communication rings. In engineering telemetry, it indicates gsp-rm rpc timeout failure requiring circuit verification.
What are the primary triggers causing error XID 119?
Diagnostic bench telemetry isolates the following common root causes: GSP RISC-V firmware execution hang or internal deadlock triggered by memory parity error in high-bandwidth memory (HBM3e); SXM5 connector pin degradation or localized 12V-to-0.8V VDD core VRM transient undervoltage trip under heavy NVSwitch line-rate traffic; PCIe Gen5 link degradation causing uncorrectable replay timer timeouts (TLP timeouts) between CPU root complex, PCIe switch, and SXM5 baseboard; NVSwitch fabric ASIC firmware desynchronization or thermals exceeding 85°C throttle limit.
How do I reset and clear NVIDIA error XID 119?
Perform a complete out-of-band cold chassis restart: 'ipmitool chassis power cycle', wait 90 seconds for baseboard power sequencer to complete, restart nvidia-fabricmanager via 'systemctl restart nvidia-fabricmanager', and verify all 8 GPUs appear clean via 'nvidia-smi -q -d PERFORMANCE'. If the error status repeats immediately after clearing, discontinue clearing and test circuit continuity.
Can I safely resolve error XID 119 without specialized shop tooling?
This fault is graded as Advanced difficulty with Critical severity. Standard diagnostics require an insulated digital multimeter and appropriate hand tools. Always observe high-voltage isolation procedures.