NVIDIA HGX H100 8-GPU Board Error SXid 119: NVSwitch Fabric Link Down Uncorrectable
An uncorrectable physical layer or data link layer error occurred on the NVLink 4.0 interconnect between an H100 Tensor Core GPU and an on-board NVSwitch 3 ASIC, triggering a fatal link-down event and hardware isolation.
Hardware Diagnostic Matrix
Spec Ref #475727b6Required Diagnostic Tools & Instrumentation
Multimeter Probing Targets & Circuit Tolerances
Bench Calibrated ±5%| Test Point / Circuit | Target Nominal Spec | Expected State | Fault Condition |
|---|---|---|---|
| Main Logic DC Bus | 3.3V / 5.0V DC (±0.1V) | Stable DC Potential | <2.9V (Brownout / Sag) |
| Thermistor / NTC Sensor | 10kΩ - 100kΩ @ 25°C | Smooth Thermal Curve | 0Ω (Short) or ∞ (Open) |
| Safety Ground / PE Continuity | < 0.5Ω Resistance | Solid Earth Bond | > 1.0Ω (Floating Chassis) |
Common Symptoms & Fault Manifestations
- Kernel dmesg reports 'SXid (PCI:0000:XX:00.0): 119, NVLink fabric link down uncorrectable error'
- NCCL communications abort with 'NCCL WARN Call to connect returned Connection refused / remote peer unreachable'
- nvidia-smi displays NVLink state as INACTIVE or degraded on affected GPU/NVSwitch ports
Root Causes & System Failure Modes
Differential high-speed trace degradation or micro-fracture along the 106.25 Gbps PAM4 NVLink 4 channels on the HGX baseboard
Transient rail voltage sag below 0.75V VDD on the associated NVSwitch ASIC core or NVLink I/O phy power rails
Thermal throttling or junction over-temperature trip exceeding 105°C on NVSwitch silicon forcing SerDes PHY shutdown
Contaminated or improperly seated high-density interposer/mezzanine connector between the GPU SXM5 socket and the HGX baseboard
High-Voltage & Mechanical Safety Isolation
Prior to opening access panels, disconnecting terminals, or probing wiring, perform complete lock-out / tag-out (LOTO). Disconnect all AC mains, DC string isolators, and lithium battery banks. Allow internal smoothing capacitors at least 3 to 5 minutes to dissipate stored residual energy below safe threshold (<30VDC).
Step-by-Step Diagnostic & Resolution Procedure
Safe Isolation & Topology Query
Drain active SLURM/Kubernetes workloads from the node. Run 'nvidia-smi nvlink -s' and 'nvsm dump log' to isolate the failing GPU index, NVSwitch ID, and physical port mask. Execute host shutdown via BMC and disconnect dual 3000W redundant AC power feeds.
Physical Inspection & Mezzanine Seating
Don ESD grounding. Open the HGX server top chassis. Inspect liquid cooling loops or cold plates for coolant leaks around the 4 NVSwitches and 8 SXM5 sockets. Verify GPU torque screws match factory spec (1.1 Nm in diagonal cross-pattern). Inspect board surface for thermal discoloration or VRM inductor cracks.
Firmware Verification & NVSwitch Reset
Reconnect power and boot into BMC. Check NVSwitch EEPROM firmware version via 'nv-hostengine' or BMC console; align all switches to the certified HGX firmware baseboard management matrix. Perform an out-of-band AC power power-cycle ('ipmitool chassis power cycle') to force cold re-enumeration of the SerDes PHY links.
Stress Testing & NVLink Fabric Recalibration
Boot host OS and run NVIDIA System Management validation: 'nvvs -t -d 3'. Execute all2all bandwidth test using 'nccl-tests' (build/all_reduce_perf -b 8G -e 8G -f 2 -g 8). Confirm zero re-train events, bit error rate (BER) below 1e-12, and symmetric NVLink throughput of 900 GB/s per GPU.
Hardware Reset & Fault Clear Sequence
Controller ProcedurePrevention & Routine Maintenance Guidelines
- Maintain inlet ambient datacenter temperature between 18°C-22°C to prevent PAM4 SerDes clock drift from thermal expansion
- Regularly poll BMC telemetry for NVSwitch rail voltages (0.75V core, 1.2V I/O) to preemptively catch VRM degradation
Frequently Asked Diagnostic Questions
What does error code SXid 119 indicate on NVIDIA HGX H100 8-GPU Board?
An uncorrectable physical layer or data link layer error occurred on the NVLink 4.0 interconnect between an H100 Tensor Core GPU and an on-board NVSwitch 3 ASIC, triggering a fatal link-down event and hardware isolation. In engineering telemetry, it indicates nvswitch fabric link down uncorrectable requiring circuit verification.
What are the primary triggers causing error SXid 119?
Diagnostic bench telemetry isolates the following common root causes: Differential high-speed trace degradation or micro-fracture along the 106.25 Gbps PAM4 NVLink 4 channels on the HGX baseboard; Transient rail voltage sag below 0.75V VDD on the associated NVSwitch ASIC core or NVLink I/O phy power rails; Thermal throttling or junction over-temperature trip exceeding 105°C on NVSwitch silicon forcing SerDes PHY shutdown; Contaminated or improperly seated high-density interposer/mezzanine connector between the GPU SXM5 socket and the HGX baseboard.
How do I reset and clear NVIDIA error SXid 119?
Perform a complete cold chassis cycle: 'ipmitool raw 0x04 0x12' (if vendor-supported) or 'ipmitool power cycle', allowing at least 60 seconds of auxiliary power hold to purge NVSwitch capacitive charge and clear hardware latch registers. If the error status repeats immediately after clearing, discontinue clearing and test circuit continuity.
Can I safely resolve error SXid 119 without specialized shop tooling?
This fault is graded as Advanced difficulty with Critical severity. Standard diagnostics require an insulated digital multimeter and appropriate hand tools. Always observe high-voltage isolation procedures.