NVIDIA HGX H100 8-GPU Baseboard Error SXid 120: NVSwitch Link Integrity NCCL Timeout
SXid 120 denotes a critical physical layer (PHY) or data link layer integrity failure on an NVLink connection between an H100 GPU and a 3rd-generation NVSwitch ASIC. When excessive symbol errors, flit CRC corruption, or lane degradation exceed threshold tolerances, the fabric drops packets, resulting in an unrecoverable NCCL all-reduce kernel timeout.
Hardware Diagnostic Matrix
Spec Ref #09320b04Required Diagnostic Tools & Instrumentation
Multimeter Probing Targets & Circuit Tolerances
Bench Calibrated ±5%| Test Point / Circuit | Target Nominal Spec | Expected State | Fault Condition |
|---|---|---|---|
| Main Logic DC Bus | 3.3V / 5.0V DC (±0.1V) | Stable DC Potential | <2.9V (Brownout / Sag) |
| Thermistor / NTC Sensor | 10kΩ - 100kΩ @ 25°C | Smooth Thermal Curve | 0Ω (Short) or ∞ (Open) |
| Safety Ground / PE Continuity | < 0.5Ω Resistance | Solid Earth Bond | > 1.0Ω (Floating Chassis) |
Common Symptoms & Fault Manifestations
- NCCL distributed training jobs stall and terminate with 'NCCL WARN: Call to connect returned Connection timed out' or 'unhandled system error'
- Host kernel dmesg logs reporting 'NVRM: Xid (PCI:0000:xx:00): 120, pid=xxxx, SXid: NVSwitch physical layer link degradation/failure'
- `nvidia-smi nvlink -s` shows degraded NVLink bandwidth or links reporting INACTIVE/DOWN state
- Fabric manager daemon (nvidia-fabricmanager) reports non-recoverable link retraining errors on NVSwitch endpoints
Root Causes & System Failure Modes
High-speed differential trace micro-fracture or high insertion loss across high-density mezzanine/cabled backplane interconnects
Interfacial socket oxidation or pin contamination on the SXM5 high-density socket (400V/54V delivery or 112G PAM4 signaling lines)
NVSwitch tray voltage regulator module (VRM) transient dip below 0.75V VDD core rail under high all-to-all communication load
Physical thermal throttling or localized hot-spotting on the NVSwitch ASIC exceeding 95°C junction temperature causing PHY lock loss
High-Voltage & Mechanical Safety Isolation
Prior to opening access panels, disconnecting terminals, or probing wiring, perform complete lock-out / tag-out (LOTO). Disconnect all AC mains, DC string isolators, and lithium battery banks. Allow internal smoothing capacitors at least 3 to 5 minutes to dissipate stored residual energy below safe threshold (<30VDC).
Step-by-Step Diagnostic & Resolution Procedure
Fabric Manager Isolation and Diagnostic Extraction
Halt all active compute containers and Kubernetes pods. Run `systemctl stop nvidia-fabricmanager` to prevent retry storms. Query telemetry using `nvidia-smi nvlink -e -i [GPU_ID]` and inspect `/var/log/nvidia-nvswitch/nvswitch.log` to pinpoint the specific NVSwitch chip number and port index exhibiting flit CRC and recovery count bursts.
Cold Power Cycle and Baseboard Discharge
Gracefully shut down the host server. Remove all redundant AC power supply feeds (C19/C20 cords). Hold the chassis power button for 20 seconds to discharge baseboard bulk decoupling capacitors. Ensure BMC status confirms auxiliary and payload rails are at 0V before chassis maintenance.
Interconnect Inspection, Reseating, and Cleaning
Open the server top cover and access the HGX baseboard. Inspect the SXM5 GPU module and NVSwitch cold plate connections. If link degradation persists on a specific module, unseat the SXM5 module using the cross-pattern torque sequence. Inspect high-speed pins for bending, burnishing, or foreign particulate. Clean with 99.9% IPA and dry nitrogen, then reseat and torque to exactly 0.6 Nm.
Firmware & Stress Verification via NVVS
Reapply power and ensure the Baseboard Management Controller (BMC) and Chassis Management Controller (CMC) boot clean. Restart nvidia-fabricmanager (`systemctl start nvidia-fabricmanager`). Execute the full level 4 diagnostics using `nvvs -t Long -d Plugin_Nvlink` and run an all-reduce test via `nccl-tests/build/all_reduce_perf -b 8M -e 8G -f 2 -g 8` to verify sustained 900 GB/s bidirectional NVLink bandwidth with 0 flit errors.
Hardware Reset & Fault Clear Sequence
Controller ProcedurePrevention & Routine Maintenance Guidelines
- Maintain intake ambient air temperatures below 25°C to avoid NVSwitch 112G PAM4 SerDes clock jitter induced by thermal drift.
- Schedule automated monthly NVLink bit-error rate (BER) checks using DCGM (Data Center GPU Manager) background telemetry.
Frequently Asked Diagnostic Questions
What does error code SXid 120 indicate on NVIDIA HGX H100 8-GPU Baseboard?
SXid 120 denotes a critical physical layer (PHY) or data link layer integrity failure on an NVLink connection between an H100 GPU and a 3rd-generation NVSwitch ASIC. When excessive symbol errors, flit CRC corruption, or lane degradation exceed threshold tolerances, the fabric drops packets, resulting in an unrecoverable NCCL all-reduce kernel timeout. In engineering telemetry, it indicates nvswitch link integrity nccl timeout requiring circuit verification.
What are the primary triggers causing error SXid 120?
Diagnostic bench telemetry isolates the following common root causes: High-speed differential trace micro-fracture or high insertion loss across high-density mezzanine/cabled backplane interconnects; Interfacial socket oxidation or pin contamination on the SXM5 high-density socket (400V/54V delivery or 112G PAM4 signaling lines); NVSwitch tray voltage regulator module (VRM) transient dip below 0.75V VDD core rail under high all-to-all communication load; Physical thermal throttling or localized hot-spotting on the NVSwitch ASIC exceeding 95°C junction temperature causing PHY lock loss.
How do I reset and clear NVIDIA error SXid 120?
Clear baseboard system latched errors via BMC IPMI/Redfish interface: issue `ipmitool raw 0x30 0x22` (OEM fabric clear) or perform a deep AC power cycle with 5-minute cold soak. Restart the host OS, re-enable the Fabric Manager service, and confirm all NVLinks indicate 'Active' via `nvidia-smi nvlink -s`. If the error status repeats immediately after clearing, discontinue clearing and test circuit continuity.
Can I safely resolve error SXid 120 without specialized shop tooling?
This fault is graded as Advanced difficulty with Critical severity. Standard diagnostics require an insulated digital multimeter and appropriate hand tools. Always observe high-voltage isolation procedures.