NVIDIA HGX H200 8-GPU Baseboard Error SXid 120 / NCCL Timeout: NVLink Link Down Timeout Error
An NVLink high-speed interconnect physical/logical link degraded or dropped unexpectedly, triggering SXid 120 (NVLink Link Down). Because point-to-point ring/tree communication halted mid-collective operation, the NCCL execution watchdog timer expired with a collective timeout.
Hardware Diagnostic Matrix
Spec Ref #4f46e05fRequired Diagnostic Tools & Instrumentation
Multimeter Probing Targets & Circuit Tolerances
Bench Calibrated ±5%| Test Point / Circuit | Target Nominal Spec | Expected State | Fault Condition |
|---|---|---|---|
| Main Logic DC Bus | 3.3V / 5.0V DC (±0.1V) | Stable DC Potential | <2.9V (Brownout / Sag) |
| Thermistor / NTC Sensor | 10kΩ - 100kΩ @ 25°C | Smooth Thermal Curve | 0Ω (Short) or ∞ (Open) |
| Safety Ground / PE Continuity | < 0.5Ω Resistance | Solid Earth Bond | > 1.0Ω (Floating Chassis) |
Common Symptoms & Fault Manifestations
- NCCL execution aborts with 'NCCL WARN [send/recv] Call to connect returned Connection refused or timed out'
- Kernel dmesg logs report 'NVRM: Xid (PCI:0000:xx:00): 120, pid=..., SXid: NVLink link down'
- nvidia-smi nvlink -s shows one or more links in 'Inactive' or 'Down' state with high FLIT CRC error counts
Root Causes & System Failure Modes
Physical NVLink interconnect trace degradation or dirty high-density board-to-board mezzanine connector pads
Thermal throttling or localized power rail sag on NVSwitch chips dropping link sync beneath 900mV VDD core limits
PCIe fabric ASPM or OS-level GPU C-states triggering unwarranted NVLink low-power states (L1/L2 entry failures)
High-Voltage & Mechanical Safety Isolation
Prior to opening access panels, disconnecting terminals, or probing wiring, perform complete lock-out / tag-out (LOTO). Disconnect all AC mains, DC string isolators, and lithium battery banks. Allow internal smoothing capacitors at least 3 to 5 minutes to dissipate stored residual energy below safe threshold (<30VDC).
Step-by-Step Diagnostic & Resolution Procedure
Safe Isolation & Node Draining
Cordon the node in the cluster orchestrator (e.g., Slurm/Kubernetes). Terminate all CUDA workloads, initiate AC host shutdown via BMC/IPMI, and disconnect all redundant 54V/12V DC power shelf feeds to ensure complete power isolation.
NVLink Telemetry & Log Analysis
Before physical tear-down, query 'nvidia-smi nvlink -e -i [GPU_ID]' to isolate the exact link number, remote peer GUID, and uncorrectable error counts. Correlate with fabric logs via 'dcgmi diag -r 3' and NVSwitch telemetry to determine if the fault is ASIC-side or link-side.
Physical Mezzanine Inspection & Re-Seat
Disassemble the baseboard cold plate/heatsink assembly adhering to torque patterns. Inspect high-speed SEARAY/mezzanine connectors under magnification for bent pins, debris, or thermal flux migration. Clean contacts using 99.7% IPA, re-seat the affected H200 module, and re-torque screws progressively in cross-pattern to factory specification (1.0 Nm).
Firmware Verification & Collective Stress Test
Boot system to maintenance OS. Verify HGX baseboard CEC, NVSwitch, and GPU VBIOS match certified firmware baselines using NVFlash. Run the NCCL all-reduce stress test ('nccl-test/all_reduce_perf -b 8M -e 8G -f 2 -g 8') for 30 minutes while monitoring link stability.
Hardware Reset & Fault Clear Sequence
Controller ProcedurePrevention & Routine Maintenance Guidelines
- Enforce strict BMC-level inlet air temperature targets below 25°C to avoid thermal delta excursions on NVSwitch silicon.
- Disable PCIe and NVLink dynamic power management (NVLink L1/L2 states) in server BIOS and NVIDIA driver profiles.
Frequently Asked Diagnostic Questions
What does error code SXid 120 / NCCL Timeout indicate on NVIDIA HGX H200 8-GPU Baseboard?
An NVLink high-speed interconnect physical/logical link degraded or dropped unexpectedly, triggering SXid 120 (NVLink Link Down). Because point-to-point ring/tree communication halted mid-collective operation, the NCCL execution watchdog timer expired with a collective timeout. In engineering telemetry, it indicates nvlink link down timeout error requiring circuit verification.
What are the primary triggers causing error SXid 120 / NCCL Timeout?
Diagnostic bench telemetry isolates the following common root causes: Physical NVLink interconnect trace degradation or dirty high-density board-to-board mezzanine connector pads; Thermal throttling or localized power rail sag on NVSwitch chips dropping link sync beneath 900mV VDD core limits; PCIe fabric ASPM or OS-level GPU C-states triggering unwarranted NVLink low-power states (L1/L2 entry failures).
How do I reset and clear NVIDIA error SXid 120 / NCCL Timeout?
Perform a complete AC cold power cycle or execute secondary bus and fabric reset via IPMI using 'ipmitool raw 0x30 0xDB' (HGX baseboard reset) followed by 'nvidia-smi --gpu-reset'. If the error status repeats immediately after clearing, discontinue clearing and test circuit continuity.
Can I safely resolve error SXid 120 / NCCL Timeout without specialized shop tooling?
This fault is graded as Advanced difficulty with Critical severity. Standard diagnostics require an insulated digital multimeter and appropriate hand tools. Always observe high-voltage isolation procedures.