NVIDIA DGX H100 Error SXid 120: NVSwitch Fatal Credit Timeout
SXid 120 indicates an unrecoverable credit timeout on an NVSwitch NVLink port, meaning an internal receiver buffer did not replenish transmission credits within the hardware-enforced threshold, stalling crossbar fabric traffic and forcing an NCCL communication abort.
Hardware Diagnostic Matrix
Spec Ref #9ee15ed9Required Diagnostic Tools & Instrumentation
Multimeter Probing Targets & Circuit Tolerances
Bench Calibrated ±5%| Test Point / Circuit | Target Nominal Spec | Expected State | Fault Condition |
|---|---|---|---|
| Main Logic DC Bus | 3.3V / 5.0V DC (±0.1V) | Stable DC Potential | <2.9V (Brownout / Sag) |
| Thermistor / NTC Sensor | 10kΩ - 100kΩ @ 25°C | Smooth Thermal Curve | 0Ω (Short) or ∞ (Open) |
| Safety Ground / PE Continuity | < 0.5Ω Resistance | Solid Earth Bond | > 1.0Ω (Floating Chassis) |
Common Symptoms & Fault Manifestations
- NCCL distributed training jobs terminate with Watchdog or 'Remote process exited' fatal errors
- Host kernel dmesg logs repeated 'nvidia-nvswitch[PID]: SXid (Fabric): 120, Fatal NVLink credit timeout'
- One or more NVLink interfaces drop to an inactive/down state in 'nvidia-smi nvlink -s' or 'nvsm show health'
Root Causes & System Failure Modes
Hardware fabric stall due to degraded NVLink high-speed differential pairs or physical baseboard socket pin degradation
GPU engine hung mid-kernel execution (XID 31/45 cascade) failing to drain incoming crossbar ingress buffers
Microcode desynchronization or firmware fault across the four third-generation NVSwitch ASICs
Under-voltage or power-rail noise transients on the 0.8V/1.2V core VDD rails supplying the NVSwitch subsystem
High-Voltage & Mechanical Safety Isolation
Prior to opening access panels, disconnecting terminals, or probing wiring, perform complete lock-out / tag-out (LOTO). Disconnect all AC mains, DC string isolators, and lithium battery banks. Allow internal smoothing capacitors at least 3 to 5 minutes to dissipate stored residual energy below safe threshold (<30VDC).
Step-by-Step Diagnostic & Resolution Procedure
Quarantine Node & Isolate Active Workloads
Drain the node from the orchestration cluster (e.g., Slurm/Kubernetes). Terminate all zombie training processes using 'pkill -9 -f python' and disable persistence mode before diagnostics.
Query Fabric Health & Pinpoint Target Link
Execute 'nvsm show health' and inspect 'dmesg -T | grep -E "SXid|NVSwitch"' to identify the specific switch ASIC physical ID (Switch 0-3) and port number experiencing credit depletion.
Full Subsystem Power-Cycle & Microcode Reload
A standard OS warm reboot cannot clear locked NVSwitch crossbar registers. Perform a complete chassis AC power-cycle via the BMC out-of-band IPMI interface: 'ipmitool chassis power cycle' or unseat/re-seat main input power after cold shutdown.
Run DCGM Diagnostic Stress & NCCL Validation
After boot, verify all 32 SXM5 NVLinks per switch are trained using 'nvidia-smi nvlink -s'. Execute 'dcgmproftester12 --no-dcgm-validation -t 1004 -d 300' followed by the standard 'all_reduce_perf' benchmark to confirm fabric throughput without credit stalls.
Hardware Reset & Fault Clear Sequence
Controller ProcedurePrevention & Routine Maintenance Guidelines
- Keep NVIDIA Fabric Manager and GPU driver strictly matched to the recommended DGX OS software stack version
- Maintain uniform chassis thermal profiles; target ambient inlet temperatures below 25°C to avoid silicon thermal throttling on NVSwitches
Frequently Asked Diagnostic Questions
What does error code SXid 120 indicate on NVIDIA DGX H100?
SXid 120 indicates an unrecoverable credit timeout on an NVSwitch NVLink port, meaning an internal receiver buffer did not replenish transmission credits within the hardware-enforced threshold, stalling crossbar fabric traffic and forcing an NCCL communication abort. In engineering telemetry, it indicates nvswitch fatal credit timeout requiring circuit verification.
What are the primary triggers causing error SXid 120?
Diagnostic bench telemetry isolates the following common root causes: Hardware fabric stall due to degraded NVLink high-speed differential pairs or physical baseboard socket pin degradation; GPU engine hung mid-kernel execution (XID 31/45 cascade) failing to drain incoming crossbar ingress buffers; Microcode desynchronization or firmware fault across the four third-generation NVSwitch ASICs; Under-voltage or power-rail noise transients on the 0.8V/1.2V core VDD rails supplying the NVSwitch subsystem.
How do I reset and clear NVIDIA error SXid 120?
Initiate an out-of-band BMC power-cycle: run 'ipmitool chassis power cycle', wait 300 seconds for baseboard and NVSwitch CPLD state machine initialization, and restart Fabric Manager: 'systemctl restart nvidia-fabricmanager'. If the error status repeats immediately after clearing, discontinue clearing and test circuit continuity.
Can I safely resolve error SXid 120 without specialized shop tooling?
This fault is graded as Advanced difficulty with Critical severity. Standard diagnostics require an insulated digital multimeter and appropriate hand tools. Always observe high-voltage isolation procedures.