An NVLink high-speed interconnect physical/logical link degraded or dropped unexpectedly, triggering SXid 120 (NVLink Link Down). Because point-to-point ring/tree communication halted mid-collective operation, the NCCL execution watchdog timer expired with a collective timeout.

⏱ 45-90 Mins Troubleshoot →

CODE XID 31 H100 SXM5 80GB Critical

H100 SXM5 XID 31 Fatal Memory Error

An uncorrectable multi-bit ECC error occurred in the High Bandwidth Memory (HBM3) stack, triggering NVIDIA driver event XID 31. The GPU driver attempted dynamic memory page retirement to isolate the failing physical address, but the retirement sequence failed or the bad page table exhausted its capacity, causing the GPU to fall off the bus and trigger a fatal kernel freeze.

⏱ 30-60 Mins Troubleshoot →

CODE SXid 63 HGX H100 8-GPU System Critical

NVSwitch Routing Credit Timeout

SXid 63 indicates a fatal port routing credit starvation timeout on the NVSwitch crossbar fabric. An NVLink endpoint failed to return flow-control credits within the hardware watchdog window, deadlocking fabric routing and triggering kernel-level GPU isolation.

⏱ 45-90 Mins Troubleshoot →

CODE SXid 63 HGX H100 SXM5 Tensor Core GPU Critical

H100 SXid 63 NVLink Error

SXid 63 designates an uncorrectable fatal hardware error on one or more NVLink4 interconnect links, typically caused by link retrain exhaustion, physical transceiver/pad degradation, or socket cold-solder/interposer micro-fractures, halting distributed NCCL training runs.

⏱ 45-90 Mins Troubleshoot →

CODE XID 119 HGX H200 8-GPU Board Critical

GSP-RM RPC Timeout Failure

An XID 119 event indicates an RPC communication failure between the host kernel driver and the GPU System Processor (GSP) firmware on one or more Hopper SXM5 modules, leading to NVLink interconnect desynchronization, NVSwitch fabric stalls, and dropped NCCL collective communication rings.

⏱ 45-90 Mins Troubleshoot →

CODE SXid 63 / NCCL 12 HGX H100 8-GPU Baseboard Critical

HGX H100 NVSwitch Link Failure

SXid 63 indicates an uncorrectable NVSwitch NVLink SerDes receiver error, causing dropped inter-GPU packets and leading to NCCL distributed training collective timeouts (Error 12).

⏱ 45-90 Mins Troubleshoot →

CODE 12V-2x6-SENSE-FAULT GeForce RTX 4090 Critical

RTX 4090 Black Screen 100% Fan

The GPU initiates an immediate safety latch-down due to an open circuit or high resistance detected on the 12VHPWR/12V-2x6 sideband sense pins (SENSE0/SENSE1). When sense continuity is lost under load, the onboard microcontroller/power management IC trips hardware protection, cutting core power to prevent thermal runaway while driving cooling fans to 100% tachometer duty cycle via emergency PWM pull-down.

⏱ 20-30 Mins Troubleshoot →

CODE SXid 62 HGX H100 8-GPU Board Critical

NVSwitch Port Link Training Abort

SXid 62 designates an NVLink physical layer (PHY) communication breakdown where high Cyclic Redundancy Check (CRC) error rates prevent link training or cause an active link to abort during link negotiation between a Hopper GPU and an NVSwitch.

⏱ 60-120 Mins Troubleshoot →

CODE Xid 79 H100 SXM5 80GB Critical

GPU Fallen Off The Bus

NVRM Xid 79 indicates an unrecoverable PCIe communication loss between host/PCIe switch and the H100 SXM5 GPU module. Root-cause mechanisms include physical baseboard socket unseating, PCIe Advanced Error Reporting (AER) fatal receiver/replay timeouts, transient 12V/48V rail power sag, or thermal throttle lockout.

⏱ 45-90 Mins Troubleshoot →

CODE CUDA error 719 / NVLink Fatal Error HGX H100 8-GPU Baseboard Critical

NVLink Transmission Failure Error 719

CUDA error 719 (cudaErrorLaunchFailure) manifests during multi-node all-reduce training when the NCCL Watchdog timer trips due to hardware-level uncorrectable transmission errors on fourth-generation NVLink interfaces, halting GPU-to-GPU memory fabric transactions.

⏱ 45-90 Mins Troubleshoot →

CODE XID 79 HGX H200 SXM5 Critical

H200 GPU Fallen Off Bus

NVIDIA driver logged XID 79 indicating the GPU has fallen off the PCIe/NVLink bus. This is triggered when an uncorrectable double-bit ECC memory error (DBE) or link-layer parity collapse isolates the HBM3e subsystem, causing the ASIC microcontroller to halt execution and sever host MMIO communication.

⏱ 45-90 Mins Troubleshoot →

CODE SXid 119 HGX H100 8-GPU Board Critical

NVSwitch Fabric Link Down Uncorrectable

An uncorrectable physical layer or data link layer error occurred on the NVLink 4.0 interconnect between an H100 Tensor Core GPU and an on-board NVSwitch 3 ASIC, triggering a fatal link-down event and hardware isolation.

⏱ 45-90 Mins Troubleshoot →

CODE 12V-2x6-SENSE-TRIP GeForce RTX 4080 Super Critical

12V-2x6 Sense Line Trip

The GPU initiates an emergency hardware failsafe due to an open-circuit or high-resistance state on the 12V-2x6 sideband Sense pins (Sense0/Sense1). When the onboard PWM buck controller loses valid ground reference across the sense lines, it detects an unsafe power delivery state, immediately cuts GPU core/memory rail power (Vcore/Vmem), drops display output, and forces fans to 100% duty cycle via default tachometer/PWM failsafe.

⏱ 20-30 Mins Troubleshoot →

CODE SXid 120 HGX H100 8-GPU Baseboard Critical

NVSwitch Link Integrity NCCL Timeout

SXid 120 denotes a critical physical layer (PHY) or data link layer integrity failure on an NVLink connection between an H100 GPU and a 3rd-generation NVSwitch ASIC. When excessive symbol errors, flit CRC corruption, or lane degradation exceed threshold tolerances, the fabric drops packets, resulting in an unrecoverable NCCL all-reduce kernel timeout.

⏱ 45-90 Mins Troubleshoot →

CODE SXid 63 HGX H100 SXM5 Critical

NVLink Uncorrectable Error SXid 63

NVIDIA driver SXid 63 reports an uncorrectable fatal error on the NVLink interconnect fabric. A port on the H100 SXM5 GPU or interconnected NVSwitch encountered physical layer packet degradation, link integrity breakdown, or bad cyclic redundancy checks (CRC) exceeding recovery thresholds.

⏱ 45-90 Mins Troubleshoot →

CODE XID 79 HGX H200 SXM5 8-GPU Subsystem Critical

GPU PCIe Fatal Dropped Bus

An unrecoverable PCIe fatal error occurred on the SXM5 interconnect fabric or upstream PCIe Gen5 switch/retimer link, triggering an XID 79 assertion where the GPU dropped entirely off the PCIe bus during heavy NCCL ring all-reduce communication.

⏱ 45-90 Mins Troubleshoot →

CODE SXid 120 DGX H100 Critical

NVSwitch Fatal Credit Timeout

SXid 120 indicates an unrecoverable credit timeout on an NVSwitch NVLink port, meaning an internal receiver buffer did not replenish transmission credits within the hardware-enforced threshold, stalling crossbar fabric traffic and forcing an NCCL communication abort.

⏱ 45-90 Mins Troubleshoot →