Common GPU Operations and Fault Handling

Content referenced or recorded while handling faults, continuously updated.

1. XID Error Events

XID is NVIDIA’s error code. You can retrieve it with the command:

1
dmesg -T | grep -i "NVRM: Xid"

1
journalctl --since `date -d "10 days ago" "+%Y-%m-%d"`|grep Xid

The XID lets you locate the fault. Below are some common XID events.

XID	Description
13	Graphics Engine Exception. Usually an array out-of-bounds or instruction error, rarely a hardware problem.
31	GPU memory page fault. Usually an illegal address access by the application, very rarely a driver or hardware problem.
43	GPU stopped processing. Usually an error in the user application itself rather than a hardware problem.
45	Preemptive cleanup, due to previous errors – Most likely to see when running multiple cuda applications and hitting a DBE. Usually the GPU application exits because the user quit manually or due to another fault (hardware, resource limits, etc.). Xid 45 is only a consequence, and log analysis is usually required.
68	NVDEC0 Exception. Usually a hardware or driver problem.
32	Invalid or corrupted push buffer stream. This event is reported by the DMA controller on the PCIE bus that manages communication between the NVIDIA driver and the GPU; usually caused by a PCI quality problem rather than by the user program.
38	Driver firmware error. Usually a driver firmware error rather than a hardware problem.
48	Double Bit ECC Error (DBE). When an uncorrectable error occurs on the GPU, an Xid48 event is reported. This error is also fed back to the user application at the same time. Usually the GPU needs to be reset or the node rebooted to clear this error.
61	Internal micro-controller breakpoint/warning. The GPU’s internal engine has stopped working, and customer workloads are already affected.
62	Internal micro-controller halt. Similar trigger scenario to Xid61.
63	ECC page retirement or row remapping recording event. When an application encounters a GPU memory hardware error, NVIDIA’s self-correction mechanism retires or remaps the faulty memory region; the retirement and remapped information must be recorded to infoROM to take effect permanently. Volt architecture: recording the ECC page retirement event to infoROM succeeded. Ampere architecture: recording the row remapping event to infoROM succeeded
64	ECC page retirement or row remapper recording failure. Similar trigger scenario to Xid63, except that Xid63 means the retirement and remapped information was successfully recorded to infoROM, while Xid64 means that recording operation failed.
74	NVLINK Error. An Xid produced by an NVLink hardware error; receiving this event means the GPU already has a serious hardware fault and needs to be taken offline for repair.
79	GPU has fallen off the bus. The GPU hardware is detected as having dropped off the card and can no longer be detected on the bus; receiving this event means the GPU already has a serious hardware fault and needs to be taken offline for repair.
92	High single-bit ECC error rate. Hardware or driver fault.
94	Contained ECC error. When an application encounters an uncorrectable GPU memory ECC error, NVIDIA’s error containment mechanism tries to contain the error to the application that hit the hardware fault, rather than letting the error affect all applications on the GPU. When the containment mechanism successfully contains the error, an Xid 94 event is generated, affecting only the application that encountered the uncorrectable ECC error.
95	Uncontained ECC error. Similar trigger scenario to Xid94, except that Xid94 means containment succeeded while Xid95 means containment failed, which indicates that all applications running on that GPU have been affected.

For details, see https://docs.nvidia.com/deploy/xid-errors/index.html

2. GPU Overheating

Normally the GPU temperature should be below 85°C; above that, clock throttling and performance degradation occur. Running the following command shows the GPU index and temperature directly.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
nvidia-smi --query-gpu=index,temperature.gpu --format=csv,noheader

0, 28
1, 40
2, 44
3, 30
4, 27
5, 27
6, 31
7, 32

Solution:

Apart from physical measures, from a purely software perspective you can simply kill the applications on the GPUs whose temperature exceeds the threshold, so that they move to other GPUs.

3. Card Loss After Reboot, nvswitch Errors

1
systemctl status nvidia-fabricmanager.service

1
less /var/log/fabricmanager.log | grep error

You see NVlink and NVSwitch errors.

Or nvidia-smi cannot find a device handle, with an Unknown Error.

Or there are fewer cards after a reboot.

Solution:

Enable nvidia-persistenced persistence mode so the driver stays loaded, which can greatly alleviate this problem.

4. nvidia-smi in a Pod Reports Function not Found

Running the command inside a Pod fails:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
nvidia-smi

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA TITAN X (Pascal)        On  | 00000000:03:00.0 Off |                  N/A |
| 23%   26C    P8               8W / 250W | Function Not Found   |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

This is because the CUDA version inside the Pod is too old and does not match the CUDA version on the node.

Solution:

Add the environment variable and restart the application.

1
LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/lib/x86_64-linux-gnu:/usr/local/nvidia/lib

5. GPU Memory Cannot Be Released

Run the command:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
nvidia-smi

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A800-SXM4-80GB          On  | 00000000:10:00.0 Off |                    0 |
| N/A   32C    P0              68W / 400W |  42058MiB / 81920MiB |      0%      Default |
|                                         |                      |             Disabled |
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+

You will see that no process is using the GPU, yet a large amount of GPU memory is still occupied.

At the same time you may notice a batch of unkillable zombie processes.

1
2
3
4
5
ps aux | grep -E '\<defunct\>'
root      966461  0.0  0.0   6432  2488 pts/0    S+   11:30   0:00 grep --color=auto -E \<defunct\>
root     2215172  0.0  0.0      0     0 ?        Zl   Apr04   1:10 [python] <defunct>
root     2215428  0.0  0.0      0     0 ?        Zl   Apr04   0:00 [python] <defunct>
root     2215442  0.0  0.0      0     0 ?        Zl   Apr04   0:00 [python] <defunct>

Solution:

Try restarting Kubelet, then Docker, then the host in turn; the GPU memory resources will be released.

6. Docker Hangs, Node NotReady

In Kubelet you see PLEG is not healthy related error logs. There are no abnormal logs in Docker.

runc may have hung, because the default pipe size is only 64 MB, which is not enough for high-performance computing scenarios.

Solution:

Set it to 1GB; here the setting is 262144 * 4K = 1GB.

1
echo "fs.pipe-user-pages-soft=262144" >> /etc/sysctl.conf && sysctl -p

7. df\ls Hangs, No Response

Inspect the hung process.

1
2
3
strace df -h

stat("/data/kubelet/pods/af4c411c-bafa-4322-9a21-e5c60ab1658e/volumes/kubernetes.io~nfs/workspace-pv",

Find the related mount point.

1
2
3
mount | grep mmt-10289-v2-1-4

1.1.1.1:/cfs-fSfmHNQjNA/workspace on /data/kubelet/pods/af4c411c-bafa-4322-9a21-e5c60ab1658e/volumes/kubernetes.io~nfs/workspace-pv type nfs (ro,relatime,vers=3,rsize=131072,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=x.x.x.x,mountvers=3,mountport=300,mountproto=tcp,local_lock=none,addr=x.x.x.x)

Confirm that the directory is no longer usable.

1
ls /data/kubelet/pods/af4c411c-bafa-4322-9a21-e5c60ab1658e/volumes/kubernetes.io~nfs/workspace-pv

At this point it should hang with no response. This is usually caused by the remote service 1.1.1.1 no longer being reachable while the mounted client was not cleaned up.

Force unmount the directory.

1
umount -f /data/kubelet/pods/af4c411c-bafa-4322-9a21-e5c60ab1658e/volumes/kubernetes.io~nfs/workspace-pv

8. df and du Data Differ Sharply

Checking with df shows / is 86% used.

1
2
3
4
df -h

Filesystem               Size  Used Avail Use% Mounted on
/dev/sda3                435G  352G   62G  86% /

But the usage counted by du is very small.

1
2
3
du -h --max-depth=2 --exclude=/data --exclude=/data1 --exclude=/var/lib/juicefs/volume / | grep '[0-9.]\+G'

32G /

The reason is that when large files were deleted, a program was still holding them open, so the kernel could not reclaim the space.

Solution:

Find the files held by programs.

1
lsof -n | grep deleted

Delete the files held by programs.

1
lsof -n | grep deleted | awk '{print $2}' | sort -u | xargs sudo kill -9

Another case is that the mount point became invalid, causing the mounted files to land on the host instead; in this case you need to unmount the mount point first and then delete the files under it.

9. Rebooting the Host with ipmitool

Rebooting the host with the reboot command only operates at the system level, and some hardware faults cannot be recovered.

1
ipmitool chassis power cycle

This is equivalent to a power-off reboot.

10. version mismatch

1
2
3
4
nvidia-smi

Failed to initialize NVML: Driver/library version mismatch
NVML library version: 535.129

1
2
3
4
5
modinfo nvidia | grep version

version:        535.183.01
srcversion:     E1D7E062E93D47A443165F6
vermagic:       5.4.0-1131-oracle SMP mod_unload modversions

Solution:

Uninstall the nvidia driver and reinstall it.

1
2
3
apt-get --purge remove "*cublas*" "cuda*"
apt-get --purge remove "*nvidia*"
apt autoremove

11. no free node

When using https://github.com/tkestack/gpu-manager to manage GPUs, note that the driver version must not be too high; for example, 535.183 is not compatible.

The compatible version we tested is 535.129.

With a higher-version driver, GPU memory cannot be reclaimed properly after allocation, so once the first allocation completes, new Pods fail to be created with a no free node error.

The root cause is that gpu-manager cannot obtain real-time GPU memory information.

If you hit this problem on a compatible version, try restarting kubelet to refresh the GPU memory information.

12. Enterprise JuiceFS Speed Fluctuates Sharply

After warm-up, access speed should be stable and fast, but if there are slow nodes in the P2P cache service node group, the access speed will drop in some places, dragging down the overall access speed.

You can enter the fluid worker pod.

1
2
3
ps aux | grep jfsmount

root         190 47.8  0.0 6160996 397776 ?      S<l  Sep13 673:59 /usr/local/juicefs/mount/jfsmount myjuicefs-storage /runtime-mnt/juicefs/mynamespace/myjuicefs-storage-fluid/juicefs-fuse -o cache-group=mynamespace-myjuicefs-storage-fluid,cache-size=5242880,free-space-ratio=0.1,cache-dir=/data1/jfs/cache:/data/jfs/cache,foreground,no-update

Inspect the connections between P2P cache service nodes.

1
lsof -p 190

Check for slow nodes that do not meet expectations; removing the relevant nodes restores performance.

13. resolv.conf: no such file or directory

A Pod fails to start with the error:

1
/run/systemd/resolve/resolv.conf: no such file or directory

Solution:

1
vim /var/lib/kubelet/kubeadm-flags.env

In KUBELET_KUBEADM_ARGS, change --resolv-conf=/etc/resolv.conf.

Then restart kubelet:

1
2
systemctl daemon-reload
systemctl restart kubelet

14. kubelet Fails to Start After Upgrading Node CPU

Error message

1
start cpu manager error: current set of available CPUs \"0-7\" doesn't match with CPUs in state \"0-3\"

Solution:

1
2
3
rm -f /var/lib/kubelet/cpu_manager_state
systemctl daemon-reload
systemctl restart kubelet

15. gpu-manager Fails to Start with can’t load container response data

Error message

1
can't load container response data, &json.SyntaxError{msg:"unexpected end of JSON input", Offset:0}

Sometimes it also reports an error like kubelet_internal_checkpoint not found.

Solution:

Try restarting Kubelet several times; if that still does not work, copy the /var/lib/kubelet/device-plugins/kubelet_internal_checkpoint file from another node to the current node.

The cause of the fault is a bug in GPU Manager that sometimes prevents the kubelet_internal_checkpoint file from being generated.

1
wget https://raw.githubusercontent.com/shaowenchen/ops-hub/refs/heads/master/gpu-manager/kubelet_internal_checkpoint  -O /var/lib/kubelet/device-plugins/kubelet_internal_checkpoint

16. cannot allocate unhealthy devices tencent.com/vcuda-core Prevents Pod Creation

gpu manager reports an error

1
E1114 03:01:12.308213   33781 server.go:133] Unable to set Type=notify in systemd service file?

Solution:

1
systemctl restart kubelet

17. DCGM Reports Failed to watch metrics

Error message

1
Failed to watch metrics: Error watching fields: Host engine is running as non-root

Solution:

Edit the dcgm-exporter DaemonSet and confirm the following Pod configuration is correct.

1
2
3
4
5
6
securityContext:
  capabilities:
    add:
      - SYS_ADMIN
  runAsNonRoot: false
  runAsUser: 0

In particular, for the capabilities configuration here, make sure SYS_ADMIN is granted; runAsUser: 0 alone is not enough.

18. PCI Cannot Detect the GPU

Run the command:

1
echo 1 > /sys/bus/pci/rescan

This attempts to re-detect the PCI device, but it does not necessarily solve the problem, since it may be a hardware issue.

19. rpc client exit with 255 When Allocating Less Than a Whole Card

Error message

1
/tmp/cuda-control/src/register.c:87 rpc client exit with 255

Solution:

Add the environment variable

1
LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/lib/x86_64-linux-gnu:/usr/local/nvidia/lib

The reason is that CUDA does not support allocating less than a whole card by default, and a GPU slicing component is required. If you are using GPU Manager, setting the above environment variable hijacks the allocation request.

20. Draining to Disable a Specific Faulty Card

View the PCI location of the card

1
2
3
4
nvidia-smi --query-gpu=index,pci.bus_id --format=csv

index, pci.bus_id
0, 00000000:18:00.0

The first column is the card index, the second is the card’s PCI location.

Disable the specified GPU card

1
nvidia-smi drain -p 0000:18:00.0 -m 1

-m 1 means drain state, -m 0 means turn off drain state. After the command completes, the card is invisible in nvidia-smi but visible in lspci.

View the drain state of the card

1
nvidia-smi drain -p 0000:18:00.0 -q

Undrain all cards

1
2
3
4
5
6
PCI_LIST=$(lspci | grep -i vga | awk '{print $1}')
for pci in $PCI_LIST; do
    full="0000:$pci"
    nvidia-smi drain -p "$full" -q || echo "Failed to undrain $full"
done
nvidia-smi

21. NCCL Cannot allocate memory

Error message, GPU memory cannot be allocated

misc/ibvwrap.cc:262 NCCL WARN Call to ibv_reg_mr failed with error Cannot allocate memory

Solution:

Add the IPC_LOCK capability to the Pod.

1
2
3
4
securityContext:
  capabilities:
    add:
      - IPC_LOCK

22. Slow DNS or Node Network Unreachable

Caused by NIC hardware checksum

The symptom is slow DNS resolution or an unreachable node network. Packet capture shows errors like the following:

1
2
3
4
5
6
7
tcpdump -i bond1  host x.x.x.x  -vvv

13:42:16.465143 IP (tos 0xc0, ttl 64, id 50859, offset 0, flags [DF], proto TCP (6), length 71)
    x.x.x.x.bgp > x.x.x.x57973: Flags [P.], cksum 0x7630 (correct), seq 590:609, ack 608, win 501, options [nop,nop,TS val 707951074 ecr 2196294139], length 19: BGP
        Keepalive Message (4), length: 19
13:42:16.465193 IP (tos 0xc0, ttl 64, id 62942, offset 0, flags [DF], proto TCP (6), length 52)
    x.x.x.x.57973 > x.x.x.x.bgp: Flags [.], cksum 0xe0be (incorrect -> 0x6c4f), seq 608, ack 609, win 501, options [nop,nop,TS val 2196297719 ecr 707951074], length 0

cksum 0xe0be (incorrect -> 0x6c4f) indicates a checksum error, which causes the network problem.

The solution is to turn off TSO and TX.

1
ethtool -K bond1 tso off tx off gso off gro off rx off

The cause may be related to the NIC not supporting checksum computation for some type of packet in the upper-layer Overlay network.

Moreover, so far this has only been encountered in mellanox environments; you can also try upgrading the relevant firmware version.

You can also try systemctl restart networking.service to restart the network service.

Network component anomalies caused by multiple NICs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
# Inspect NICs
ifconfig -a
# Disable a NIC
ip link set eth0x down
# Remove the NIC IP
ip addr flush dev eth0x
# Restart the network component
# ...
# Bring the NIC up
ip link set eth0x up
# Add the NIC IP
ip addr add 10.0.0.10/26 dev eth0x

23. MPI No such file or directory

Set the environment variables

1
2
export LD_LIBRARY_PATH=/usr/local/mpi/lib:LD_LIBRARY_PATH
export OPAL_PREFIX=/opt/hpcx/ompi/

24. CUDA Error 802

1
ERROR 01-17 04:52:28 engine.py:366] RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 802: system not yet initialized

Encountered on A100 and H20; the nvidia-fabricmanager service was not installed, so initialization failed. After installing it, if the error persists, try rebooting the machine.

25. IB NIC Down

1
2
3
4
5
6
7
ibdev2netdev

mlx5_0 port 1 ==> ibs10 (Up)
mlx5_1 port 1 ==> ibs11 (Down)
mlx5_4 port 1 ==> ibs18 (Up)
mlx5_5 port 1 ==> ibs19 (Up)
mlx5_bond_0 port 1 ==> bond1 (Up)

Solution:

Check the logs

1
2
3
cat /var/log/syslog |grep ibs11

ibs11: Lost carrier

This situation is usually a hardware problem, and the network cable needs to be checked.

Of course, you can also try bringing the NIC up directly.

1
ip link set ibs11 up

26. smi Hangs When Installing the GPU Driver

There are several possibilities:

The installed driver version does not meet the graphics card’s minimum requirement
The installed driver version does not match the current card model
The system kernel is incompatible with the driver, and the kernel needs to be upgraded

27. Disk Expansion

Inference nodes do not need an extra data disk; a 2T system disk can be used directly. If you did not start with a large system disk, you need to expand the system disk in the cloud vendor’s console.

Then run the following commands to expand the system disk.

View disk information

1
2
3
4
5
6
7
lsblk -f
NAME FSTYPE FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
vda
├─vda1
│    vfat   FAT32       0E90-C51D                             190.8M     3% /boot/efi
└─vda2
     ext4   1.0         6ee63166-f2b3-4cff-9234-2a911c9c994d   74.4G    20% /var/lib/kubelet/pods/a8e3dd18-e66b-45ac-8f03-937ca9ce08d2/volume-subpaths/config/filebeat/0

Extend the partition size

1
growpart system-disk partition-number

For example, growpart /dev/vda 2

Extend the filesystem size

If the partition is ext4, use the resize2fs command.

1
resize2fs disk-partition

For example, resize2fs /dev/vda2

If the partition is xfs, use the xfs_growfs command.

1
xfs_growfs disk-partition

For example, xfs_growfs /dev/vda2

28. Fabric Manager Operation not permitted

The common problem with Frabric Manager is a driver version mismatch. But the following error has also been encountered.

1
Aug 14 14:16:06 node-a800-gc-31 systemd[1]: nvidia-fabricmanager.service: Can't open PID file /run/nvidia-fabricmanager/nv-fabricmanager.pid (yet?) after start: Operation not permitted

This appeared after upgrading the GPU driver and Fabric Manager versions. According to the provider, the cause is different modules, and some machines cannot use a high-version driver, so it had to be downgraded to version 535.

29. nvidia-smi Hangs

After running some tasks involving multi-GPU communication, zombie processes appear that cannot be released.

1
2
3
4
journalctl -p 3 | grep -i "nvidia"

Sep 03 06:03:17 bj6-e-ai-kas-node-a800-gc-29 kernel: nvidia-peermem nv_get_p2p_free_callback:125 ERROR detected invalid context, skipping further processing
Sep 03 06:03:17 bj6-e-ai-kas-node-a800-gc-29 kernel: nvidia-peermem nv_get_p2p_free_callback:125 ERROR detected invalid context, skipping further processing

Upgrading the driver version may solve it, but on some devices fabric fails to start after the driver upgrade, so the device can only be rebooted first.

30. Clearing ECC Counters

Cumulative ECC error count

1
for i in {0..7}; do nvidia-smi -i $i -p 1; done

Volatile ECC error count

1
for i in {0..7}; do nvidia-smi -i $i -p 0; done

If the clearing process reports errors, you can stop all processes occupying the card and reset the card.

1
2
3
systemctl stop nvidia-persistenced.service nvidia-fabricmanager.service
nvidia-smi -r
systemctl start nvidia-persistenced.service nvidia-fabricmanager.service

31. Accessing a Path Overlaid by a Mounted Disk

A mounted disk overlays the system data path, making the previous system data path unusable.

1
2
mkdir -p /mnt/rootfs
mount --bind / /mnt/rootfs

If the data disk is mounted under the /data directory, then /mnt/rootfs/data is the original system data path.

32. Some Pods on CPU Nodes Fail to Start

GPU clusters usually also include some CPU nodes. On CPU nodes, some Pods fail to start with the following error:

1
2
Error: failed to create containerd task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1,
stdout: , stderr: Auto-detected mode as 'legacy' nvidia-container-cli: initialization error: nvml error: driver not loaded: unknown

The cause is that the CPU nodes use the GPU nodes’ system image, and the Containerd configuration file contains GPU-related configuration. This error is triggered when using the Transformers library in a Pod.

33. Timeout from GSP

The error Timeout waiting for RPC from GSP! appears, possibly because GSP (GPU Shared Protocol) failed to initialize correctly; you can try disabling GSP.

GSP’s purpose is to move the GPU management logic from the CPU kernel driver into the GPU itself.

1
echo options nvidia NVreg_EnableGpuFirmware=0 > /etc/modprobe.d/nvidia-gsp.conf

After rebooting the machine, check whether it took effect.

1
cat /proc/driver/nvidia/params | grep EnableGpuFirmware:

If the output is 0, GSP has been disabled.

34. RDMA Cannot allocate memory

1
E20260611 15:24:10.244277    12 rdma_context.cpp:131] Failed to create completion queue: Cannot allocate memory [12]

Solution: adjust LimitMEMLOCK.

1
vim /lib/systemd/system/containerd.service

Or use sed

1
grep -q '^LimitMEMLOCK=infinity$' /lib/systemd/system/containerd.service || sed -i '/^LimitNPROC=infinity$/i LimitMEMLOCK=infinity' /lib/systemd/system/containerd.service

Add the following alongside the other Limit settings:

1
2
LimitMEMLOCK=infinity #newly added
LimitNPROC=infinity

Restart containerd

1
2
systemctl daemon-reload
systemctl restart containerd

35. unknown CDI devices

Error message, CDI devices cannot be injected

1
Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: could not apply required modification to OCI specification: error modifying OCI spec: failed to inject CDI devices: unresolvable CDI devices runtime.nvidia.com/gpu=all: unknown

A GPU image is running on a CPU node configured with the NVIDIA Runtime; the containerd configuration file needs to be regenerated and changed to runc.

36. doesn’t have valid Grid license

Error

1
RuntimeError: CUDA error: device doesn't have valid Grid license

If the machine was rebooted recently, check the status of nvidia-gridd; registration may have failed at startup.

1
nvidia-smi -q | grep -A5 License

A reboot usually fixes it.

1
systemctl restart nvidia-gridd.service

37. Node Driver Suddenly Reports version mismatch

1
2
Failed to initialize NVML: Driver/library version mismatch
NVML library version: 580.173

Check the update history of the driver package

1
awk '/Start-Date:/{flag=1;line=$0} flag; /End-Date:/{flag=0}' /var/log/apt/history.log | grep -B10 -A10 "nvidia-kernel-common"

1
2
Commandline: /usr/bin/unattended-upgrade
Install: nvidia-kernel-common-580:amd64 (580.173.02-0ubuntu0.22.04.1, automatic), nvidia-firmware-580-580.173.02:amd64 (580.173.02-0ubuntu0.22.04.1, automatic), libnvidia-compute-580:amd64 (580.173.02-0ubuntu0.22.04.1, automatic)

Disable automatic updates to prevent another automatic upgrade next time

1
2
systemctl stop unattended-upgrades
systemctl disable unattended-upgrades

Hot fix

1
apt remove -y nvidia-kernel-common-580 libnvidia-compute-580 nvidia-firmware-580-580.173.02