Troubleshooting
Common GPU Operations and Fault Handling
· β˜• 18 min read
Content referenced or recorded while handling faults, continuously updated. 1. XID Error Events XID is NVIDIA’s error code. You can retrieve it with the command: 1 dmesg -T | grep -i "NVRM: Xid" or 1 journalctl --since `date -d "10 days ago" "+%Y-%m-%d"`|grep Xid The XID lets you locate the fault.

Pod Health Checks Drained /run Storage Space and Nearly Sent Me Packing
· β˜• 3 min read
If you use nvidia-container-runtime, this is worth your attention β€” especially if JuiceFS is also in the picture. 1. An Alert Out of Nowhere, and I Panicked Over the weekend I was studying TensorRT LLM, and while I was at it I installed Dragonfly on the largest production cluster. Then

From CPU to Network: A Record of Troubleshooting Application Slowness
· β˜• 11 min read
1. Symptoms The business side reported that the API of application app-a was slow. Looking at the logs, one particular Pod was slow, and deleting that Pod so it moved to another node fixed it. From the monitoring metrics you can see that the Pod’s CPU usage did indeed spike sharply.

Common Problems When Using Alpine Images
· β˜• 1 min read
1. DNS Request Timeout Cause: alpine uses the musl library, which has some limitations in DNS resolution [1] Solution: Do not use an apline image, and add the options single-request-reopen setting to the container’s resolv.conf file. This is because the single-request-reopen option only takes effect for the glibc library, while apline images use the musl library.

The Scope of Credentials During Pipeline Builds
· β˜• 4 min read
In the client, we have already seen the Docker CLI passing credentials to the Docker Daemon via X-Registry-Config when sending the build context. But recent build feedback still shows some phenomena that cannot be explained. This post mainly runs some basic tests to make troubleshooting easier. 1. Building under host Docker The Docker Daemon is started with root privileges.