Troubleshooting
Pod Health Checks Drained /run Storage Space and Nearly Sent Me Packing
· ☕ 3 min read
If you use nvidia-container-runtime, this is worth your attention — especially if JuiceFS is also in the picture. 1. An Alert Out of Nowhere, and I Panicked Over the weekend I was studying TensorRT LLM, and while I was at it I installed Dragonfly on the largest production cluster. Then

From CPU to Network: A Record of Troubleshooting Application Slowness
· ☕ 11 min read
1. Symptoms The business side reported that the API of application app-a was slow. Looking at the logs, one particular Pod was slow, and deleting that Pod so it moved to another node fixed it. From the monitoring metrics you can see that the Pod’s CPU usage did indeed spike sharply.

The Scope of Credentials During Pipeline Builds
· ☕ 4 min read
In the client, we have already seen the Docker CLI passing credentials to the Docker Daemon via X-Registry-Config when sending the build context. But recent build feedback still shows some phenomena that cannot be explained. This post mainly runs some basic tests to make troubleshooting easier. 1. Building under host Docker The Docker Daemon is started with root privileges.

Troubleshooting Slow IO When Building Images
· ☕ 10 min read
1. The Problem Project details: File size 5.6 GB File count 529352 Dockerfile 1 2 3 FROM golang:1.13 COPY ./ /go/src/code The build command and its output are as follows: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 time DOCKER_BUILDKIT=1 docker build --no-cache -t test:v3 -f Dockerfile .

How to Repair a Kubernetes Master Node After Reinstalling the OS
· ☕ 2 min read
Recently I ran into this twice: a host had to be reinstalled because of a failure. In one case only a single Etcd node was left, and the whole cluster stayed down for half an hour before it recovered. This post mainly records the process of initializing a fresh Ubuntu