If you use nvidia-container-runtime, this is worth your attention — especially if JuiceFS is also in the picture.
1. An Alert Out of Nowhere, and I Panicked
Over the weekend I was studying TensorRT LLM, and while I was at it I installed Dragonfly on the largest production cluster. Then I went out to buy groceries.
In the afternoon I noticed that the Dragonfly daemon on one node had not started and was alerting continuously, so I went to that node and restarted the Kubelet.
About 10 minutes later, production alerts started coming in.
1
2
3
4
5
6
7
8
| 日志分析告警
时间：2024-01-20 15:26:25
标题：过去20s 状态码503数量超过阈值 547 >= 3
详情：
前三失败接口:
path: /api/xxx/v2/models/xxx/versions/1/infer 数量: 237
path: /api/xxx/v2/models/xxx/versions/1/infer 数量: 188
path: /api/xxx/v2/models/xxx/versions/1/infer 数量: 122
|
And the alerts kept coming. I had not been in the AI department for long, had not even trained a model yet, and then this happened. I had not done anything, hmph!
Still, solve the problem first. I took a look at the Kubelet log.
1
| kubelet[4031671]: E0120 15:31:57.357711 4031671 remote_runtime.go:209] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to start sandbox container for pod \"dragonfly-dfdaemon-wznfl\": Error response from daemon: write /run/containerd/io.containerd.runtime.v2.task/moby/2b6a70e32f77707d88b73d28054bb83aed34d9ac90c0993df9d1209bd5402b84/config.json: no space left on device: unknown"
|
Check the disk situation.
1
2
3
| df -h
tmpfs 51G 51G 0 100% /run
|
Found that /run was out of space, so I looked for large files under the /run/containerd/io.containerd.runtime.v2.task/moby directory.
1
2
3
4
5
| 6.3G /run/containerd/io.containerd.runtime.v2.task/moby/6be974f9293ab553bf86f0eda38b7813f315a98e63895eddaccb6b290ef6a1ac/log.json
6.3G /run/containerd/io.containerd.runtime.v2.task/moby/811c9a0aabb50f5ca73e6ee529f41745c2e18568a160f42314caaba142562c6b/log.json
7.9G /run/containerd/io.containerd.runtime.v2.task/moby/398ecf3c1488b4f8ec0f0ad12ac0a1080355fbd8102f6ed21980c7d7637ec7d2/log.json
8.1G /run/containerd/io.containerd.runtime.v2.task/moby/ab7253b7bbd05c8fe017008de2ec4494b4c11f2d55b980927d2fdcb3b306c924/log.json
9.6G /run/containerd/io.containerd.runtime.v2.task/moby/6030f2ad8532162cfa0effb479a9cd3f31c894c2152dfe34ddc59244b53f6241/log.json
|
Case solved. I emptied the contents of these files directly, and the service recovered immediately.
2. Reproducing the log.json Growth
2.1 Inspecting log.json and Tracing Its Source
First, look at the log contents.
1
| {"level":"info","msg":"Running with config:\n{\n \"AcceptEnvvarUnprivileged\": true,\n \"NVIDIAContainerCLIConfig\": {\n \"Root\": \"\"\n },\n \"NVIDIACTKConfig\": {\n \"Path\": \"nvidia-ctk\"\n },\n \"NVIDIAContainerRuntimeConfig\": {\n \"DebugFilePath\": \"/dev/null\",\n \"LogLevel\": \"info\",\n \"Runtimes\": [\n \"docker-runc\",\n \"runc\"\n ],\n \"Mode\": \"auto\",\n \"Modes\": {\n \"CSV\": {\n \"MountSpecPath\": \"/etc/nvidia-container-runtime/host-files-for-container.d\"\n },\n \"CDI\": {\n \"SpecDirs\": null,\n \"DefaultKind\": \"nvidia.com/gpu\",\n \"AnnotationPrefixes\": [\n \"cdi.k8s.io/\"\n ]\n }\n }\n },\n \"NVIDIAContainerRuntimeHookConfig\": {\n \"Path\": \"/usr/bin/nvidia-container-runtime-hook\",\n \"SkipModeDetection\": false\n }\n}","time":"2024-01-23T09:43:36+08:00"}
|
It does not look like an application log, and in fact it is not — it is a runc log, or more precisely a log from nvidia-container-runtime.
The info log level here should be changeable in nvidia-container-runtime’s config file; simply removing info-level logging is enough.
But does this problem exist only with nvidia-container-runtime? No — this is a general problem that has been overlooked.
2.2 Building a Test Workload
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
| cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: demo-whomai
spec:
replicas: 1
selector:
matchLabels:
app: demo-whomai
template:
metadata:
labels:
app: demo-whomai
spec:
containers:
- name: whomai
image: shaowenchen/demo:whomai
readinessProbe:
exec:
command:
- sh
- -c
- '[ -e /random/ ]'
initialDelaySeconds: 1
periodSeconds: 1
EOF
|
After repeated testing, I came up with the workload above. You only need to pay attention to two things to trigger continuous growth of the log.json file.
- Configure a probe
- The probe command must fail to execute
Running sh in the shaowenchen/demo:whomai image returns an error. Probing once per second, a log.json file accumulated over two months can reach several GB.
2.3 Creating the Workload to Test
Create a workload from the YAML above.
1
2
3
4
| kubectl get pod -l app=demo-whomai
NAME READY STATUS RESTARTS AGE
demo-whomai-966dd7875-jvvzr 0/1 Running 0 48m
|
Because the health check fails, the Pod never reaches Ready.
1
| CONTAINER_ID=$(kubectl get pod -l app=demo-whomai -ojson | jq -r '.items[0].status.containerStatuses[0].containerID | sub("docker://"; "")')
|
- Inspect the container’s log.json file
With Docker this should be under the moby namespace,
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
| ls -alh /run/containerd/io.containerd.runtime.v2.task/moby/$CONTAINER_ID
total 1.4M
drwx------ 3 root root 240 Jan 23 10:40 .
drwx--x--x 162 root root 3.2K Jan 23 11:29 ..
-rw-rw-rw- 1 root root 89 Jan 23 10:40 address
-rw-r--r-- 1 root root 9.5K Jan 23 10:40 config.json
-rw-r--r-- 1 root root 7 Jan 23 10:40 init.pid
prwx------ 1 root root 0 Jan 23 10:40 log
-rw-r--r-- 1 root root 1.4M Jan 23 11:31 log.json
-rw------- 1 root root 23 Jan 23 10:40 options.json
drwxr-xr-x 1 root root 4.0K Jan 23 10:40 rootfs
-rw------- 1 root root 0 Jan 23 10:40 runtime
-rw------- 1 root root 32 Jan 23 10:40 shim-binary-path
lrwxrwxrwx 1 root root 118 Jan 23 10:40 work -> /data/containerd/io.containerd.runtime.v2.task/k8s.io/240ccf68446af4a761273e5db08f6aebc362715f64d51efea785c26900c569c5
|
With Containerd this should be under the k8s.io namespace, ls -alh /run/containerd/io.containerd.runtime.v2.task/k8s.io/$CONTAINER_ID
- Inspect the contents of the container’s log.json file
With Docker this should be under the moby namespace,
1
2
3
4
5
| cat /run/containerd/io.containerd.runtime.v2.task/moby/$CONTAINER_ID/log.json
{"level":"error","msg":"exec failed: unable to start container process: exec: \"sh\": executable file not found in $PATH","time":"2024-01-23T11:32:01+08:00"}
{"level":"error","msg":"exec failed: unable to start container process: exec: \"sh\": executable file not found in $PATH","time":"2024-01-23T11:32:01+08:00"}
{"level":"error","msg":"exec failed: unable to start container process: exec: \"sh\": executable file not found in $PATH","time":"2024-01-23T11:32:01+08:00"}
|
With Containerd this should be under the k8s.io namespace, cat /run/containerd/io.containerd.runtime.v2.task/k8s.io/$CONTAINER_ID/log.json
3. Solutions
3.1 Clean Up Directly
1
| curl -sfL https://raw.githubusercontent.com/shaowenchen/ops/main/getcli.sh |VERSION=latest sh -
|
If it is already installed, run opscli upgrade to update it.
- Find log.json files larger than 100M
Command for Docker:
1
| opscli task -f ~/.ops/tasks/clear-biglog.yaml --logpath /run/containerd/io.containerd.runtime.v2.task/moby/ --logname "log.json" --size 100M -i ~/.kube/config --all
|
Command for Containerd:
1
| opscli task -f ~/.ops/tasks/clear-biglog.yaml --logpath /run/containerd/io.containerd.runtime.v2.task/k8s.io/ --logname "log.json" --size 100M -i ~/.kube/config --all
|
- Clean up log.json files larger than 100M
Add the --clear flag on top of the inspection command, and it will directly clean up any log.json file larger than 100M.
3.2 Change the Log Level of nvidia-container-runtime
Edit the nvidia-container-runtime config file.
1
2
3
4
| vim /etc/nvidia-container-runtime/config.toml
[nvidia-container-runtime]
log-level = "info"
|
Changing log-level = "info" to log-level = "error" avoids emitting logs like {"level":"info","msg":"Running with config:\n{\n \"AcceptEnvvarUnprivileged\": true,\n \"NVIDIAContainerCLIConfig\": {\n \"Root\": \"\"\n },\n \"NVIDIACTKConfig\": {\n \"Path\": \"nvidia-ctk\"\n },\n \"NVIDIAContainerRuntimeConfig\": {\n \"DebugFilePath\": \"/dev/null\",\n \"LogLevel\": \"info\",\n \"Runtimes\": [\n \"docker-runc\",\n \"runc\"\n ],\n \"Mode\": \"auto\",\n \"Modes\": {\n \"CSV\": {\n \"MountSpecPath\": \"/etc/nvidia-container-runtime/host-files-for-container.d\"\n },\n \"CDI\": {\n \"SpecDirs\": null,\n \"DefaultKind\": \"nvidia.com/gpu\",\n \"AnnotationPrefixes\": [\n \"cdi.k8s.io/\"\n ]\n }\n }\n },\n \"NVIDIAContainerRuntimeHookConfig\": {\n \"Path\": \"/usr/bin/nvidia-container-runtime-hook\",\n \"SkipModeDetection\": false\n }\n}","time":"2024-01-23T09:43:36+08:00"}.
This log shows up under io.containerd.runtime.v2 for both Docker and Containerd, but not under io.containerd.runtime.v1.
3.3 Change Containerd’s state Directory
1
2
3
| vim /etc/containerd/config.toml
state = "/run/containerd"
|
Change state = "/run/containerd" to state = "/data/containerd", and mount the /data directory on an additional large disk, so that even a very large log.json file will not easily fill up the storage space.
4. References
