https://www.chenshaowen.com/en/tags/troubleshooting/Recent content in Troubleshooting on Shaowen Chen's WebsiteHugo -- gohugo.ioen©2016 - {year}, All Rights Reserved.Tue, 02 Sep 2025 00:00:00 +0000weeklyhttps://www.chenshaowen.com/en/blog/how-upgrade-kubernetes-node-cgroup-version.htmlTue, 02 Sep 2025 00:00:00 +0000Tue, 02 Sep 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-upgrade-kubernetes-node-cgroup-version.html1. Interface Path Differences Between cgroup v1 and v2 v1 1 2 3 4 /sys/fs/cgroup/cpu/cpu.cfs_quota_us /sys/fs/cgroup/cpu/cpuacct.usage /sys/fs/cgroup/memory/memory.limit_in_bytes /sys/fs/cgroup/memory/memory.usage_in_bytes v2 1 2 3 4 /sys/fs/cgroup/cpu.max /sys/fs/cgroup/cpu.stat /sys/fs/cgroup/memory.max /sys/fs/cgroup/memory.current cgroup v2 is the upgraded version of v1, with advantages such as more unified resource hierarchy management and precise resource isolation. But it also means that when writing code, the paths for reading the relevant interface files are different, so compatibility handling is required.微信公众号KubernetesUpgradeOperationscgroupLearningNodesResource IsolationTroubleshootingKubelethttps://www.chenshaowen.com/en/blog/the-use-guide-of-nvidia-dcgm.htmlWed, 13 Aug 2025 00:00:00 +0000Wed, 13 Aug 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-use-guide-of-nvidia-dcgm.html1. What DCGM Is DCGM (Data Center GPU Manager) is a toolset NVIDIA provides for managing and monitoring GPUs in the data center. It offers the following capabilities: GPU behavior monitoring GPU configuration management GPU policy supervision GPU health and diagnostics GPU accounting and process statistics NVSwitch configuration and monitoring微信公众号AINVIDIADCGMGPUMonitoringOperationsTroubleshootinghttps://www.chenshaowen.com/en/blog/ascend-dmi-tool-usage-guide.htmlTue, 17 Jun 2025 00:00:00 +0000Tue, 17 Jun 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/ascend-dmi-tool-usage-guide.html1. Installing Dependencies MindCluster ToolBox is a system-level toolset for cluster operations and hardware management, mainly used for operations such as device monitoring, performance testing, and log collection. MindCluster ToolBox includes the Ascend DMI tool, the log collection tool, and the Ascend Cert tool. MindCluster ToolBox works by calling low-level DCMI (Device Control Management Interface) and AscendCL (Ascend Computing Language) interfaces to perform the relevant detection functions; system-level information queries are implemented by calling the general-purpose libraries provided by the system.微信公众号AIDMIAscendHuaweiNPUTroubleshootingPerformance Testinghttps://www.chenshaowen.com/en/blog/why-top-node-free-grafana-data-not-match.htmlFri, 26 Jul 2024 01:00:00 +0000Fri, 26 Jul 2024 01:00:00 +0000https://www.chenshaowen.com/en/blog/why-top-node-free-grafana-data-not-match.html1. top Shows Node Resource Usage Above 100% 1 2 3 4 5 6 kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% master-1 995m 16% 13760Mi 118% master-2 827m 13% 10672Mi 92% master-3 889m 14% 10244Mi 88% This is because usage is computed against allocatable resources by default, which excludes the portion reserved by Kubelet.微信公众号MonitoringMemoryKubernetesGrafanaOperationsTroubleshootinghttps://www.chenshaowen.com/en/blog/common-gpu-operation-and-fault-handling.htmlMon, 18 Mar 2024 00:00:00 +0000Mon, 18 Mar 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/common-gpu-operation-and-fault-handling.htmlContent referenced or recorded while handling faults, continuously updated. 1. XID Error Events XID is NVIDIA’s error code. You can retrieve it with the command: 1 dmesg -T | grep -i "NVRM: Xid" or 1 journalctl --since `date -d "10 days ago" "+%Y-%m-%d"`|grep Xid The XID lets you locate the fault.微信公众号GPUNVIDIAAIHardwareOperationsTroubleshootinghttps://www.chenshaowen.com/en/blog/health-check-runs-out-of-root-run-storage.htmlTue, 23 Jan 2024 00:00:00 +0000Tue, 23 Jan 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/health-check-runs-out-of-root-run-storage.htmlIf you use nvidia-container-runtime, this is worth your attention — especially if JuiceFS is also in the picture. 1. An Alert Out of Nowhere, and I Panicked Over the weekend I was studying TensorRT LLM, and while I was at it I installed Dragonfly on the largest production cluster. Then微信公众号KubernetesNVIDIAJuiceFSExperienceTroubleshootingOperationshttps://www.chenshaowen.com/en/blog/record-a-troubleshooting-process-for-application-slowness.htmlWed, 08 Nov 2023 00:00:00 +0000Wed, 08 Nov 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/record-a-troubleshooting-process-for-application-slowness.html1. Symptoms The business side reported that the API of application app-a was slow. Looking at the logs, one particular Pod was slow, and deleting that Pod so it moved to another node fixed it. From the monitoring metrics you can see that the Pod’s CPU usage did indeed spike sharply.微信公众号CPUNetworkApplicationTroubleshootingKubernetesOperationshttps://www.chenshaowen.com/en/blog/the-scope-of-credential-in-building.htmlThu, 06 Jul 2023 00:00:00 +0000Thu, 06 Jul 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-scope-of-credential-in-building.htmlIn the client, we have already seen the Docker CLI passing credentials to the Docker Daemon via X-Registry-Config when sending the build context. But recent build feedback still shows some phenomena that cannot be explained. This post mainly runs some basic tests to make troubleshooting easier. 1. Building under host Docker The Docker Daemon is started with root privileges.微信公众号CICDDevOpsDockerCredentialsKubernetesOperationsTroubleshootinghttps://www.chenshaowen.com/en/blog/troubleshoot-slow-io-when-building-dockerfile.htmlThu, 02 Feb 2023 00:00:00 +0000Thu, 02 Feb 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/troubleshoot-slow-io-when-building-dockerfile.html1. The Problem Project details: File size 5.6 GB File count 529352 Dockerfile 1 2 3 FROM golang:1.13 COPY ./ /go/src/code The build command and its output are as follows: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 time DOCKER_BUILDKIT=1 docker build --no-cache -t test:v3 -f Dockerfile .微信公众号DockerImagesCICDBuildTroubleshootingOperationshttps://www.chenshaowen.com/en/blog/how-to-repair-k8s-master-after-reinstalling-os.htmlTue, 17 Jan 2023 00:00:00 +0000Tue, 17 Jan 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-repair-k8s-master-after-reinstalling-os.htmlRecently I ran into this twice: a host had to be reinstalled because of a failure. In one case only a single Etcd node was left, and the whole cluster stayed down for half an hour before it recovered. This post mainly records the process of initializing a fresh Ubuntu微信公众号KubernetesRepairOperationsEtcdTroubleshootingClusterKubekeyLearninghttps://www.chenshaowen.com/en/blog/kubernetes-cluster-troubleshooting.htmlTue, 01 Nov 2022 01:00:00 +0000Tue, 01 Nov 2022 01:00:00 +0000https://www.chenshaowen.com/en/blog/kubernetes-cluster-troubleshooting.htmlFailedCreatePodSandBox Error Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:402: getting the final child's pid from pipe caused: EOF: unknown Resolution Clear the cache 1 echo 3 > /proc/sys/vm/drop_caches Cause Too much memory fragmentation calico-node Keeps Restarting increase max user Error runtime: failed to微信公众号KubernetesClusterTroubleshootingOperationsLearningFault DiagnosisPodBest PracticesProblem Localizationhttps://www.chenshaowen.com/en/blog/how-to-repair-the-kubernetes-cluster-after-changing-ip.htmlTue, 25 Oct 2022 00:00:00 +0000Tue, 25 Oct 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-repair-the-kubernetes-cluster-after-changing-ip.htmlA record of repairing a cluster failure caused by an IP change. There are two clusters: one single-node (allinone) cluster, and one four-node cluster (3 masters, 1 node). 1. Update the Etcd Certificates [On every Etcd node] Back up the Etcd certificates 1 cp -R /etc/ssl/etcd/ssl /etc/ssl/etcd/ssl-bak View the domains微信公众号KubernetesOperationsRepairTroubleshootingLearningEtcdCertificatesClusterNetworkhttps://www.chenshaowen.com/en/blog/traffic-restriction-using-linux-tc.htmlSat, 20 Aug 2022 00:00:00 +0000Sat, 20 Aug 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/traffic-restriction-using-linux-tc.html1. Traffic Control Principles Under Linux By queuing packets, we can control how data packets are sent. This kind of control is called data shaping (shape the data), and it includes the following operations on data: Adding latency Dropping packets Reordering Duplicating, corrupting Rate control Under the qdisc-class-filter structure, controlling traffic requires three steps:微信公众号LinuxTCTraffic ControlOperationsNetworkLearningRate LimitingTroubleshootinghttps://www.chenshaowen.com/en/blog/add-ssh-key-login-os-and-troubleshooting.htmlThu, 10 Mar 2022 00:00:00 +0000Thu, 10 Mar 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/add-ssh-key-login-os-and-troubleshooting.html1. Steps to Add a Key 1.1 Generate an SSH Key on the Client There are two key formats: Old format, the private key begins with -----BEGIN RSA PRIVATE KEY----- 1 ssh-keygen -m PEM -t rsa -b 4096 -C "mail@chenshaowen.com" New format, the private key begins with -----BEGIN OPENSSH PRIVATE KEY----- 1 ssh-keygen -t rsa -b 4096 -C "mail@chenshaowen.微信公众号SSHOperationsConfigurationSecurityTroubleshootinghttps://www.chenshaowen.com/en/blog/common-commands-for-cleaning-up-kubernetes-cluster-resources.htmlWed, 08 Dec 2021 00:00:00 +0000Wed, 08 Dec 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/common-commands-for-cleaning-up-kubernetes-cluster-resources.htmlClusters that run for a long time often face various kinds of resource exhaustion. On top of that, when disk space runs low, Kubelet will proactively clean up images on its own, adding another source of uncertainty. This article provides some command snippets for cleanup work. 1. Cleaning Up Basic Kubernetes Objects Clean up Pods in Evicted state 1 sudo kubectl get pods --all-namespaces -o wide | grep Evicted | awk '{print $1,$2}' | sudo xargs -L1 kubectl delete pod -n Clean up Pods in Error state 1 sudo kubectl get pods --all-namespaces -o wide | grep Error | awk '{print $1,$2}' | sudo xargs -L1 kubectl delete pod -n Clean up Pods in Completed state 1 sudo kubectl get pods --all-namespaces -o wide | grep Completed | awk '{print $1,$2}' | sudo xargs -L1 kubectl delete pod -n Clean up unused PVs 1 sudo kubectl describe -A pvc | grep -E "^Name:.微信公众号KubernetesPracticeCleanupOperationsLearningkubectlTroubleshootingResource Managementhttps://www.chenshaowen.com/en/blog/attack-vectors-under-kubernetes.htmlFri, 22 Jan 2021 00:00:00 +0000Fri, 22 Jan 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/attack-vectors-under-kubernetes.htmlThis document is mainly meant to demonstrate the dangers of Docker privileged mode, so please proceed with caution. Users without CLI access can copy the example YAML and directly create cluster workloads such as Pod, Job, and DaemonSet to carry out the operations. 1. Directly Deleting All Resources If you can log in to the machine, pack your things, and run the command:微信公众号KubernetesSecurityAttack VectorsOperationsDockerTroubleshootinghttps://www.chenshaowen.com/en/blog/how-to-cross-the-limit-of-dockerhub.htmlWed, 09 Dec 2020 00:00:00 +0000Wed, 09 Dec 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-cross-the-limit-of-dockerhub.html1. DockerHub Limits In the end, there is no getting around this error: 1 Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit Starting November 2, 2020, DockerHub officially began limiting the pull rate for non-paying users:微信公众号DockerImagesDockerHubOperationsTroubleshootingRate LimitingProxyhttps://www.chenshaowen.com/en/blog/the-question-of-lightweight-checkout-in-jenkins.htmlMon, 22 Jun 2020 00:00:00 +0000Mon, 22 Jun 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-question-of-lightweight-checkout-in-jenkins.html1. What Problem We Hit A Jenkins build log threw an error: 1 2 3 4 5 6 Started by user admin Lightweight checkout support not available, falling back to full checkout. Checking out git https://github.com/shaowenchen/pipeline-test.git into /var/jenkins_home/workspace/abc@script to read Jenkinsfile ... ... Unable to access '.git/index.lock': File exists. Cause analysis:微信公众号JenkinsGitDevOpsTroubleshootinghttps://www.chenshaowen.com/en/blog/solving-the-problem-of-celery-processe-large-file-failure.htmlWed, 24 Oct 2018 00:00:00 +0000Wed, 24 Oct 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/solving-the-problem-of-celery-processe-large-file-failure.html1. A Small Requirement We run into small requirements all the time, but they are not always simple to implement. Here is a simple file upload requirement, broken into the following steps: The user uploads a large file on the page The large file is temporarily stored in the internal微信公众号CeleryPythonDjangoFailureTroubleshootingLarge FilesR&D