Kubernetes Cluster Troubleshooting

FailedCreatePodSandBox

Error

Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:402: getting the final child's pid from pipe caused: EOF: unknown

Resolution

Clear the cache

1
echo 3 > /proc/sys/vm/drop_caches

Cause

Too much memory fragmentation

calico-node Keeps Restarting increase max user

Error

runtime: failed to create new OS thread (have 11 already; errno=11)，runtime: may need to increase max user processes (ulimit -u)

Resolution

Raise the ulimit limit

1
ulimit -u unlimited

Cause

The user process count is exhausted

calico-node BIRD is not ready

Error

Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/calico/bird.ctl: connect: connection refused

Resolution

Run ifconfig to find the NIC bound to the current host IP, for example ens192.

kubectl -n kube-system edit ds calico-node

Change

1
2
- name: IP_AUTODETECTION_METHOD
  value: can-reach=$(NODEIP)

1
2
- name: IP_AUTODETECTION_METHOD
  value: "interface=ens192"

so that the interface value can regex-match ens192.

Cause

Calico did not automatically identify the correct NIC.

cgroup Memory Leak cannot allocate memory

Error

mkdir /sys/fs/cgroup/memory/kubepods/burstable/pod7a1e89bd-b85e-46c6-9674-bbfd3ead02d1: cannot allocate memory

If there is any mention of fork, it may be PID exhaustion.

Temporary Resolution

Clear the cache

1
echo 3 > /proc/sys/vm/drop_caches

Permanent Resolution

Edit /etc/default/grub

Add cgroup.memory=nokmem to GRUB_CMDLINE_LINUX

Generate the configuration

/usr/sbin/grub2-mkconfig -o /boot/grub2/grub.cfg

Reboot the machine

reboot

Cause

cgroup memory leak

kubectl 404 page not found

Error

When running kubectl exec, the error error: unable to upgrade connection: 404 page not found is reported

Resolution

Add the current node’s IP to the kubelet startup arguments, Environment="KUBELET_EXTRA_ARGS=--node-ip=x.x.x.x"

Cause

The installation tool failed to accurately identify the host IP

System Call Errors and Permission Denied Inside Containers

Error

Problem executing scripts Post-Invoke Sub-process returned an error code with a permission denied message

Resolution

At runtime, add the argument --security-opt seccomp=unconfined to disable seccomp

Cause

The Seccomp security module in the kernel restricts the container’s ability to make system calls to the host.

NodePort Service Not Accessible via localhost

Error

A service exposed through NodePort cannot be accessed via localhost:port, only via the host’s ip:port.

Resolution

Check the loopback forwarding parameter

1
sysctl net.ipv4.conf.all.route_localnet

Take effect temporarily

1
sysctl -w net.ipv4.conf.all.route_localnet=1

Take effect permanently

1
echo "net.ipv4.conf.all.route_localnet=1" >> /etc/sysctl.conf && sysctl -p

Cause

ipvs mode disables this forwarding path by default

Pod Creation Fails fork/exec /usr/bin/runc

Error

1
OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/488165d6dd80c997d252ac1a5f36f41edc567cc828d98c0c0b8f1c2acf2e2524/log.json: no such file or directory):

Resolution

Check the PID limit

1
cat /proc/sys/kernel/pid_max

Check current usage

1
ps -eLf | wc -l

Permanently raise the PID limit

echo "kernel.pid_max=65535 " >> /etc/sysctl.conf && sysctl -p

kubelet volume subpaths are still present on disk

Error

A large number of kubelet error logs

1
Jan 29 01:37:40 k8s-node-2510 kubelet[2080]: E0129 01:37:40.567812    2080 kubelet_volumes.go:154] orphaned pod "1683eaf9-1b46-4ea1-99e5-337bc9c2232c" found, but volume subpaths are still present on disk : There were a total of 14 errors similar to this. Turn up verbosity to see them.

Cause

When a Pod is forcibly deleted with --force --grace-period=0, its resources are not reclaimed.

Resolution

Find the relevant Pod under /var/lib/kubelet/pods/, confirm the Pod has stopped, then delete the Pod’s directory.

Check the Pod name

1
2
3
4
5
6
7
cat /var/lib/kubelet/pods/1683eaf9-1b46-4ea1-99e5-337bc9c2232c/etc-hosts

# Kubernetes-managed hosts file.
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback

10.233.73.154 fmovedaemon-b9c68cd45-qm2wr

fmovedaemon-b9c68cd45-qm2wr is the Pod name; once you confirm the Pod has stopped, just delete the directory.

1
rm -rf /var/lib/kubelet/pods/1683eaf9-1b46-4ea1-99e5-337bc9c2232c/

kubelet MountVolume failed

Error

1
2
3
kubelet MountVolume.MountDevice failed for volume "pvc-4c79c2aa-3a55-4dde-92f2-636b30ea8921" : rpc error: code = Internal desc = format of disk "/dev/longhorn/pvc-4c79c2aa-3a55-4dde-92f2-636b30ea8921" failed: type:("ext4") target:("/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4c79c2aa-3a55-4dde-92f2-636b30ea8921/globalmount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.46.4 (18-Aug-2021)
/dev/longhorn/pvc-4c79c2aa-3a55-4dde-92f2-636b30ea8921 is apparently in use by the system; will not make a filesystem here!
)

Resolution

Edit the multipath file

1
vim /etc/multipath.conf

Add the following:

1
2
3
blacklist {
    devnode "^sd[a-z0-9]+"
}

Restart the service

1
systemctl restart multipathd.service

Cause

Multipath created a multipath device for every matching device path, including Longhorn storage volume devices, which caused the Kubelet mount error.

Large-Memory Pod Fails to Start page allocation failure

Error

1
starting container process caused \"process_linux.go:245: running exec setns process for init caused \\\"exit status 6

Resolution

1
echo 3 > /proc/sys/vm/drop_caches

Cause

System memory is fragmented, so there are not enough large pages to create the system namespace. You can check memory usage with the following command; if there are many 0s, memory is heavily fragmented:

1
cat /proc/buddyinfo

Under IPVS Mode, Service Reports No route to host

Error

No route to host

Cause

When there are a large number of short-lived connections, many connections are in the TIME_WAIT state, and the kernel reuses those connection ports.

When the kernel parameter net.ipv4.vs.conn_reuse_mode is set to 0 and a port is reused, IPVS forwards traffic directly to the previous RS, bypassing load balancing, so some traffic is forwarded to a Pod that has already been destroyed, resulting in No route to host.

Resolution

Before kernel version 5.9, using iptables mode is recommended.

But in iptables mode, once the number of services in the cluster exceeds 2000, the efficiency of rule changes and forwarding starts to drop noticeably and CPU usage rises.

After kernel version 5.9, using IPVS mode is recommended.

Set net.ipv4.vs.conn_reuse_mode to 1, forcing reused connections through load balancing
Set net.ipv4.vs.conntrack to 0, preventing IPVS from performing DROP SYNC on reused connections

Cluster kube-apiserver P99 Approaches 20s

Error

Resolution

Delete the nodes that have already been shut down and are in the NotReady state.

Cause

A possible cause is that some nodes in the cluster have been shut down but were not removed from the cluster. As a result, some request passing through kube-apiserver has to wait for a timeout, and the timeout is 20s.

The exact cause still needs further verification, but after deleting the nodes that have been shut down, the kube-apiserver P99 returns to normal.

Slow Pod Creation

Error

Sep 11 08:23:17 node3 kubelet[1437]: E0911 08:23:17.770706    1437 kubelet_volumes.go:225] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"10ff3c51-ebf2-47dd-b837-fd584319a754\" found, but error not a directory occurred when trying to remove the volumes dir" numErrs=10

Cause

One possible cause is that creating the Pod depends on resources such as Secrets and ConfigMaps, but in the current namespace these resources do not exist, so the Kubelet keeps trying to fetch them until it times out, which affects Pod creation.

Resolution

Find the missing resources and create them.

Creating a Debug Pod

Create a DaemonSet

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: demo-ubuntu-daemonset
  namespace: default
spec:
  selector:
    matchLabels:
      app: demo-ubuntu-daemonset
  template:
    metadata:
      labels:
        app: demo-ubuntu-daemonset
    spec:
      containers:
      - name: ubuntu
        image: shaowenchen/demo:ubuntu
EOF

Create a Deployment on a specified node

1
export NODE_NAME=MyHostName

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-ubuntu-deploy
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demo-ubuntu-deploy
  template:
    metadata:
      labels:
        app: demo-ubuntu-deploy
    spec:
      nodeName: $NODE_NAME
      containers:
      - name: demo-ubuntu
        image: shaowenchen/demo:ubuntu
EOF

Calico Reports timeout

1
2
2025-02-24 09:24:35.783 [ERROR][1] cni-installer/<nil> <nil>: Unable to create token for CNI kubeconfig error=Post "https://10.96.0.1:443/api/v1/namespaces/calico-system/serviceaccounts/calico-node/token": dial tcp 10.96.0.1:443: i/o timeout
2025-02-24 09:24:35.783 [FATAL][1] cni-installer/<nil> <nil>: Unable to create token for CNI kubeconfig error=Post "https://10.96.0.1:443/api/v1/namespaces/calico-system/serviceaccounts/calico-node/token": dial tcp 10.96.0.1:443: i/o timeout

Check the network connectivity between the node hosting the Calico Pod and the master node.

Configuring sysctl Parameter does not exist

1
[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist

1
2
modprobe br_netfilter
echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.conf

Kubelet Reports rpc error

1
err="rpc error: code = DeadlineExceeded desc = context deadline exceeded"

When Kubelet calls Containerd, the timeout is too short, causing the request to time out. Upgrading runc fixes it:

1
2
wget https://github.com/opencontainers/runc/releases/download/v1.2.5/runc.amd64
chmod +x runc.amd64

After downloading, replace runc

1
2
mv /usr/bin/runc /usr/bin/runc.bak
mv runc.amd64 /usr/bin/runc

Restart Kubelet and Containerd

1
2
systemctl restart containerd
systemctl restart kubelet

Try to use Containerd 1.7.0 or later.

Kube-Proxy Reports xtables lock

1
Another app is currently holding the xtables lock; still 1s 100000us time ahead to have a chance to grab the lock...

Some program is contending with Kube-Proxy for the xtables lock, possibly a Pod that keeps restarting or retrying. Common ones are

Flannel; increasing its CPU and Memory resources fixes it

The CNI Component Cannot Reach apiserver

1
cni-installer/<nil> <nil>: Unable to create token for CNI kubeconfig error=Post "https://10.96.0.1:443/api/v1/namespaces/calico-system/serviceaccounts/calico-node/token": dial tcp 10.96.0.1:443: i/o timeout^C

You can try resetting the node first

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
kubeadm reset -f

systemctl restart containerd
systemctl restart kubelet

iptables -F
iptables -t nat -F
iptables -t mangle -F
iptables -X

rm -rf /etc/cni/net.d
rm -rf /var/lib/cni/
ip link delete cni0
ip link delete flannel.1

If that still does not work, it is most likely a firewall configuration problem.

Istio 426 Upgrade Required

When accessing an Istio service, you encounter a 426 Upgrade Required or low_version error.

Istio enables only HTTP/1.1 and HTTP/2 support by default. To enable HTTP/1.0 support, set:

1
2
3
4
5
6
7
8
spec:
  template:
    spec:
      containers:
        - name: istio-proxy
          env:
            - name: ISTIO_META_HTTP10
              value: "1"

nerdctl failed to create default network

When using nerdctl, the following error is reported:

1
failed to create default network: needs CNI plugin "bridge" to be installed in CNI_PATH (/opt/cni/bin)

Fix:

1
2
3
4
5
mkdir -p /opt/cni/bin

curl -L -o cni-plugins.tgz https://github.com/containernetworking/plugins/releases/download/v1.8.0/cni-plugins-linux-amd64-v1.8.0.tgz

tar -C /opt/cni/bin -xzf cni-plugins.tgz

Nodes Joining the Cluster with a Custom Image Cause NotReady

1
2
  Warning  Rebooted                 5m3s (x819 over 3h14m)  kubelet           Node l2d01rt6b2e9hnq has been rebooted, boot id: 6a2086cb-c63e-4ae0-9b84-caf332e92cc1
  Warning  Rebooted                 99s (x704 over 121m)    kubelet           Node l2d01rt6b2e9hnq has been rebooted, boot id: c0617e04-729b-4ad8-af6f-425eba67a8f6

Fix:

1
vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Change Environment="KUBELET_EXTRA_ARGS=--node-ip=x.x.x.x --hostname-override=xxx" to the correct values or remove it entirely.

The hostname was duplicated, so a single hostname uploaded the status of multiple nodes.

Restart kubelet

1
2
systemctl daemon-reload
systemctl restart kubelet

cni0 already has an ip address

1
Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "main" network for pod "gpu-manager-containerd-daemonset-device-g2hj2": networkPlugin cni failed to set up pod "gpu-manager-containerd-daemonset-device-g2hj2" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.6.1/24

Fix:

1
2
3
4
5
6
7
8
9
# 删除 cni0 网桥
ip link set cni0 down
ip link delete cni0

# 同时建议清理 flannel.1 接口
ip link delete flannel.1
# 删除 CNI 缓存
rm -rf /var/lib/cni/*
rm -rf /var/run/flannel/subnet.env

This usually happens because leftover cni0 configuration on the node prevents Flannel from initializing the node properly; you need to delete the CNI-related configuration and reinitialize.

Node Reaches the Pod Limit

Check the number of Pods on the node

1
export NODE_NAME=your-node-name

1
kubectl describe node $NODE_NAME | grep -i pods

1
2
3
  pods:                      110
  pods:                      110
Non-terminated Pods:          (69 in total)

Change the maximum number of Pods on the node

1
vim /var/lib/kubelet/config.yaml

1
maxPods: 250

Set maxPods to a larger value, for example 250. The value set here must match the network segment settings, otherwise it will cause network conflicts.

Restart kubelet

1
2
systemctl daemon-reload
systemctl restart kubelet

Node User Password Expired Your password has expired

1
2
WARNING: Your password has expired.
You must change your password now and login again!

Check the user’s password expiration information

1
chage -l kooper

Make the user’s password never expire

1
chage -M -1 kooper

Slow Traffic for Istio-Managed Pods on a Node

Symptom

Traffic to Pods running on the node is very slow.

kubelet reports the following error:

1
2
downstream [2585] terminated with unexpected error send error for type url type.googleapis.com/envoy.config.route.v3.RouteConfiguration:
rpc error: code = Canceled desc = context canceled

Resolution

1
systemctl restart kubelet

The likely cause is excessive pressure on Istiod or a network connectivity problem, which interrupts the connection to the Envoy proxy on the node; restarting kubelet re-establishes the connection.

Istio Permission denied

Error message:

1
2
3
4
error envoy config external/envoy/source/common/listener_manager/listener_manager_impl.cc:1182 listener '0.0.0.0_80' failed to bind or apply socket options: cannot bind '0.0.0.0:80': Permission denied thread=44
warning envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:138 gRPC config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) 0.0.0.0_80: cannot bind '0.0.0.0:80': Permission denied
 thread=44
warn Envoy proxy is NOT ready: config received from XDS server, but was rejected: cds updates: 2 successful, 0 rejected; lds updates: 0 successful, 1 rejected

Normally, when a Pod is created, the following configuration is applied to allow listening on ports below 1024.

1
2
3
4
securityContext:
  sysctls:
    - name: net.ipv4.ip_unprivileged_port_start
      value: "0"

But it may not take effect; you can set it manually:

Verify the problem

1
sysctl net.ipv4.ip_unprivileged_port_start

Take effect temporarily

1
sysctl -w net.ipv4.ip_unprivileged_port_start=0

Take effect permanently

1
2
echo "net.ipv4.ip_unprivileged_port_start = 0" >> /etc/sysctl.conf
sysctl -p

Application Reports CSI Driver not found

Error message:

1
kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name fuse.csi.fluid.io not found in the list of registered CSI drivers

Fix:

Restart the storage-registered CSI Driver
Restart Kubelet

A Sudden Flood of PUT: /resource/apiservices/v1

A large number of similar errors appear in the apiserver logs

1
E0331 07:03:10.251586       1 authentication.go:70] "Unable to authenticate the request" err="[x509: certificate has expired or is not yet valid: current time 2026-03-31T07:03:10Z is after 2026-03-31T06:42:46Z, verifying certificate SN=8103692049818249166, SKID=, AKID=61:EC:11:70:82:D5:DD:91:10:A7:6B:63:9A:11:2D:88:52:C7:04:0B failed: x509: certificate has expired or is not yet valid: current time 2026-03-31T07:03:10Z is after 2026-03-31T06:42:46Z]"

The apiserver metric monitoring shows a large number of requests under the /resource/apiservices/v1 endpoint.

The cause is that the certificate on a node expired, but the other nodes are fine, so the cluster as a whole is still usable.

Fix:

Renew the certificate on the abnormal apiserver node.

Automatically Cleaning Up Failed + Succeeded Pods

Edit the kube-controller-manager configuration file:

1
vim /etc/kubernetes/manifests/kube-controller-manager.yaml

Add the following argument:

1
- --terminated-pod-gc-threshold=5

Uneven Pod Connection Counts on a Single Node

Modify the kube-proxy configuration

1
kubectl edit cm kube-proxy -n kube-system

1
2
3
4
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
ipvs:
  scheduler: "lc"

Set it to the least-connection algorithm, so that the number of connections per RS is as even as possible.

Restart kube-proxy

1
kubectl -n kube-system rollout restart daemonset kube-proxy

Verify it takes effect

1
ipvsadm -L -n

You can see that the lc algorithm is in effect.

1
2
TCP  current-node lc
  -> 10.233.73.204:8000           Masq    1      0          0

Note that IPVS only affects traffic on the current node and cannot perceive global traffic.

Istiod XDS Push P99 Reaches 30s

Exempt some namespaces from pushes

1
kubectl -n istio-system edit cm  istio

1
2
3
4
5
6
7
8
9
apiVersion: v1
data:
  mesh: |-
    discoverySelectors:
    - matchExpressions:
      - key: kubernetes.io/metadata.name
        operator: NotIn
        values:
        - monitoring    

cgroup driver different

Check

Check Kubelet:

1
grep cgroupDriver /var/lib/kubelet/config.yaml

Check Docker:

1
docker info --format '{{.CgroupDriver}}'

Check Containerd:

1
containerd config dump | grep -i SystemdCgroup

1
            SystemdCgroup = true

Resolution

Taking unify to systemd as an example.

Modify the Kubelet configuration /var/lib/kubelet/config.yaml:

1
cgroupDriver: systemd

Modify the Docker configuration /etc/docker/daemon.json:

1
2
3
{
  "exec-opts": ["native.cgroupdriver=systemd"]
}

Modify the Containerd configuration /etc/containerd/config.toml:

1
2
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

Machine Crashes After Enabling ACS and iommu in BIOS

Once ACS is enabled, the PCIe Switch forbids direct P2P communication between downstream devices, forcing all GPU<->GPU and GPU<->other PCIe device data to be forwarded through the CPU Root Complex.

Once IOMMU (Intel VT-d / AMD-Vi) is enabled, it remaps and permission-checks the DMA addresses of all devices.

If you run cross-machine training or inference tasks on the machine, it may suddenly become unresponsive.