kubectl logs Cannot View Pod Logs and Reports NotFound

1. Symptoms

Pod information can be viewed

1
2
3
4
kubectl -n my-testns get pod my-testpod

NAME         READY   STATUS    RESTARTS   AGE
my-testpod   1/1     Running   0          2d13h

Pod logs cannot be viewed

1
2
3
kubectl -n my-testns logs my-testpod -f

Error from server (NotFound): the server could not find the requested resource ( pods/log my-testpod)

On the host where the Pod runs, the container logs can be viewed with docker logs.

Testing the Kubelet health status returns OK

1
curl -k https://x.x.x.x:10250/healthz

Here you need to use the host’s IP address; the kubectl logs command calls the Kubelet API directly to fetch container logs.

exec into the Pod does not work

1
2
3
kubectl -n my-testns exec -it  my-testpod -- bash

Error from server:

There is no detailed error output.

No related error logs in kube-apiserver

1
kubectl -n kube-system logs -l component=kube-apiserver -f

Metrics cannot be viewed for some nodes

1
2
3
kubectl top node node-3

Error from server (NotFound): nodemetrics.metrics.k8s.io "node-3" not found

1
2
3
4
5
6
7
kubectl top node

node-03     246m         0%          4921Mi          7%
node-11     1929m        1%          152378Mi        14%
node-26     <unknown>    <unknown>   <unknown>       <unknown>
node-28     <unknown>    <unknown>   <unknown>       <unknown>
node-01     <unknown>    <unknown>   <unknown>       <unknown>

Metrics cannot be viewed for the node where the Metrics Server Pod runs; sometimes some of them work.

The Metrics Server container reports abnormal errors

1
E0221 06:27:28.020225       1 scraper.go:149] "Failed to scrape node" err="request failed, status: \"403 Forbidden\"" node="node-01"

2. Solutions

Add node-ip in Kubelet

1
vim /var/lib/kubelet/kubeadm-flags.env

Add --node-ip=x.x.x.x, then restart kubelet; here x.x.x.x is the host’s IP address.

This is the workaround mentioned in the Issue, but it had no effect for us.

Delete the Metrics Server on the problematic node

This solves the problem temporarily; the Metrics Server will be redeployed to another node, causing a similar problem to appear.

Change the Metrics Server version

On Kubernetes 1.25.6 with Docker 20.10.7, downgrading Metrics Server from 0.7.1 to 0.6.2 restored it; this is likely a compatibility problem between the higher Metrics Server version and Docker.

3. References

https://github.com/kubernetes/kubernetes/issues/125783