1. Problem Description
1
2
3
4
5
6
| kubectl -n istio-system get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
istiod-647c7c9d95-7n7n6 1/1 Running 0 77m 10.244.173.51 docs-ai-a800-4 <none> <none>
istiod-647c7c9d95-k6l88 1/1 Running 0 30m 10.244.210.160 ai-a40-2 <none> <none>
istiod-647c7c9d95-pj82r 1/1 Running 0 51m 10.244.229.217 docs-ai-a800-2 <none> <none>
|
1
2
3
4
| kubectl -n istio-system get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
istiod ClusterIP 10.99.225.56 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP 645d
|
1
2
3
4
| kubectl -n istio-system get endpoints
NAME ENDPOINTS AGE
istiod 10.244.173.51:15012,10.244.210.160:15012,10.244.229.217:15012 + 9 more... 645d
|
The Endpoints match the Pod IPs.
Run a Pod on the abnormal node to test network connectivity.
1
2
3
4
5
| telnet 10.244.173.51 15012
Trying 10.244.173.51...
Connected to 10.244.173.51.
Escape character is '^]'.
^CConnection closed by foreign host.
|
1
2
3
4
5
| telnet 10.244.210.160 15012
Trying 10.244.210.160...
Connected to 10.244.210.160.
Escape character is '^]'.
^CConnection closed by foreign host.
|
1
2
3
4
5
| telnet 10.244.229.217 15012
Trying 10.244.229.217...
Connected to 10.244.229.217.
Escape character is '^]'.
^CConnection closed by foreign host.
|
The Pods serving the Service are reachable, but the Service itself is not.
1
2
| telnet 10.99.225.56 15012
Trying 10.99.225.56...
|
2. Problem Analysis
2.1 Checking the kube-apiserver Logs
1
| kubectl -n kube-system logs kube-apiserver-ai-kas-master-01 --tail 100 -f
|
1
| E0214 07:03:19.604150 1 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has expired]"
|
2.2 Checking the Node’s kube-proxy Logs
1
| kubectl -n kube-system logs kube-proxy-6c9gr -f
|
1
2
3
4
5
6
7
| E0328 05:01:29.303620 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Unauthorized
W0328 05:01:59.745555 1 reflector.go:324] k8s.io/client-go/informers/factory.go:134: failed to list *v1.EndpointSlice: Unauthorized
E0328 05:01:59.745603 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Unauthorized
W0328 05:02:09.815386 1 reflector.go:324] k8s.io/client-go/informers/factory.go:134: failed to list *v1.Service: Unauthorized
E0328 05:02:09.815433 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Unauthorized
W0328 05:02:34.999987 1 reflector.go:324] k8s.io/client-go/informers/factory.go:134: failed to list *v1.EndpointSlice: Unauthorized
E0328 05:02:35.000026 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Unauthorized
|
It looks like kube-proxy is not updating the Service information into the iptables rules, which prevents the Pod from accessing the Service.
2.3 Checking the kube-proxy Credential Configuration
1
| kubectl -n kube-system exec -it kube-proxy-6c9gr cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
| ---
-----BEGIN CERTIFICATE-----
MIIC/jCCAeagAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
cm5ldGVzMB4XDTIzMDUxODA5MjUyOVoXDTMzMDUxNTA5MjUyOVowFTETMBEGA1UE
AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK7m
ZOYNrulW7CrJrJIG1UAojwfVpbC4nT3sclaCLhn/RsdMWrCcjOzxVUVV7fNyhOU1
dGBuja8OVO8191FDioworcXebjdWtDt+35Tas8/J1z3qH4cuLK9T0SIWMnShAOp1
TvE9/gIbDuPDwlqPsCuPANW9DXDmCxbzGwMqFdLLeClEKASITc4a6cPOuFJP4/lp
tZDfA0VuKnXiUFnt31jmIefFaLtZDbY3v5ry+ubrIKxfSmw3PfN/u0/LR+eg1GEG
YGIGBp8Kix/QQzzxhcfNWLRbYmqBJuR5DsXv/qS2ILNR/Jbbfgm7HiA3JKP+7pDr
56jaVDb4LcTv/9bKQAsCAwEAAaNZMFcwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB
/wQFMAMBAf8wHQYDVR0OBBYEFOAoDQT7ZFaOU6QpRsUE0xGN0XDeMBUGA1UdEQQO
MAyCCmt1YmVybmV0ZXMwDQYJKoZIhvcNAQELBQADggEBAJ5IvgmCUPlwLL94Joll
i9YDla8pWFXemBub/aNsN7ub6bSerYH8vs1vS/ooerVSEojmC75HOPo1zq55s0iK
gpaLQtgmFYtt6GGDzhzjwg5BFEu4f7SO24aY2WCmbwsmYrLSNfeoVOw+02ammAw+
MwwdlaeNcV1UGYQSSYXM4L0F032SIqTVJgrM6uTKWHmdCutRIXAVPLgXGhIl1yaM
HXJVqstshnqR5GC/EVIx9e1onb518ItnpHwSnJaRZerV7itznu2SVYQQMksm1hTn
hElvYLbtWwM99NwWDVMz8F5TiO7y5xTa/3lUXzDvgIiTz8szOFgC5iJFtEnfEMXu
IOI=
-----END CERTIFICATE-----
|
Save the certificate as the file kube-proxy-temp.crt, then check the certificate’s validity period.
1
| openssl x509 -in kube-proxy-temp.crt -noout -enddate
|
1
| notAfter=May 15 09:25:29 2033 GMT
|
1
2
3
| kubectl -n kube-system exec -it kube-proxy-7jf78 cat /var/run/secrets/kubernetes.io/serviceaccount/token
eyJhbGciOiJSUzI1NiIsImtpZCI6IjFjSGk1VWwweHE5cUJiTmhWV0dJTEdnejFEc2xIa21JVjIwOXM3MWVFem8ifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNzI2NjU0MjU0LCJpYXQiOjE2OTUxMTgyNTQsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsInBvZCI6eyJuYW1lIjoia3ViZS1wcm94eS03amY3OCIsInVpZCI6ImIzOTkwNDVmLWFlZjMtNGM3Yy1iOWMwLTVmZmIyZDFhMTJmNCJ9LCJzZXJ2aWNlYWNjb3VudCI6eyJuYW1lIjoia3ViZS1wcm94eSIsInVpZCI6IjAyNmYzZTIyLTkxOTMtNDdkMS04M2IxLWVjNjVjYmY3YjA2NCJ9LCJ3YXJuYWZ0ZXIiOjE2OTUxMjE4NjF9LCJuYmYiOjE2OTUxMTgyNTQsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTprdWJlLXByb3h5In0.WykJsfpP1wzjiI6Q6AMDmWLbcrPaSy6NGWhhP90Xfz5Oix3rVEEthAITJyjJHEcPBLNxgNuBc6OD3FYW10nBEeTjnv7dcTJnxxKy3q-u1aywOtOjherJOR3jimRclqFAmGf5TgnZ1qpI_UXRw4--K-WDIltRkz5EYXeNStCNsHMAoJdwY-H_l_ZT3MmEKo7zCmsgAuFarSKpuaffG3RirXNZ3SuzosIhbN6KpBQ_uzI9JZOanf7i5-n8fhGR6SMqxCEYhyFvBx4AwXNPjHfCXs7K3yVk3EzrJMr6aifxh86Xzpqs-mN7E1MJGxXilTa03Xd2YlfhCT45D6yjcTdqHQ
|
Save the token as the file token, then check the token’s validity period.
1
2
3
| cat token | cut -d "." -f 2 | base64 -d 2>/dev/null | jq .exp
1726654254
|
1
2
3
| date -d @1726654254
Wed 18 Sep 2024 06:10:54 PM CST
|
It has already expired, but why did kubelet not automatically renew the Token?
3. Resolving the Problem
3.1 Restart the Abnormal Pod
1
| kubectl -n kube-system delete pod kube-proxy-6c9gr
|
After deleting the abnormal Pod, the token in the Pod is regenerated, which works around the Token expiry problem.
3.2 Check the Other kube-proxy Instances
kube-proxy directly affects traffic forwarding, so I went through the kube-proxy logs again here and found similar errors on other nodes.
1
| kubectl -n kube-system logs -l k8s-app=kube-proxy -f --max-log-requests 999 --prefix | grep --line-buffered "Unauthorized"
|
1
2
| [pod/kube-proxy-7jf78/kube-proxy] E0331 01:21:25.284912 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Unauthorized
[pod/kube-proxy-872lj/kube-proxy] W0331 01:20:00.708171 1 reflector.go:324] k8s.io/client-go/informers/factory.go:134: failed to list *v1.Service: Unauthorized
|
3.3 Check the Token Validity of All Pods
1
2
3
4
5
6
7
8
9
10
11
12
13
| kubectl get pods -A --field-selector=status.phase=Running -o json | jq -r '.items[] | {pod: .metadata.name, namespace: .metadata.namespace, node: .spec.nodeName} | @base64' | while read line; do
data=$(echo $line | base64 --decode)
pod=$(echo $data | jq -r '.pod')
namespace=$(echo $data | jq -r '.namespace')
node=$(echo $data | jq -r '.node')
token=$(kubectl exec -n "$namespace" "$pod" -- cat /var/run/secrets/kubernetes.io/serviceaccount/token 2>/dev/null)
exp=$(echo "$token" | cut -d "." -f2 | base64 -d 2>/dev/null | jq -r .exp 2>/dev/null)
now=$(date +%s)
echo "Pod: $pod, Namespace: $namespace, Node: $node, Exp: $exp"
if [[ "$exp" -lt "$now" && "$exp" != "" ]]; then
echo "Expired Token: Pod=$pod, Node=$node, Namespace=$namespace, Expiry=$(date -d @$exp)"
fi
done
|
This problem mainly occurs in K8s system-related components — kube-proxy, kube-controller-manager, kube-scheduler — and requires restarting the Pod to resolve.
4. Summary
This article mainly documented the problem of Pods on a specific node being unable to access a Service. By checking the kube-apiserver and kube-proxy logs, the cause turned out to be an expired Token.
There are two main solutions: restart the Pod, or restart kubelet. But in this case, restarting kubelet did not work.
In addition, according to the relevant documentation, if the client-go version is lower than v11.0.0 or v0.15.0, the system does not automatically reload and renew the Token, which creates a risk of Token expiry. The current K8s version is v1.23.6, which is not within that range.
Nodes running Containerd do not have this problem; the problematic nodes were all in Docker environments, so I did not dig deeper into the issue and merely recorded the solution.
Drawing an analogy, if kubelet hangs or misbehaves such that the kube-proxy token cannot be renewed, that will also cause traffic forwarding problems.
