Setting Reasonable Req and Limit

1. Without Req and Limit set, when an application’s CPU or MEM spikes, it harms other Pods on the same node, and can even cause cluster nodes to be crushed one after another.

2. Req and Limit have four values in total. If only some of them are set, then when node resource usage reaches the threshold preset by Kubelet, Kubelet evicts Pods in the order Guaranteed > Burstable > Best-Effort

Where:

• Guaranteed: all containers have CPU and MEM requests and limits set, and they are equal
• Burstable: at least one container has a CPU or MEM request or limit
• BestEffort: no container has a CPU or MEM request or limit set

Req and Limit should not differ too much; usually Limit = Req * 1.5, or set them based on monitoring history. The more resources a Pod consumes, the closer its Req and Limit should be.

Watch Out for Application CPU Throttling

Because CPU is a compressible resource, once a Limit is set, an application is usually still available even under high load. But it will be very slow.

This is because the system allocates CPU in slices. Within one period, if an application’s use of CPU slices reaches the Limit, the application must wait for the next period before it gets another chance to use CPU. At this point, the application is in a CPU-throttled state.

CPU throttling causes an application’s responses to become slower.

Note that CPU throttling does not happen only when a Pod’s CPU usage reaches its Limit; if the node’s CPU load is high, throttling also occurs. The resources an application can use are only what remains after the node’s own consumption.

Restarting Prometheus by Scaling

Do not do a rolling restart. A rolling restart doubles component resource consumption for a short period, affecting cluster stability.

1
2

kubectl scale deployment prom-prometheus-server --replicas=0
kubectl scale deployment prom-prometheus-server --replicas=1

Also, by default, without the --storage.tsdb.no-lockfile flag enabled, Prometheus cannot be rolling-restarted and can only be restarted using the method above.

Otherwise it reports an error, opening storage failed: lock DB directory: resource temporarily unavailable.

One storage volume with multiple Prometheus instances can cause dirty data.

“Running a Container: exec user process caused: no such file or directory”

There are two cases:

• The base image is alpine:latest; try switching to another image
• An encoding problem with the ENTRYPOINT script; change CRLF on Windows to LF on Linux

“Application OOMKilled Exit Code: 137”

There are two cases:

• The container’s memory usage exceeds the limit
• The node is out of memory

“Application getting the final child’s pid from pipe caused "EOF": unknown”

The kernel version is 3.x; it needs to be upgraded to 5.x.

Application Memory Spikes

Symptom:

The application’s instantaneous memory usage surges, then returns to normal. It may be accompanied by OOMKilled.

Cause:

Memory surges caused by concatenating strings in a for loop.

Resolution:

Use strings.Builder instead of + to concatenate strings

“Application Evicted: EmptyDir volume exceeds the limit”

Symptom:

The application is in Failed state, evicted by Kubelet, reporting the error Usage of EmptyDir volume “tmp” exceeds the limit “50Gi” .

Cause:

By default, Kubelet limits the ephemeral storage a Pod can use to 10% of the disk size and 5% of inodes. Once the limit is exceeded, Kubelet evicts the Pod.

Resolution:

1
2
3
4
5

resources:
  limits:
    ephemeral-storage: 100Gi
  requests:
    ephemeral-storage: 100Gi

You can use ephemeral-storage to specify the size of ephemeral storage.

“Application Errors: Cannot Connect to 10.233.0.1”

Symptom:

1

E0923 02:18:49.329868   42404 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.15/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: Get "https://10.233.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Djuvwi2pr849d6ug&limit=500&resourceVersion=0": dial tcp 10.233.0.1:443: connect: no route to host

Resolution:

x.x.0.1 is a special address, usually assigned to the gateway. If accessing such a service errors, it means a system component is abnormal. The error above is the exception log thrown when the kube-flannel component is abnormal.