Troubleshooting
Troubleshooting Slow IO When Building Images
· β˜• 10 min read
1. The Problem Project details: File size 5.6 GB File count 529352 Dockerfile 1 2 3 FROM golang:1.13 COPY ./ /go/src/code The build command and its output are as follows: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 time DOCKER_BUILDKIT=1 docker build --no-cache -t test:v3 -f Dockerfile .

How to Repair a Kubernetes Master Node After Reinstalling the OS
· β˜• 2 min read
Recently I ran into this twice: a host had to be reinstalled because of a failure. In one case only a single Etcd node was left, and the whole cluster stayed down for half an hour before it recovered. This post mainly records the process of initializing a fresh Ubuntu

How to Estimate Prometheus's Local Storage and Memory Consumption
· β˜• 2 min read
1. Local Storage Capacity Required disk size (GB) = data retention period _ samples fetched per second _ metric data size / 1024 / 1024 / 1024 Where Samples fetched per second: rate(prometheus_tsdb_head_samples_appended_total[1d]) Average size of a sample within one hour: rate(prometheus_tsdb_compaction_chunk_size_bytes_sum[1d])/rate(prometheus_tsdb_compaction_chunk_samples_sum[1d]) The disk consumption for one day (86400 seconds) can be queried directly in Prometheus:

Kubernetes Application Troubleshooting
· β˜• 4 min read
Setting Reasonable Req and Limit Without Req and Limit set, when an application’s CPU or MEM spikes, it harms other Pods on the same node, and can even cause cluster nodes to be crushed one after another. Req and Limit have four values in total. If only some of them are set, then when node resource usage reaches the threshold preset by Kubelet, Kubelet evicts Pods in the order Guaranteed > Burstable > Best-Effort

Kubernetes Cluster Troubleshooting
· β˜• 6 min read
FailedCreatePodSandBox Error Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:402: getting the final child's pid from pipe caused: EOF: unknown Resolution Clear the cache 1 echo 3 > /proc/sys/vm/drop_caches Cause Too much memory fragmentation calico-node Keeps Restarting increase max user Error runtime: failed to