This post comes mainly from an internal sharing session, and is also a summary of some recent work. 1. The Common Failure Handling Process The figure above shows a typical operations exception handling process. Along the timeline, there are the following key points in time: Failure occurs Failure detected Failure

1. Why Migrate Metadata Storage from Redis to PGSQL PGSQL is cheaper Redis stores metadata in memory, while PGSQL stores metadata on disk, so the cost difference is obvious. PGSQL performance is tunable Different PGSQL services offer different performance IOPS. If you do not have a continuously high performance requirement, PGSQL is a good choice.

1. top Shows Node Resource Usage Above 100% 1 2 3 4 5 6 kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% master-1 995m 16% 13760Mi 118% master-2 827m 13% 10672Mi 92% master-3 889m 14% 10244Mi 88% This is because usage is computed against allocatable resources by default, which excludes the portion reserved by Kubelet.

1. Checking the CPU Check the CPU model 1 lscpu Check the instruction set 1 cat /proc/cpuinfo | grep -iE "flags|instruction set" 2. Checking Memory Memory usage and size 1 free -h Memory module model 1 dmidecode -t memory 3. Checking Disks Check disk mounts 1 lsblk -o NAME,TYPE,SIZE,MODEL,UUID,MOUNTPOINT Check disk usage 1 df -H | grep -vE '^Filesystem|tmpfs|cdrom|loop|udev' | awk '{ print $5 "/" $2 " " $1 }' |grep " "/ Check disk speed 1 hdparm -t /dev/sda3 Check disk temperature and more 1 apt install smartmontools 1 (for disk in /dev/sd?

This article is compiled from part of an internal sharing session I gave. 1. Core Elements of a Storage System 1.1 Security Credentials for object storage buckets, authorization when using storage PVCs, and control over access sources — these are all issues security needs to address. But they are also very easy to overlook, and when something goes wrong it is a big problem.