How to Repair a Kubernetes Master Node After Reinstalling the OS

Recently I ran into this twice: a host had to be reinstalled because of a failure. In one case only a single Etcd node was left, and the whole cluster stayed down for half an hour before it recovered. This post mainly records the process of initializing a fresh Ubuntu 20.04 system. Once initialization is complete, the add nodes command of the excellent cluster installation tool Kubekey re-joins the node to the cluster with a single command, with no need to modify the configuration files.

1. Recover Etcd

A three-node Etcd cluster cannot work with only one node running. So you must repair the Etcd cluster first. Fortunately, the IP did not change after the system was reinstalled; otherwise the certificates would have to be regenerated — see the other post, How to Repair a Kubernetes Cluster After Changing IPs.

On the reinstalled node, copy the Etcd binary, configuration file, and service file from another node

1
2
3
4
scp etcd-node1:/usr/local/bin/etcd /usr/local/bin/
scp etcd-node1:/etc/etcd.env /etc/
scp etcd-node1:/etc/systemd/system/etcd.service /etc/systemd/system/
scp -r etcd-node1:/etc/ssl/etcd /etc/ssl/

Usually all nodes in an Etcd cluster start the same way, so fully copying another node is all that is needed.

On the reinstalled node, edit the configuration file and replace the host information

1
vim /etc/etcd.env

What mainly needs replacing here is the IP and NodeName; just replace them with those of the current node. These affect the validity of the node information in the certificates and in the Etcd WAL data. Keeping them the same as before helps to recover the Etcd cluster quickly.

Start Etcd

1
systemctl start etcd

Enable Etcd on boot

1
systemctl enable etcd

As long as two of the three Etcd nodes are running, the cluster works normally. What remains is to initialize the node and add it back to the cluster. What is a bit special here is that this cluster is mainly used for continuous integration, so it needs an extra storage disk mounted and a specific Docker version installed.

2. Initialize and Format the Data Disk

Install the SSD driver

Because this uses a Baocun enterprise SSD, a driver also has to be installed. According to the documentation provided, there are two ways to install the driver: one is a deb install, the other is a compile-and-install. But the deb version provided did not match Ubuntu 20.04, and the compile-and-install produced a strange error that could not be resolved.

The workable approach is to first compile a deb driver package, then install that deb package. A speed test of the Baocun SSD is given below for reference.

Check the disks

Once the SSD driver is installed, the operating system can recognize the disk.

1
2
3
4
5
6
7
lsblk

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 558.4G  0 disk
├─sda1   8:1    0     1M  0 part
└─sda2   8:2    0 558.4G  0 part /
dfa    252:0    0   2.9T  0 disk

Here /dev/dfa is the extra SSD data disk.

Use parted to create a GPT partition table

Because the disk is larger than 2TB, fdisk cannot handle it, so parted is required

1
parted /dev/dfa

1
mklabel gpt

Format the partition as btrfs

1
mkfs.btrfs /dev/dfa

Mount the partition at /data

1
2
mkdir /data
mount /dev/dfa /data

Edit /etc/fstab to enable automatic mounting of the disk

Add the following:

1
/dev/dfa  /data btrfs defaults  0  0

After making the change, you must test the following steps, or the machine may fail to boot.

Unmount the device mounted at /data

1
umount /data

Mount the devices automatically

1
mount -a

Check whether the automatic mount succeeded

1
df -h

Use fio to test random read/write speed

1
2
3
4
4K 随机写: (groupid=0, jobs=1)
  write: IOPS=78.2k, BW=306MiB/s (320MB/s)(3072MiB/10051msec)
4K 随机读: (groupid=0, jobs=1)
  read: IOPS=120k, BW=468MiB/s (490MB/s)(3072MiB/6571msec)

3. Create Symlinks Pointing to the Data Disk

Docker

1
2
3
mkdir /data/docker
ln -s /data/docker /var/lib/docker
ls -al /var/lib/docker

Kubelet

1
2
3
mkdir /data/kubelet
ln -s /data/kubelet /var/lib/kubelet
ls -al /var/lib/kubelet

Openebs

1
2
3
mkdir /data/openebs
ln -s /data/openebs /var/openebs
ls -al /var/openebs

4. Install a Specific Docker Version

Uninstall Docker

1
apt-get autoremove docker docker-ce docker-engine  docker.io  containerd runc

Add the Docker repository

1
2
curl -fsSL https://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | sudo apt-key add -
add-apt-repository "deb [arch=amd64] https://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable"

Update the system repositories

1
apt-get update

Check the available Docker versions

1
2
3
4
apt-cache madison docker-ce

 docker-ce | 5:20.10.22~3-0~ubuntu-focal | https://mirrors.aliyun.com/docker-ce/linux/ubuntu focal/stable amd64 Packages
 docker-ce | 5:20.10.12~3-0~ubuntu-focal | https://mirrors.aliyun.com/docker-ce/linux/ubuntu focal/stable amd64 Packages

Install the specified version

1
apt-get install docker-ce=5:20.10.12~3-0~ubuntu-focal

At this point the initialization of the new system is complete. The rest is left to Kubekey: since the IP did not change, you can use the configuration file from the original installation to add the Master node back to the cluster with a single command.

1
sudo ./kk add nodes -f config.yaml