This page looks best with JavaScript enabled

How to Repair a Kubernetes Cluster After Changing IPs

 ·  ☕ 3 min read

A record of repairing a cluster failure caused by an IP change. There are two clusters: one single-node (allinone) cluster, and one four-node cluster (3 masters, 1 node).

1. Update the Etcd Certificates

  • [On every Etcd node] Back up the Etcd certificates
1

cp -R /etc/ssl/etcd/ssl /etc/ssl/etcd/ssl-bak
  • View the domains in the Etcd certificate
1
2
3

openssl x509 -in /etc/ssl/etcd/ssl/node-node1.pem -noout -text|grep DNS

                DNS:etcd, DNS:etcd.kube-system, DNS:etcd.kube-system.svc, DNS:etcd.kube-system.svc.cluster.local, DNS:localhost, DNS:node1, IP Address:127.0.0.1, IP Address:0:0:0:0:0:0:0:1, IP Address:x.x.x.1

You need to record all the DNS and IP values, so they can be used to generate the new certificates.

  • [On every Etcd node] Clean up the old Etcd certificates
1

rm -f /etc/ssl/etcd/ssl/*
  • [On one Etcd node] Generate the Etcd certificate configuration
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

vim /etc/ssl/etcd/ssl/openssl.conf

[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name

[req_distinguished_name]

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[ ssl_client ]
extendedKeyUsage = clientAuth, serverAuth
basicConstraints = CA:FALSE
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer
subjectAltName = @alt_names

[ v3_ca ]
basicConstraints = CA:TRUE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
authorityKeyIdentifier=keyid:always,issuer

[alt_names]
DNS.1 = localhost
DNS.2 = etcd.kube-system.svc.cluster.local
DNS.3 = etcd.kube-system.svc
DNS.4 = etcd.kube-system
DNS.5 = etcd
DNS.6 = xxx
IP.1 = 127.0.0.1
IP.2 = x.x.x.x

It needs to include the hostname and IP address of every node that runs Etcd.

  • [On one Etcd node] Generate the Etcd CA certificate
1
2
3

cd /etc/ssl/etcd/ssl
openssl genrsa -out ca-key.pem 2048
openssl req -x509 -new -nodes -key ca-key.pem -days 3650 -out ca.pem -subj "/CN=etcd-ca"
  • [On one Etcd node] Generate the Etcd Admin certificate for each node

Set a different environment variable via export host=node1 to generate a certificate for each node. Here node1 is the hostname; keep it consistent with before, so that certificates cannot be found because of a rename.

1
2
3

openssl genrsa -out admin-${host}-key.pem 2048
openssl req -new -key admin-${host}-key.pem -out admin-${host}.csr -subj "/CN=etcd-admin-${host}"
openssl x509 -req -in admin-${host}.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out admin-${host}.pem -days 3650 -extensions ssl_client  -extfile openssl.conf
  • [On one Etcd node] Generate the Etcd Member certificate for each node

Switch nodes via export host=node1 to generate a certificate for each node.

1
2
3

openssl genrsa -out member-${host}-key.pem 2048
openssl req -new -key member-${host}-key.pem -out member-${host}.csr -subj "/CN=etcd-member-${host}" -config openssl.conf
openssl x509 -req -in member-${host}.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out member-${host}.pem -days 3650 -extensions ssl_client -extfile openssl.conf
  • [On one Etcd node] Generate the Etcd Node certificate for each node

Switch nodes via export host=node1 to generate a certificate for each node.

1
2
3

openssl genrsa -out node-${host}-key.pem 2048
openssl req -new -key node-${host}-key.pem -out node-${host}.csr -subj "/CN=etcd-node-${host}"
openssl x509 -req -in node-${host}.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out node-${host}.pem -days 3650 -extensions ssl_client  -extfile openssl.conf
  • [On one Etcd node] Distribute the generated certificates

You need to distribute the certificates under /etc/ssl/etcd/ssl/ to each Etcd node.

  • [On one Etcd node] View the etcd configuration

Here Etcd is started as a binary, so the location of the etcd configuration file can be found in systemd.

1
2
3
4

cat /etc/systemd/system/etcd.service

...
EnvironmentFile=/etc/etcd.env
  • [Every Etcd node] Replace the IPs

Since there are multiple Etcd nodes, multiple sets of IPs need to be replaced. Here we take three nodes as an example.

1
2
3
4
5
6
7
8

export oldip1=x.x.x.1
export newip1=x.x.10.1

export oldip2=x.x.x.2
export newip2=x.x.10.2

export oldip3=x.x.x.3
export newip3=x.x.10.3

1
2
3

sed -i "s/$oldip1/$newip1/" /etc/etcd.env
sed -i "s/$oldip2/$newip2/" /etc/etcd.env
sed -i "s/$oldip3/$newip3/" /etc/etcd.env

/etc/hosts also needs its IPs replaced, because the configuration files sometimes use hostnames.

1
2
3

sed -i "s/$oldip1/$newip1/" /etc/hosts
sed -i "s/$oldip2/$newip2/" /etc/hosts
sed -i "s/$oldip3/$newip3/" /etc/hosts

If there is a scheduled backup task, the related IPs also need to be replaced.

1
2
3

sed -i "s/$oldip1/$newip1/" /usr/local/bin/kube-scripts/etcd-backup.sh
sed -i "s/$oldip2/$newip2/" /usr/local/bin/kube-scripts/etcd-backup.sh
sed -i "s/$oldip3/$newip3/" /usr/local/bin/kube-scripts/etcd-backup.sh
  • [Every Etcd node] Restore Etcd data from the backup

If it is a single-node Etcd, you can skip this step. Because the node IPs have changed, the Etcd cluster can no longer run. A multi-node Etcd can only be restored from backup data, because Etcd’s node information is stored in the disk data, and merely modifying the configuration file does not help.

Distribute the Etcd backup file snapshot.db to every Etcd node.

Run the following commands on each node:

1

rm -rf /var/lib/etcd

1
2
3
4
5

etcdctl snapshot restore snapshot.db --name etcd-node1 \
        --initial-cluster "etcd-node1=https://x.x.10.1:2380,etcd-node2=https://x.x.10.2:2380,etcd-node3=https://x.x.10.3:2380" \
        --initial-cluster-token k8s_etcd \
        --initial-advertise-peer-urls https://x.x.10.1:2380 \
        --data-dir=/var/lib/etcd

Note that the etcd-node1 name and the --initial-advertise-peer-urls parameter differ on each node.

  • [Every Etcd node] Restart etcd
1

systemctl restart etcd
  • [Every Etcd node] Check the etcd status
1

systemctl status etcd

2. Update the K8s Certificates

  • Back up the certificates
1

cp -R /etc/kubernetes/ /etc/kubernetes-bak
  • [Every Kubernetes node] Replace the IP addresses in the related files
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# master 节点
export oldip1=x.x.x.1
export newip1=x.x.10.1

export oldip2=x.x.x.2
export newip2=x.x.10.2

export oldip3=x.x.x.3
export newip3=x.x.10.3

# node 节点
export oldip4=x.x.x.4
export newip4=x.x.10.4

1
2
3
4

find /etc/kubernetes -type f | xargs sed -i "s/$oldip1/$newip1/"
find /etc/kubernetes -type f | xargs sed -i "s/$oldip2/$newip2/"
find /etc/kubernetes -type f | xargs sed -i "s/$oldip3/$newip3/"
find /etc/kubernetes -type f | xargs sed -i "s/$oldip4/$newip4/"

1
2
3
4

sed -i "s/$oldip1/$newip1/" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
sed -i "s/$oldip2/$newip2/" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
sed -i "s/$oldip3/$newip3/" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
sed -i "s/$oldip4/$newip4/" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

1
2
3
4

sed -i "s/$oldip1/$newip1/" /etc/kubernetes/kubeadm-config.yaml
sed -i "s/$oldip2/$newip2/" /etc/kubernetes/kubeadm-config.yaml
sed -i "s/$oldip3/$newip3/" /etc/kubernetes/kubeadm-config.yaml
sed -i "s/$oldip4/$newip4/" /etc/kubernetes/kubeadm-config.yaml

1
2
3
4

sed -i "s/$oldip1/$newip1/" /etc/hosts
sed -i "s/$oldip2/$newip2/" /etc/hosts
sed -i "s/$oldip3/$newip3/" /etc/hosts
sed -i "s/$oldip4/$newip4/" /etc/hosts
  • [On one master node] Generate the certificates
1

rm -f /etc/kubernetes/pki/apiserver*

1

kubeadm init phase certs all --config /etc/kubernetes/kubeadm-config.yaml
  • [Every Kubernetes node] Distribute the generated certificates to the nodes

Node nodes do not need the key, only the crt.

3. Update the Conf Files of the Cluster Components

  • [On one master node] Generate the new configuration files
1
2

cd /etc/kubernetes
rm -f admin.conf kubelet.conf controller-manager.conf scheduler.conf

1

kubeadm init phase kubeconfig all --config /etc/kubernetes/kubeadm-config.yaml
  • [Every Kubernetes node] Distribute the new configuration files to each node

Every node needs /etc/kubernetes/kubelet.conf, and every master node needs /etc/kubernetes/controller-manager.conf and /etc/kubernetes/scheduler.conf.

  • [On the nodes that need kubectl] Configure the user access credentials
1
2


cp /etc/kubernetes/admin.conf $HOME/.kube/config
  • [Every Kubernetes node] Restart kubelet
1
2

systemctl daemon-reload
systemctl restart kubelet
  • [Every Kubernetes node] Check the kubelet status
1

systemctl status kubelet

4. Repair the ConfigMap

  • Replace the IPs
1

kubectl -n kube-system edit cm kube-proxy

kube-proxy affects node communication. If an LB or domain name is used as the Apiserver entry point, you can also skip the replacement. As for kubeadm-config, it was already replaced automatically in the steps above, so no extra handling is needed.

5. Summary

It is strongly recommended that you do not modify the IP addresses of cluster hosts. If a host IP change is expected, you can rebuild the cluster by way of backup and restore.

If a host IP change is unexpected, it is recommended to repair in the following order:

  1. Etcd
  2. K8s certificates
  3. K8s Master node and Node node core components
  4. Cluster ConfigMap configuration

Although the above records the repair process, when repairing multiple master nodes, the scene was very complex. Containers kept restarting, and throughout that time there were constant port conflict errors; to deal with this I even rebooted the machine once. The recorded repair process may have imperfections, but as long as you follow the order and repair one component at a time, it should not be a big problem.

微信公众号
WRITTEN BY
微信公众号

See Also