How to Separate Events from Kubernetes to a Dedicated Etcd

The disk pressure on the original etcd is reduced to below 5%, the risk factor is very low, and the cluster shows no anomalies.

1. Deploying the Events Etcd Cluster

Events have lower consistency requirements than the main Etcd. Deploying a 3-node cluster is recommended. SSD is recommended for the disk, and the data directory should be kept separate from the main Etcd.

Deploy it as static Pods, so that kubelet manages the Etcd processes automatically, consistent with how kube-apiserver is deployed.

1.1 Cluster Deployment

Set environment variables on all nodes

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
export EVENTS_ETCD_IP_1=10.0.0.1
export EVENTS_ETCD_NAME_1=events-etcd-1

export EVENTS_ETCD_IP_2=10.0.0.2
export EVENTS_ETCD_NAME_2=events-etcd-2

export EVENTS_ETCD_IP_3=10.0.0.3
export EVENTS_ETCD_NAME_3=events-etcd-3

export EVENTS_ETCD_IMAGE=registry.aliyuncs.com/google_containers/etcd:3.5.6-0
export EVENTS_ETCD_CLIENT_PORT=2479
export EVENTS_ETCD_PEER_PORT=2480
export EVENTS_ETCD_METRICS_PORT=2481
export EVENTS_ETCD_CLUSTER="events-etcd-1=http://${EVENTS_ETCD_IP_1}:${EVENTS_ETCD_PEER_PORT},events-etcd-2=http://${EVENTS_ETCD_IP_2}:${EVENTS_ETCD_PEER_PORT},events-etcd-3=http://${EVENTS_ETCD_IP_3}:${EVENTS_ETCD_PEER_PORT}"

Print the kube-apiserver configuration

1
echo "etcd-servers-overrides=/events#http://${EVENTS_ETCD_IP_1}:${EVENTS_ETCD_CLIENT_PORT};http://${EVENTS_ETCD_IP_2}:${EVENTS_ETCD_CLIENT_PORT};http://${EVENTS_ETCD_IP_3}:${EVENTS_ETCD_CLIENT_PORT}"

Create the static Pod on each node

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
export EVENTS_ETCD_IP=$EVENTS_ETCD_IP_3
export EVENTS_ETCD_NAME=$EVENTS_ETCD_NAME_3

```bash
mkdir -p /data/etcd-events

tee /etc/kubernetes/manifests/etcd-events.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: etcd-events
  namespace: kube-system
  labels:
    component: etcd-events
spec:
  hostNetwork: true
  priorityClassName: system-cluster-critical
  containers:
  - name: etcd
    image: ${EVENTS_ETCD_IMAGE}
    command:
    - etcd
    - --name=${EVENTS_ETCD_NAME}
    - --data-dir=/data/etcd-events
    - --listen-client-urls=http://0.0.0.0:${EVENTS_ETCD_CLIENT_PORT}
    - --advertise-client-urls=http://${EVENTS_ETCD_IP}:${EVENTS_ETCD_CLIENT_PORT}
    - --listen-peer-urls=http://0.0.0.0:${EVENTS_ETCD_PEER_PORT}
    - --initial-advertise-peer-urls=http://${EVENTS_ETCD_IP}:${EVENTS_ETCD_PEER_PORT}
    - --listen-metrics-urls=http://0.0.0.0:${EVENTS_ETCD_METRICS_PORT}
    - --initial-cluster=${EVENTS_ETCD_CLUSTER}
    - --initial-cluster-state=new
    - --initial-cluster-token=events-etcd-cluster
    - --quota-backend-bytes=8589934592
    volumeMounts:
    - name: data
      mountPath: /data/etcd-events
  volumes:
  - name: data
    hostPath:
      path: /data/etcd-events
      type: DirectoryOrCreate
EOF

1.2 Verifying Cluster Status

You need to wait a few minutes.

Check the Pod status

1
kubectl -n kube-system get pod -l component=etcd-events -o wide

Check the Etcd member list

1
ETCDCTL_API=3 etcdctl --endpoints=http://${EVENTS_ETCD_IP}:2479 member list

Check Etcd Metrics

1
curl http://${EVENTS_ETCD_IP}:2481/metrics

2. Configuring kube-apiserver

Modify the kube-apiserver static Pod configuration on all Master nodes:

1
vim /etc/kubernetes/manifests/kube-apiserver.yaml

Add the overrides configuration under command:

1
- --etcd-servers-overrides=/events#http://

During the update, keep an eye on the state of kube-apiserver.

1
kubectl -n kube-system get pod -l component=kube-apiserver