Installing a Kubernetes Cluster with Kubeadm

1. Cluster Planning

Prepare three hosts: one Master and two Nodes.

Operating system: CentOS 7
Specs: 2 Core 4 GB
Docker version: 18.06.3
Kubernetes version: 1.15.3

If you are using a purchased cloud host, open the following ports:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
# Master
TCP     6443*       Kubernetes API Server
TCP     2379-2380   etcd server client API
TCP     10250       Kubelet API
TCP     10251       kube-scheduler
TCP     10252       kube-controller-manager
TCP     10255       Read-Only Kubelet API

# Nodes
TCP     10250       Kubelet API
TCP     10255       Read-Only Kubelet API
TCP     30000-32767 NodePort Services

2. Operations on Master and Node Hosts

Before installing Kubernetes with Kubeadm, all nodes need some basic configuration and installation.

2.1 hosts Configuration (Optional)

Configuring hosts is about being able to reach other hosts by hostname.

First check the hostname with:

1
hostname

Here we assume the hostnames are i-6fns0nua (192.168.10.2), i-m69skuyd (192.168.10.3), and i-h29fw205 (192.168.10.4).

Configure hosts on each host:

1
2
3
4
cat /etc/hosts
192.168.10.2 i-6fns0nua master
192.168.10.3 i-m69skuyd node1
192.168.10.4 i-h29fw205 node2

2.2 System Configuration

Stop and disable the firewall

1
2
systemctl stop firewalld
systemctl disable firewalld

Stop and disable selinux

1
2
setenforce 0
sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/sysconfig/selinux

Stop and disable swap

1
2
swapoff -a
sed -i 's/.*swap.*/#&/' /etc/fstab

The current Kubernetes version does not support swap.

Configure forwarding-related parameters

1
2
3
4
5
6
7
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward=1
vm.swappiness=0
EOF
sysctl --system

Prerequisites for enabling ipvs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4
yum install -y ipset ipvsadm

2.3 Installing Docker

On each host, run the command to install the latest Docker version:

1
2
3
4
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum install -y docker-ce-18.06.3.ce-3.el7
systemctl start docker.service & systemctl enable docker.service

Running docker info shows that Docker’s default Cgroup Driver is cgroupfs, while Kubelet uses systemd — the two are inconsistent.

Here we choose to change Docker’s Cgroup Driver to systemd.

mkdir -p /etc/docker
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
systemctl restart docker

Starting with version 1.13, Docker changed its default firewall rules and disabled the FOWARD chain in the iptables filter table. This causes Pods on different Nodes in a Kubernetes cluster to be unable to communicate.

Check whether the default policy of the FOWARD chain in the iptables filter table is ACCEPT.

1
2
3
4
5
6
iptables -nvL
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
14105 3771K KUBE-FORWARD  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding rules */
   43  2656 KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes service portals */
   43  2656 DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0

If it is not ACCEPT, run:

1
iptables -P FORWARD ACCEPT

2.4 Installing kubeadm, kubelet, and kubectl

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
yum install -y kubelet-1.15.3-0.x86_64 kubeadm-1.15.3-0.x86_64 kubectl-1.15.3-0.x86_64
systemctl start kubelet && systemctl enable kubelet

If the host’s network is restricted, you can use another yum repository.

3. Master Node Configuration

3.1 Initializing the Cluster with kubeadm init

On the Master node, run:

1
2
3
4
kubeadm init \
  --kubernetes-version=v1.15.3 \
  --pod-network-cidr=10.244.0.0/16 \
  --apiserver-advertise-address=192.168.10.2

kubernetes-version: specifies the version to install
pod-network-cidr: specifies the network the Pods belong to
apiserver-advertise-address: specifies the Master node

After installation completes, the console prints a message:

1
2
3
4
Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.10.2:6443 --token 7deqem.n42r8n2rnmpzfuq7 \
    --discovery-token-ca-cert-hash sha256:7a86632f54de1004bb3f38124b663f837399d6ba9aa803d58c6707a76c02a6cb

The kubeadm join command is what you will use to add Node hosts.

3.2 Configuring Kubectl

Copy the access credentials into the logged-in user’s home directory:

1
2
3
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config *

Check the cluster status:

1
2
3
4
5
kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-0               Healthy   {"health":"true"}

3.3 Installing a Pod Network Plugin

Choose one of the two network plugins:

Flannel

1
kubectl apply -f  https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Calico

You need to replace POD_CIDR with the value of pod-network-cid from kubeadn init; the default value in calico is 192.168.0.0/16.

1
2
3
curl https://docs.projectcalico.org/v3.8/manifests/calico.yaml -O
sed -i -e "s?192.168.0.0/16?10.244.0.0/16?g" calico.yaml
kubectl apply -f calico.yaml

3.4 Allowing the Master to Run Pods

1
2
kubectl taint nodes --all node-role.kubernetes.io/master-
node/i-6fns0nua untainted

3.5 Installing the Dashboard

Download the latest kubernetes-dashboard.yaml

1
wget https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml

Edit kubernetes-dashboard.yaml to change the Service type and port

To allow direct access from outside, change the Dashboard Service to NodePort and specify an access port.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

...
# ------------------- Dashboard Service ------------------- #

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  type: NodePort # 添加
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30002 # 指定端口（可选，如果不指定，端口将随机分配）
...

Create the dashboard

1
kubectl create -f kubernetes-dashboard.yaml

Add the admin user

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
cat dashboard-user.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: dashboard-admin
  namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: dashboard-admin
subjects:
  - kind: ServiceAccount
    name: dashboard-admin
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

Run the command:

1
kubectl create -f dashboard-user.yaml

View the token for the admin user login:

1
kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin | awk '{print $1}') | grep token: | awk -F : '{print $2}' | xargs echo

Access the dashboard

Check the service port:

1
2
3
kubectl -n kube-system get svc kubernetes-dashboard
NAME                   TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
kubernetes-dashboard   NodePort   10.110.76.188   <none>        443:30002/TCP   100m

Open the address https://<host_ip>:30002/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/. Choose token authentication and enter the string printed by the console in the previous step.

Besides changing the Dashboard Service type, you can also use kubectl proxy to allow outside access to the dashboard.

1
kubectl proxy --address 0.0.0.0 --accept-hosts '.*'

3.5 Testing Whether Cluster DNS Works

Create a container and enter its terminal:

1
2
3
kubectl run curl --image=radial/busyboxplus:curl -it
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
If you don't see a command prompt, try pressing enter.

Use the curl command to test DNS and the network:

1
[ root@curl-66959f6557-c5n47:/ ]nslookup kubernetes.default

4. Adding Nodes

On each Node host, with root privileges, run:

1
2
kubeadm join 192.168.10.2:6443 --token 7deqem.n42r8n2rnmpzfuq7 \
    --discovery-token-ca-cert-hash sha256:7a86632f54de1004bb3f38124b663f837399d6ba9aa803d58c6707a76c02a6cb

5. Configuring Remote Kubectl Access

On the Master node, view the access credentials

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
cat /etc/kubernetes/admin.conf
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRU...0tLQo=
    server: https://host_ip:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: LS0tLS....tLS0tCg==

On your local machine, add the configuration

Replace host_ip in the configuration from the previous step with one of kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local. Take kubernetes.default.svc.cluster.local as the example.

Add the hosts configuration:

1
2
cat /etc/hosts
<host_ip> kubernetes.default.svc.cluster.local

If you do not configure hosts for access, you will get a certificate mismatch error.

Test access from your local machine:

1
2
3
4
5
kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-0               Healthy   {"health":"true"}

6. Summary of Installation Commands

On a fresh CentOS 7, install the kubelet environment.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
systemctl stop firewalld
systemctl disable firewalld

setenforce 0
sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/sysconfig/selinux

swapoff -a
sed -i 's/.*swap.*/#&/' /etc/fstab

cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward=1
vm.swappiness=0
EOF
sysctl --system

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4
yum install -y ipset ipvsadm

yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum install -y docker-ce-18.06.3.ce-3.el7
systemctl start docker.service & systemctl enable docker.service

mkdir -p /etc/docker
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
systemctl restart docker
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
yum install -y kubelet-1.15.3-0.x86_64 kubeadm-1.15.3-0.x86_64 kubectl-1.15.3-0.x86_64
systemctl start kubelet && systemctl enable kubelet