How to Integrate Kata into a Kubernetes Cluster

1. What Problem Kata Solves

Security and isolation are what make Kata Container stand out from Docker Container.

Kata Container comes from the merger of the Intel Clear Containers and Hyper runV projects. Intel Clear Containers used Intel VT-x technology to provide containers through lightweight virtual machines, solving the security problem while delivering excellent performance. Hyper runV, on the other hand, was aimed at Docker’s runc, providing a container runtime that follows the OCI runtime specification.

2. Kata in Kubernetes

2.1 OCI and CRI-O

The OCI standard emerged to prevent the container standard from being held hostage by Docker alone.

The CRI standard decouples Kubelet from the runtime, so any container manager implementing the CRI standard can be used for Pod creation.

The CRI-O plugin implements both the CRI and OCI standards, and can be used to replace Containerd to interface directly with OCI runtimes such as runc.

As shown below:

2.2 Kata and Containerd

To stay compatible with the OCI standard, Docker split the runtime management functionality out of the Docker Daemon, forming Containerd. When running containers, you can replace Docker’s runc with kata-runtime.

2.3 Integration of Kata with Kuberntes

As shown below, what Kata mainly replaces is the OCI runtime layer; the rest is no different from a Kubernetes based on Docker runc. At the same time, Pods based on kata-runtime and Pods based on runc can coexist in the same cluster.

The main problem at present is that Kata does not support the host network. In Kubernetes, etcd, nodelocaldns, kube-apiserver, kube-scheduler, metrics-server, node-exporter, kube-proxy, calico, kube-controller-manager, and so on — that is, Static Pods and Daemonsets — all use the host network. So during installation and deployment, runc is still used as the default runtime, while kata-runtime is made available as an optional runtime for specific workloads.

3. Integrating Kata into a Kubernetes Cluster

3.1 Installing the Kubernetes Cluster

Installing a cluster with Kubeadm is very convenient; you can refer to the earlier document Installing a Kubernetes Cluster with Kubeadm.

You can also refer directly to the official documentation, https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/ .

Since Kata requires hardware virtualization support, and IaaS vendors generally do not enable the relevant features, I used physical machines and could only use the domestic network. The script below can be used to download the relevant images in advance.

Query the image list

1
2
3
4
5
6
7
8
9
kubeadm config images list

k8s.gcr.io/kube-apiserver:v1.17.9
k8s.gcr.io/kube-controller-manager:v1.17.9
k8s.gcr.io/kube-scheduler:v1.17.9
k8s.gcr.io/kube-proxy:v1.17.9
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:v3.3.12
k8s.gcr.io/coredns:1.6.9

Download the images

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
images=(
  kube-apiserver:v1.17.9
  kube-controller-manager:v1.17.9
  kube-scheduler:v1.17.9
  kube-proxy:v1.17.9
  pause:3.1
  etcd:v3.3.12
  coredns:1.6.9
)

for imageName in ${images[@]} ; do
    docker pull registry.cn-beijing.aliyuncs.com/google_containers/$imageName
    docker tag registry.cn-beijing.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName
    docker rmi registry.cn-beijing.aliyuncs.com/google_containers/$imageName
done

Check the installed Kubernetes version

1
2
3
4
kubectl version

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.9", GitCommit:"4fb7ed12476d57b8437ada90b4f93b17ffaeed99", GitTreeState:"clean", BuildDate:"2020-07-15T16:18:16Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.9", GitCommit:"4fb7ed12476d57b8437ada90b4f93b17ffaeed99", GitTreeState:"clean", BuildDate:"2020-07-15T16:10:45Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

3.2 Installing the Kata Command-Line Tool

Taking the CentOS operating system as an example:

1
2
3
4
5
6
source /etc/os-release
yum -y install yum-utils
ARCH=$(arch)
BRANCH="${BRANCH:-master}"
yum-config-manager --add-repo "http://download.opensuse.org/repositories/home:/katacontainers:/releases:/${ARCH}:/${BRANCH}/CentOS_${VERSION_ID}/home:katacontainers:releases:${ARCH}:${BRANCH}.repo"
yum -y install kata-runtime kata-proxy kata-shim

3.3 Checking Whether the Hardware Supports Kata

Kata’s hardware requirements are satisfied by any one of the following:

Intel VT-x technology.
ARM Hyp mode (virtualization extension).
IBM Power Systems.
IBM Z mainframes.

After installing kata-runtime, run the check command:

1
2
3
4
kata-runtime kata-check

System is capable of running Kata Containers
System can currently create Kata Containers

This output means the running environment supports Kata Containers.

3.4 Configuring and Testing Docker

Configure the kata-runtime parameters

1
vim /etc/docker/daemon.json

Add the following content; runc is still used by default, but Kata can be used by specifying the runtime parameter.

1
2
3
4
5
6
7
{
  "runtimes": {
    "kata-runtime": {
      "path": "/usr/bin/kata-runtime"
    }
  }
}

Restart the Docker service

1
2
systemctl daemon-reload
systemctl restart docker

Test whether Kata was installed successfully

1
2
3
docker run --runtime=kata-runtime  busybox uname -a

Linux 249a23f53475 5.4.60-65.1.container #1 SMP Thu Jan 1 00:00:00 UTC 1970 x86_64 GNU/Linux

1
2
3
docker run busybox uname -a

Linux b4812ed8990c 3.10.0-1127.el7.x86_64 #1 SMP Tue Mar 31 23:36:51 UTC 2020 x86_64 GNU/Linux

The kernel version used by the kata-runtime container differs from that of the host, which shows that kata-runtime was configured successfully.

3.5 Configuring Kubelet

Add a configuration file

1
2
3
4
5
mkdir -p  /etc/systemd/system/kubelet.service.d/
cat << EOF | sudo tee  /etc/systemd/system/kubelet.service.d/0-containerd.conf
[Service]
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock"
EOF

Restart to take effect

1
2
systemctl daemon-reload
systemctl restart kubelet

Here containerd is used. If you use CRI-O, the configuration will be different.

3.6 Providing kata-runtime to Kubernetes

You can use kata-runtime by creating a Container directly. But in a cluster, how do we tell Kubernetes which workloads need to use kata-runtime? Depending on the version, Kata provides different ways to do this.

First, you need to generate the containerd configuration file in all cases

1
containerd config default > /etc/containerd/config.toml

The RuntimeClass approach

This approach has version requirements for the relevant components:

Kata Containers v1.5.0 or above (including 1.5.0-rc)
Containerd v1.2.0 or above
Kubernetes v1.12.0 or above

In the config.toml configuration file, add the following content:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
    [plugins.cri.containerd]
      no_pivot = false
    [plugins.cri.containerd.runtimes]
      [plugins.cri.containerd.runtimes.runc]
         runtime_type = "io.containerd.runc.v1"
         [plugins.cri.containerd.runtimes.runc.options]
           NoPivotRoot = false
           NoNewKeyring = false
           ShimCgroup = ""
           IoUid = 0
           IoGid = 0
           BinaryName = "runc"
           Root = ""
           CriuPath = ""
           SystemdCgroup = false
      [plugins.cri.containerd.runtimes.kata]
         runtime_type = "io.containerd.kata.v2"
      [plugins.cri.containerd.runtimes.katacli]
         runtime_type = "io.containerd.runc.v1"
         [plugins.cri.containerd.runtimes.katacli.options]
           NoPivotRoot = false
           NoNewKeyring = false
           ShimCgroup = ""
           IoUid = 0
           IoGid = 0
           BinaryName = "/usr/bin/kata-runtime"
           Root = ""
           CriuPath = ""
           SystemdCgroup = false

Here the kata in [plugins.cri.containerd.runtimes.kata] will be used as the RuntimeClass handler key.

The untrusted_workload_runtime approach

For environments that do not meet the version requirements above, the earlier approach can be used.

Add the following content to the configuration file:

1
2
3
    [plugins.cri.containerd.untrusted_workload_runtime]
      runtime_type = "io.containerd.runtime.v1.linux"
      runtime_engine = "/usr/bin/kata-runtime"

Finally, in all cases you need to restart containerd.

1
2
containerd systemctl daemon-reload
systemctl restart containerd

4. Using kata-runtime

4.1 The RuntimeClass Approach

Create a RuntimeClass

kata-runtime.yaml

1
2
3
4
5
kind: RuntimeClass
apiVersion: node.k8s.io/v1beta1
metadata:
  name: kata-containers
handler: kata

You can also create a RuntimeClass for runc

1
2
3
4
kubectl get runtimeclass

NAME              CREATED AT
kata-containers   2020-08-30

Create the workload kata-pod.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
apiVersion: v1
kind: Pod
metadata:
  name: kata-nginx
spec:
  runtimeClassName: kata-containers
  containers:
    - name: nginx
      image: nginx
      ports:
        - containerPort: 80

1
kubectl apply -f kata-pod.yaml

View the workload

1
kata-runtime list

4.2 The untrusted_workload_runtime Approach

untrusted_workload_runtime uses annotations to tell the Kubernetes cluster which workloads need to use kata-runtime.

1
2
annotations:
  io.kubernetes.cri.untrusted-workload: "true"

Below is an example, kata-pod-untrusted.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
apiVersion: v1
kind: Pod
metadata:
  name: kata-nginx-untrusted
  annotations:
    io.kubernetes.cri.untrusted-workload: "true"
spec:
  containers:
    - name: nginx
      image: nginx
      ports:
        - containerPort: 80

1
kubectl apply -f kata-pod-untrusted.yaml