1. What This Article Mainly Discusses

Never build a high tower on shifting sand. Business growth and the evolution of business forms both need solid, powerful IT systems to support them. Business content is transparent to the market, but IT systems cannot be built to perfection overnight. In the future, competition between companies will mainly come from competition between their IT systems, and the ability to respond quickly to business requirements will be the key to winning.

IT systems are also constantly evolving. Building an efficient, intelligent IT system is very costly. At first it only needs to be adequate, then pleasant to use, and finally it becomes a core competitive advantage.

Change is not what is frightening; a heavy historical burden is. For technical people, a new requirement is not much of a challenge — what is hard is swapping parts out while flying at high speed. It has to keep the existing functionality working, satisfy the new requirement, and replace the IT infrastructure all at the same time.

In the process of moving to containers and Kubernetes, how to distribute files directly to virtual machines (VMs) and run scripts on them is the focus of this article. Operating on virtual machines directly does not fit the cloud-native definition of immutable infrastructure, but legacy business scenarios require it, so as an IT platform provider we need to offer a solution. This article gives an answer.

2. Why a PaaS Platform Is Needed

Only when an IT operations team starts building a PaaS have they truly stood up.

In today’s environment, business models and forms are no longer trade secrets. The rapid movement of information and people leaves companies facing each other naked. Whatever business you have, I can have too; whatever feature I have, you can add too. The era of explosive short-term business growth is over; we are in an era of refined operations and data-driven decisions.

The new era places more demands on IT systems, and these demands cannot be met under the traditional model. The traditional model develops SaaS services for specific scenarios, encapsulates skills into fixed processes, cuts costs, increases efficiency, and controls risk. That was adequate early on, but as the business scales, operations staff fall into an endless cycle of overtime spent changing and adding features.

The purpose of a PaaS is to abstract certain common capabilities. Middle platforms are built the same way: stable domain implementations land on the platform and expose service interfaces to the outside, while the front end binds directly to the business to respond to rapid market change.

Only with a PaaS platform do IT skills have a direction in which to accumulate, can IT staff step back from repetitive, tedious tasks to think about the business, and only through assembly can they support the business quickly.

3. How to Implement File Distribution and Script Execution

3.1 Under a Traditional PaaS Platform

If you ask an operations person to distribute a file or run a script in bulk, Ansible lets them do it very quickly.

But as mentioned above, we want to free up hands and build a PaaS platform. Below is a traditional IT infrastructure architecture diagram:

In the traditional IT process, every machine purchased has to be registered in the CMDB and then have an Agent installed to manage it. Through the file and script pipelines the Agent exposes, the upper-layer platform can provide file distribution and script execution.

But developing an Agent is very expensive. It takes countless business failures to polish an Agent that is highly concurrent, high-performance, highly available, highly stable, and highly secure. In some open-source solutions, the Agent, being the core of the company’s IT, is not open-sourced.

3.2 Under Kubernetes

In the cloud-native context, directly modifying the state of a VM at the IaaS layer is not allowed — this is called immutable infrastructure. In some practices the container’s SSHD is even disabled, so any SSH login makes the container exit immediately.

Under Kubernetes, distributing files directly to nodes and running scripts on them is discouraged.

The logic of immutable infrastructure (IaC) is to guarantee that state is reproducible and consistent with declarative semantics. Directly modifying infrastructure is a procedural operation on infrastructure that is already running; it carries a lot of uncertainty and cannot be described precisely.

Below is an IT infrastructure architecture diagram for the cloud-native world:

Kubernetes takes over the resources at the IaaS layer and controls how the whole system operates. Business services are mainly delivered through image registries, while log collection and monitoring for the business still rely on other open-source components.

4. Plan for Distributing Files and Running Scripts with Kubernetes

4.1 Preparation for the Exercise

Here is the checklist:

• A Kubernetes cluster where the kubectl command can be run
• The VMs to be distributed to have already been added as cluster nodes
• A Docker environment and a Docker Hub account

4.2 What the Exercise Covers

The exercise is divided into the following steps:

1. Prepare the script and file to be run
2. Build and push the images
3. Create a Kubernetes Job to perform the distribution

4.3 Goals of the Exercise

The goals of the exercise are:

• Run a web service on the virtual machine that provides file downloads
• Distribute a file to the virtual machine and add it to the download service

5. Distributing Files and Running Scripts with Kubernetes

5.1 Cluster Description

1
2
3
4

kubectl get node -o wide

NAME   STATUS   ROLES           AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
test   Ready    master,worker   6d2h   v1.17.9   10.160.6.35   <none>        CentOS Linux 7 (Core)   3.10.0-957.21.3.el7.x86_64   docker://20.10.6

Due to a limited budget, no multi-node environment was set up here. But to stay close to a real scenario, when running the Job we use nodeSelector to pick specific nodes, so the distribution process does not run out of control.

5.2 Preparing the File to Distribute and the Script to Run

1. File directory structure

• demo
  • Dockerfile
  • start.sh

All the image build commands below are run inside the demo directory.

2. Contents of the script start.sh

1
2

cd /data
nohup python -m SimpleHTTPServer 8000&

The Kubernetes cluster runs CentOS 7, which ships with a Python 2 interpreter. For simplicity, we use SimpleHTTPServer to provide the download service.

3. Contents of the Dockerfile

1
2
3

FROM docker.io/alpine:3.12
ARG file
ADD ${file} /data/

4. The file to be distributed

The file can be a local file in the build environment, or any URL that links to a file. Here I chose a link to a PDF file:
https://www.chenshaowen.com/static/file/ui-autotest.pdf

5.3 Building the Images

Kubernetes works with OCI images, so the file and script have to be wrapped up: package them into images and distribute them through an image registry.

• Package the file to be distributed into an image

1

docker build --build-arg file=https://www.chenshaowen.com/static/file/ui-autotest.pdf -t shaowenchen/file-1:latest ./

Push the image:

1

docker push shaowenchen/file-1:latest

• Package the script to be executed into an image

1

docker build --build-arg file=./start.sh -t shaowenchen/shell-1:latest ./

Push the image:

1

docker push shaowenchen/shell-1:latest

• View the images in Docker Hub

5.4 Kubernetes Node Preprocessing

Besides adding the nodes to be distributed to into the Kubernetes cluster, another important point is preprocessing the nodes.

Node preprocessing mainly means adding labels to nodes to mark them so that distribution can be accurate. In production, networks are usually partitioned, so two dimensions of marking are introduced: zone and ip.

• Label the node with zone and ip

zone represents the partition, marked here as a. ip represents the virtual machine’s IP address within that partition. In practice, this can be handled in bulk when installing the Kubernetes cluster.

1
2

kubectl label node test zone=a
kubectl label node test ip=10.160.6.35

• View the applied labels

1
2
3
4

kubectl get nodes --show-labels

NAME   STATUS   ROLES           AGE    VERSION   LABELS
test   Ready    master,worker   6d2h   v1.17.9   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ip=10.160.6.35,kubernetes.io/arch=amd64,kubernetes.io/hostname=test,kubernetes.io/os=linux,node-role.kubernetes.io/master=,node-role.kubernetes.io/worker=,zone=a

5.5 Distributing the Script to a Specified Node and Running It

Here we mainly use hostpath to mount files from the container onto the host, then use nsenter to enter the host’s namespaces to perform the operation.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

cat <<EOF | kubectl apply -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: shell-1
spec:
  template:
    spec:
      containers:
      - name: shell-1
    spec:
      containers:
      - name: shell-1
        command: ["sh"]
        args: ["-c", "cp /data/start.sh /hostdata/; echo 'sh /data/start.sh&' | nsenter -t 1 -m -u -i -n;sleep 99999"]
        image: shaowenchen/shell-1:latest
        securityContext:
          privileged: true
        volumeMounts:
        - name: hostdata
          mountPath: /hostdata
      hostIPC: true
      hostNetwork: true
      hostPID: true
      volumes:
      - name: hostdata
        hostPath:
          path: /data/
      restartPolicy: Never
      nodeSelector:
        zone: a
        ip: 10.160.6.35
EOF

Since the image is very small, the script runs quickly. Log in to the virtual machine and check whether the relevant service process exists:

1
2
3
4

ps aux |grep SimpleHTTPServer

root     16523  0.1  0.0 198028 10120 ?        S    22:38   0:00 python -m SimpleHTTPServer 8000
root     17558  0.0  0.0 112684  1000 pts/1    S+   22:39   0:00 grep --color=auto SimpleHTTPServer

This shows that the SimpleHTTPServer service is running successfully on the virtual machine.

5.6 Distributing the File to a Specified Node

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

cat <<EOF | kubectl apply -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: file-1
spec:
  template:
    spec:
      containers:
      - name: file-1
    spec:
      containers:
      - name: file-1
        command: ["sh"]
        args: ["-c", "cp -R /data/* /hostdata/; sleep 10"]
        image: shaowenchen/file-1:latest
        securityContext:
          privileged: true
        volumeMounts:
        - name: hostdata
          mountPath: /hostdata
      hostIPC: true
      hostNetwork: true
      hostPID: true
      volumes:
      - name: hostdata
        hostPath:
          path: /data/
      restartPolicy: Never
      nodeSelector:
        zone: a
        ip: 10.160.6.35
EOF

Visiting the page in a browser shows the download page it provides:

View the distributed file on the virtual machine:

1
2
3

ls /data/

start.sh  ui-autotest.pdf

6. Summary

This article demonstrated, under Kubernetes, how to distribute files to and run scripts on virtual machines, offering a few ideas for designing a PaaS platform.

• Treat Kubelet as a traditional Agent. Kubelet manages Pods, while an Agent manages IaaS — there are commonalities between the two worth thinking about. In addition, a single Kubernetes cluster supports up to 5,000 nodes, which covers the vast majority of scenarios. Multiple clusters can support even more nodes.
• Distribution can support binaries from more sources. The example uses an https file, but a local file works too, and a file in S3 can be downloaded locally before packaging. Meanwhile, the final image is only a few MB larger than the original file.
• Script execution can be optimized further. When the Job finishes, the script’s execution ends too. In practice, you should add a managed service to the host. For simplicity of demonstration, that is not explored here.
• Directly using Pods with hostIPC/hostPID to replace service processes on traditional virtual machines is another option.

7. References

• How to debug containers on a host and operate the host from inside a container