Documentation as Tooling - The Ops Tool

1. What Documentation as Tooling Means

“Documentation as tooling, tooling as a product” is a slogan I have repeated many times in my previous blog posts.

A good document is no match for a tool that is pleasant to use. A script or a single command is more direct than reading documentation and can solve the problem faster. At the same time, a lot of documentation leaves readers dizzy with knowledge; when they are in an urgent window to solve a problem and cannot fill in their knowledge system in time, misunderstandings form easily.

Knowledge with a domain barrier like this needs a bit of design folded in to transform it before users can make better use of it. That is the meaning of documentation as tooling.

From documentation to tooling, from tooling to product, the marks of refinement become ever more visible. It is a fascinating process: constantly stripping away redundancy and unnecessary things so that users can focus more on solving the problem.

2. The Ops Tool

With these ideas in mind, I had long wanted to package some of my everyday work operations as tools, so that those operations could be reused more effectively.

More than half a year ago, I started a new project called ops, mainly to help me carry out certain operations in my work.

As shown above, this is the vision for the project. The design philosophy is that the core of an operations tool lies in text distribution and script execution; implementing these two capabilities is enough to satisfy the functional needs of operations.

At present, the targets I operate on are Host machines and Kubernetes clusters, and rarely containers directly. So at the OpsObject layer I implemented the Host and Cluster objects, corresponding to hosts and Kubernetes clusters respectively.

On top of that, I implemented file distribution and script execution for hosts, and file distribution and script execution for Kubernetes clusters — that is, the Core capability layer.

Thanks to the capabilities of the Core layer, I can already carry out some simple operations tasks, such as adding hosts in bulk, or installing and changing Prometheus in bulk.

But that is not enough. Many operations cannot be completed in a single step, so some flow control is needed. For example, before backing up a cluster you need to install Velero first, and before installing Velero you need to install Helm, and so on. So I introduced a new object, Task, to handle orchestration, which in effect makes it very much like Ansible.

At the Tools layer I provide three entry points: opscli, opsserver, and opscontroller. So far I have mainly completed two of them, opscli and opscontroller.

3. opscli

opscli is a static binary that supports Linux and macOS, and can be downloaded and used directly with the curl command.

3.1 Installation

If your network connection to GitHub is good, you can install with the following command:

1
curl -sfL https://raw.githubusercontent.com/shaowenchen/ops/main/getcli.sh | VERSION=latest sh -

If your network connection to GitHub is poor, you can install with the following command:

1
curl -sfL https://raw.githubusercontent.com/shaowenchen/ops/main/getcli.sh |VERSION=latest sh -

3.2 Configuring Autocompletion

If you use bash

1
echo 'source <(opscli completion bash)' >>~/.bashrc

If you use zsh

1
echo 'source <(opscli completion zsh)' >>~/.zshrc

3.3 Usage

View help

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
opscli --help

Usage:
  opscli [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  create      command about Ops Resource
  file        transfer between local and remote file
  help        Help about any command
  shell       run shell on hosts
  task        command about task
  upgrade     upgrade to latest version
  version     get current version

Against hosts, execute commands remotely

1
2
3
opscli shell -i 1.1.1.1 --port 2222 --username root --content "uname -a"

Linux node1 5.4.219-1.el7.elrepo.x86_64 #1 SMP Sun Oct 16 10:03:45 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux

Here -i can point to a single IP, or to a file containing multiple IPs, one per line. --content points to the command to execute, and it can also be a script file.

Against a cluster, execute commands remotely

1
opscli shell -i ~/.kube/config --content "docker pull shaowenchen/ops-cli:latest" --sudo  --all

Here -i points to a kubeconfig file, and --all means the command is executed on every node in the cluster. Without --all, the command runs only on the master node; --nodename lets you specify the node the command targets.

There is also a special parameter, --incluster, which means the command will run in one of the cluster’s Pods. This is very useful for troubleshooting from inside a cluster.

Against hosts, file distribution

File distribution is a mapping process: two targets, local and remote, plus the direction of the data flow.

1
opscli file --remotefile /etc/hosts --localfile ./file1 --direction download -i 1.1.1.1 --port 2222 --username root

Here --direction indicates the direction of the data flow: download means pulling from remote to local, and upload means pushing from local to remote.

Against a cluster, file distribution

1
opscli file --remotefile /etc/hosts --localfile ./file1 --direction download -i ~/.kube/config --nodename node1

Here --nodename indicates the target node.

In fact, file distribution also supports source files coming from images and S3; I will not enumerate them all here.

Using tasks

1
2
3
4
5
6
7
8
ls ~/.ops/tasks/

add-file2image.yaml          alert-http-status.yaml       app-openebs.yaml             get-kubeconfig.yaml          pull-file.yaml               upgrade-1.16m.yaml           upgrade-kernel3to5.yaml
add-imagepullsecret.yaml     alert-promql.yaml            app-prometheus.yaml          get-osstaus.yaml             push-file.yaml               upgrade-1.16n.yaml           velero-backup.yaml
add-localbinpath.yaml        app-descheduler.yaml         clear-docker.yaml            k8s-drain-node.yaml          renew-kube-cert.yaml         upgrade-1.17m.yaml           velero-install.yaml
add-opscli.yaml              app-grafana.yaml             clear-kube.yaml              k8s-hpa.yaml                 set-docker-liverestore.yaml  upgrade-1.17n.yaml           velero-restore.yaml
add-reqlimit.yaml            app-istio.yaml               clear-opstask.yaml           list-podimage.yaml           set-hosts.yaml               upgrade-1.19m.yaml           velero-status.yaml
add-superuser.yaml           app-metricsserver.yaml       get-imagefile.yaml           list-svc.yaml                set-kernel.yaml              upgrade-base.yaml

Under the ~/.ops/tasks/ directory there are many task yaml files that you can use directly.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
opscli task -f ~/.ops/tasks/get-osstaus.yaml -i 1.1.1.1 --port 2222 --username root

> Run Task  /  on  1.1.1.1
(1/11) Kernel Version
5.4.219-1.el7.elrepo.x86_64
(2/11) CPU Usage Percent/Load/Total
10.11%/0.00/4
(3/11) Mem Usage Percent/Total
33.02%/7.8G
(4/11) Disk Usage Percent/Total
69%/52G /dev/mapper/centos-root
35%/1.1G /dev/sda1
4%/26G /dev/mapper/centos-home
(5/11) NF_Conntrack Usage/Total
1344/131072
(6/11) PID Usage
1.57%/65535
(7/11) ARP Router
0.02%/80000
(8/11) Open Files Number
1.29%/1048576
(9/11) User Instances
0.23%/8192
(10/11) User Watches
0.00%/524288
(11/11) User Processes
360/31711

For a cluster, you only need to point -i at a kubeconfig file and then add --all or specify --nodename.

4 opsserver

This part is still under development. It is mainly meant to provide an HTTP API for other systems and scripts to call.

5 opscontroller

5.1 Installation

Install Helm

1
curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

Add the Helm repository

1
helm repo add ops https://www.chenshaowen.com/ops/charts

Install ops-controller

1
helm install myops ops/ops --version 1.0.0 --namespace ops-system --create-namespace

Check the installation result

1
kubectl get pods -n ops-system

By default, opscontroller only handles CRD resource objects in the ops-system namespace.

If you need to change this, you can modify the value of ACTIVE_NAMESPACE in the environment to name a single namespace; if it is empty, all namespaces are handled.

5.2 Usage

Create a host object

1
opscli create host --name dev1 -i 1.1.1.1 --port 2222 --namespace ops-system

1
2
3
4
5
6
7
8
kubectl -n ops-system get hosts

NAME   HOSTNAME   ADDRESS       DISTRIBUTION   ARCH     CPU   MEM    DISK   HEARTTIME   HEARTSTATUS
dev1   node1      1.1.1.1       centos         x86_64   4     7.8G   52G    59s         successed
dev2   node2      1.1.1.1       centos         x86_64   4     7.8G   52G    60s         successed
dev3   node3      1.1.1.1       centos         x86_64   4     7.8G   52G    0s          successed
dev4   node4      1.1.1.1       centos         x86_64   4     7.8G   52G    2s          successed
dev5   node5      1.1.1.1       centos         x86_64   1     1.8G   95G    0s          successed

The controller periodically checks host status and updates it into the host object.

Create a cluster object

1
opscli create cluster --name dev1 -i ~/.kube/config --namespace ops-system

1
2
3
4
5
6
7
8
kubectl -n ops-system get cluster

NAME   SERVER                       VERSION   NODE   RUNNING   TOTALPOD   CERTDAYS   STATUS
dev1   https://1.1.1.1:6443         v1.21.0   1      14        14         193        successed
dev2   https://1.1.1.1:6443         v1.23.0   3      22        22         350        successed
intl   https://1.1.1.1:6443         v1.21.4   1      18        95         268        successed
prod   https://1.1.1.1:6443         v1.21.4   10     134       1098       183        successed
test   https://1.1.1.1:6443         v1.21.4   1      23        23         227        successed

The controller periodically checks cluster status and updates it into the cluster object.

Tasks can also be created through opscli, but I prefer creating them directly with yaml

The controller mainly executes the tasks in task objects on a schedule — cleaning up clusters, providing alerts, and so on.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
kubectl -n ops-system get task

NAME                             CRONTAB       TYPEREF   NAMEREF   NODENAME   ALL    STARTTIME   RUNSTATUS
alert-http-status-dockermirror   */1 * * * *
alert-http-status-harbor        */1 * * * *
alert-http-status-git           */1 * * * *
alert-http-status-mirror-go      */1 * * * *
alert-http-status-mirror-maven   */1 * * * *
alert-http-status-mirror-npm     */1 * * * *
alert-http-status-mirror-pypi    */1 * * * *
alert-promql-node-cpu            */1 * * * *
alert-promql-node-disk           */1 * * * *
alert-promql-node-io             */1 * * * *
alert-promql-node-mem            */1 * * * *
alert-promql-pending             */1 * * * *
alert-promql-pending-pod         */1 * * * *
alert-promql-pvc                 */1 * * * *
alert-trigger-autotest           */5 * * * *
clear-docker                     15 * * * *    cluster   prod                 true
clear-opstask-dev1               10 * * * *    cluster   dev1
clear-opstask-dev2               10 * * * *    cluster   dev2
clear-opstask-prod               10 * * * *    cluster   prod
clear-opstask-test               10 * * * *    cluster   test

TYPEREF points to the object type and NAMEREF points to the object name. The above is my production configuration, and below is an alert notification that was sent.

The alert is shown above. It mainly supports two modes: one is a promql query, the other is an http request.

The scheduled task clear-docker cleans up build leftovers on every node of the cluster, and clear-opstask cleans up the Pods left behind when a cluster Ops execution fails.

6. Summary

This project is mainly meant to make it easier for me to operate hosts and manage clusters, and it also provides periodic task and alerting capabilities. It is still under development and will continue to be improved.

Extracting a project out of your work is not an easy thing; you have to find the point of fit between the project and the work. Many work projects are tightly coupled with the business and cannot be extracted; and if a project is detached from work, with no real business scenarios to refine it, it is hard for it to have much value either.

At first opscli had many subcommands to cover various scenarios, but later I reimplemented them through task objects. The benefit of this is that task objects can be stored and configured separately as company-sensitive data, while the implementation part can be opened up without worrying about leaking sensitive information.

Of course, Ansible implements similar functionality too, and one of a programmer’s pleasures is endlessly reinventing the wheel. In the process of reinventing the wheel, I also relearned CRD development, learned Helm Chart development, and published it to https://artifacthub.io/.