With the Operator approach, Kubernetes functionality can be extended in a friendly way. Operator = CRD + Controller. First you generate the CRD from a yaml definition, then the Controller continuously watches the data in etcd and performs the corresponding actions. There is a lot of tedious and repetitive work involved in developing an Operator. KubeBuilder can help us quickly generate skeleton code and develop a Kubernetes extension feature. For more details, refer to the documentation: Kubernetes Complex Stateful Application Management Framework – Operator. This document is mainly an attempt to develop an Operator using KubeBuilder.
1. Environment Preparation
Mine is a single-node Kubernetes 1.15.3.
- Local Kubectl permission to access the remote cluster
Just copy the cluster’s /etc/kubernetes/admin.conf to the local ~/.kube/config.
2. Hello, Kubebuilder
2.1 Installing kubebuilder and kustomize
Taking OS X as an example, the current kubebuilder version is 2.x:
1
2
| brew install kubebuilder
brew install kustomize
|
2.2 Initializing the project
1
2
3
4
| export GOPATH=$(go env GOPATH)
mkdir -p $GOPATH/src/github.com/kube-api
cd $GOPATH/src/github.com/kube-api
kubebuilder init --domain k8s.chenshaowen.com --license apache2 --owner "chenshaowen"
|
After initializing the project, KubeBuilder generates a set of configuration and code scaffolds.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
| tree -L 3
.
├── Dockerfile
├── Makefile
├── PROJECT
├── bin
│ └── manager
├── config
│ ├── certmanager
│ │ ├── certificate.yaml
│ │ ├── kustomization.yaml
│ │ └── kustomizeconfig.yaml
│ ├── default
│ │ ├── kustomization.yaml
│ │ ├── manager_auth_proxy_patch.yaml
│ │ ├── manager_prometheus_metrics_patch.yaml
│ │ ├── manager_webhook_patch.yaml
│ │ └── webhookcainjection_patch.yaml
│ ├── manager
│ │ ├── kustomization.yaml
│ │ └── manager.yaml
│ ├── rbac
│ │ ├── auth_proxy_role.yaml
│ │ ├── auth_proxy_role_binding.yaml
│ │ ├── auth_proxy_service.yaml
│ │ ├── kustomization.yaml
│ │ ├── leader_election_role.yaml
│ │ ├── leader_election_role_binding.yaml
│ │ └── role_binding.yaml
│ └── webhook
│ ├── kustomization.yaml
│ ├── kustomizeconfig.yaml
│ └── service.yaml
├── go.mod
├── go.sum
├── hack
│ └── boilerplate.go.txt
└── main.go
8 directories, 28 files
|
2.3. Adding an API
1
2
3
4
5
6
| kubebuilder create api --group groupa --version v1beta1 --kind ApiExampleA
Create Resource [y/n]
y
Create Controller [y/n]
y
......
|
KubeBuilder adds the CRD and Controller to the project.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
| tree -L 3
.
├── Dockerfile
├── Makefile
├── PROJECT # new resources section added
├── api
│ └── v1beta1 # new API description added
│ ├── apiexamplea_types.go
│ ├── groupversion_info.go
│ └── zz_generated.deepcopy.go
├── bin
│ └── manager
├── config
│ ├── certmanager
│ │ ├── certificate.yaml
│ │ ├── kustomization.yaml
│ │ └── kustomizeconfig.yaml
│ ├── crd # new CRD definition added
│ │ ├── kustomization.yaml
│ │ ├── kustomizeconfig.yaml
│ │ └── patches
│ ├── default
│ │ ├── kustomization.yaml
│ │ ├── manager_auth_proxy_patch.yaml
│ │ ├── manager_prometheus_metrics_patch.yaml
│ │ ├── manager_webhook_patch.yaml
│ │ └── webhookcainjection_patch.yaml
│ ├── manager
│ │ ├── kustomization.yaml
│ │ └── manager.yaml
│ ├── rbac
│ │ ├── auth_proxy_role.yaml
│ │ ├── auth_proxy_role_binding.yaml
│ │ ├── auth_proxy_service.yaml
│ │ ├── kustomization.yaml
│ │ ├── leader_election_role.yaml
│ │ ├── leader_election_role_binding.yaml
│ │ └── role_binding.yaml
│ ├── samples # new example for creating a CRD object added
│ │ └── groupa_v1beta1_apiexamplea.yaml
│ └── webhook
│ ├── kustomization.yaml
│ ├── kustomizeconfig.yaml
│ └── service.yaml
├── controllers # new Controller added
│ ├── apiexamplea_controller.go
│ └── suite_test.go
├── go.mod # new dependency packages added
├── go.sum
├── hack
│ └── boilerplate.go.txt
└── main.go # new handling logic added
14 directories, 36 files
|
2.4 Modifying the config for the domestic network environment
When compiling inside a container, network issues in the country mean the Go dependency packages and dependency images cannot be pulled, so the Dockerfile has to be modified.
- Add a Go proxy
- Change the image source
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
| @@ -7,7 +7,7 @@ COPY go.mod go.mod
COPY go.sum go.sum
# cache deps before building and copying source so that we don't need to re-download as much
# and so that source changes don't invalidate our downloaded layer
-RUN go mod download
+RUN GOPROXY=https://gocenter.io go mod download
# Copy the go source
COPY main.go main.go
@@ -19,7 +19,7 @@ RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -a -o manager
# Use distroless as minimal base image to package the manager binary
# Refer to https://github.com/GoogleContainerTools/distroless for more details
-FROM gcr.io/distroless/static:latest
+FROM gcr.azk8s.cn/distroless/static:latest
WORKDIR /
COPY --from=builder /workspace/manager .
ENTRYPOINT ["/manager"]
|
2.5. Building and pushing an image to test
- Build and push the image locally
Since a remote Kubernetes environment is being used, an image registry is needed for deployment.
1
2
3
4
5
6
7
| @@ -1,6 +1,6 @@
# Image URL to use all building/pushing image targets
-IMG ?= controller:latest
+IMG ?= docker.io/shaowenchen/controller:latest
# Produce CRDs that work back to Kubernetes 1.11 (no version conversion)
CRD_OPTIONS ?= "crd:trivialVersions=true"
|
- Log in to the docker.io registry
1
2
3
| docker login docker.io -u shaowenchen
Password:
Login Succeeded
|
1
| make docker-build & make docker-push
|
You can also change the image name by adding an IMG variable when running commands such as make docker-build.
- Deploy to Kubernetes
1
2
3
| kubectl get crd
NAME CREATED AT
apiexampleas.groupa.k8s.chenshaowen.com 2019-09-24T07:24:45Z
|
1
2
3
| kubectl get deploy -n kube-api-system
NAME READY UP-TO-DATE AVAILABLE AGE
kube-api-controller-manager 1/1 1 1 46s
|
1
| kubectl apply -f config/samples/groupa_v1beta1_apiexamplea.yaml
|
1
2
3
4
| kubectl get apiexampleas.groupa.k8s.chenshaowen.com
NAME AGE
apiexamplea-sample 61s
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| kubectl get apiexampleas.groupa.k8s.chenshaowen.com apiexamplea-sample -o yaml
apiVersion: groupa.k8s.chenshaowen.com/v1beta1
kind: ApiExampleA
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"groupa.k8s.chenshaowen.com/v1beta1","kind":"ApiExampleA","metadata":{"annotations":{},"name":"apiexamplea-sample","namespace":"default"},"spec":{"foo":"bar"}}
creationTimestamp: "2019-09-24T07:29:16Z"
generation: 1
name: apiexamplea-sample
namespace: default
resourceVersion: "635450"
selfLink: /apis/groupa.k8s.chenshaowen.com/v1beta1/namespaces/default/apiexampleas/apiexamplea-sample
uid: 05398ab4-7d4a-4f2e-af30-b59e61680c7e
spec:
foo: bar
|
3. Writing Logic into the Project
Through the operations above, we added a new Kubernetes object type apiexampleas.groupa.k8s.chenshaowen.com (ApiExampleA), and instantiated objects to operate on it.
These operations are merely operations on etcd data; they do not trigger any effective action. Below, let’s try to inject a bit of custom logic into the project. Implement a simple feature: add two fields, FirstName and SecondName, to the custom CRD; when an object is created, fetch these two fields in the Controller and output them to the log.
- Modify the code
In the api/v1beta1/apiexamplea_types.go file, add two fields to the ApiExampleASpec struct:
1
2
3
4
5
6
| type ApiExampleASpec struct {
// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
// Important: Run "make" to regenerate code after modifying this file
FirstName string `json:"firstname"` // add
SecondName string `json:"secondname"` // add
}
|
First, add the log package to the imports:
1
2
3
| import (
"log" // add
...
|
Then handle the logic in the Reconcile function
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
| func (r *ApiExampleAReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
// _ = context.Background()
// _ = r.Log.WithValues("apiexamplea", req.NamespacedName)
// // your logic here
// return ctrl.Result{}, nil
ctx := context.Background()
_ = r.Log.WithValues("apiexamplea", req.NamespacedName)
obja := &groupav1beta1.ApiExampleA{}
if err := r.Get(ctx, req.NamespacedName, obja); err != nil {
log.Println(err, "unable to fetch New Object")
} else {
log.Println("fetch New Object:", obja.Spec.FirstName, obja.Spec.SecondName)
}
return ctrl.Result{}, nil
}
|
- Change the image tag
The image tag is changed so that when the Deployment is deployed remotely to the remote Kubernetes cluster, the image can be re-pulled and the specified image used for deployment.
Change the IMG variable in the Makefile to: IMG ?= docker.io/shaowenchen/controller:1
- Build and release
1
| make & make docker-build & make docker-push
|
- Deploy
- Generate a CRD object instance
In the config/samples/groupa_v1beta1_apiexamplea.yaml file, change the name value and add two fields to the spec field:
1
2
3
4
5
6
7
8
9
| apiVersion: groupa.k8s.chenshaowen.com/v1beta1
kind: ApiExampleA
metadata:
name: apiexamplea-sample2
spec:
# Add fields here
# foo: bar
firstname: shaowen
secondname: chen
|
Create the CRD object
1
| kubectl create -f config/samples/groupa_v1beta1_apiexamplea.yaml
|
View the Controller’s Pod Name
1
2
3
| kubectl get pod -n kube-api-system
NAME READY STATUS RESTARTS AGE
kube-api-controller-manager-7d8bb9fc6f-8bmg9 2/2 Running 0 22m
|
View the creation log
1
2
3
4
5
6
7
8
9
| kubectl logs kube-api-controller-manager-7d8bb9fc6f-8bmg9 -c manager -n kube-api-system
2019-09-25T07:09:51.124Z INFO controller-runtime.metrics metrics server is starting to listen {"addr": "127.0.0.1:8080"}
2019-09-25T07:09:51.216Z INFO controller-runtime.controller Starting EventSource {"controller": "apiexamplea", "source": "kind source: /, Kind="}
2019-09-25T07:09:51.217Z INFO setup starting manager
2019-09-25T07:09:51.217Z INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
2019-09-25T07:10:08.111Z INFO controller-runtime.controller Starting Controller {"controller": "apiexamplea"}
2019-09-25T07:10:08.111Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"ConfigMap","namespace":"kube-api-system","name":"controller-leader-election-helper","uid":"bf307b9a-829f-478e-9306-68b6c47671fa","apiVersion":"v1","resourceVersion":"871886"}, "reason": "LeaderElection", "message": "kube-api-controller-manager-7d8bb9fc6f-8bmg9_77815cc0-df63-11e9-b8c4-e6f6cfac380f became leader"}
2019-09-25T07:10:08.211Z INFO controller-runtime.controller Starting workers {"controller": "apiexamplea", "worker count": 1}
2019/09/25 07:10:08 fetch New Object: shaowen chen
|
4. Reference
