目录

    通过 Operator 的方案,可以对 Kubernetes 的功能进行友好地扩展。Operatpr = CRD + Controller。首先通过 yaml 定义,生成 CRD ,然后 Controller 不断地监听 etcd 中的数据,执行相应动作。开发 Operator 时,有很多繁琐且重复的事情。KubeBuilder 可以帮助我们快速生成骨架代码,开发一个 Kubernetes 的扩展功能, 更多介绍可以参考文档:Kubernetes 复杂有状态应用管理框架 – Operator 。本篇文档,主要是尝试使用 KubeBuilder 开发一个 Operator 。

    1. 环境准备

    • 远程 Kubernetes 环境。

    我准备的是,单节点 Kubernetes 1.15.3 。

    • 本地 Kubectl 访问远程集群的权限。

    将集群 /etc/kubernetes/admin.conf 拷贝到本地 ~/.kube/config 即可。

    2. Hello,Kubebuilder

    2.1 安装 kubebuilder

    以 OS X 为例:

    brew install kubebuilder
    kubebuilder version
    Version: version.Version{KubeBuilderVersion:"2.0.0", KubernetesVendor:"1.14.1", GitCommit:"b31cc5d96dbc91749eb49c2cf600bd951a46d4bd", BuildDate:"2019-08-22T23:39:53Z", GoOs:"unknown", GoArch:"unknown"}
    

    2.2 初始化项目

    mkdir kube-api
    cd kube-api
    kubebuilder init --domain k8s.chenshaowen.com --license apache2 --owner "chenshaowen"
    

    初始化工程之后,KubeBuilder 会生成一系列配置和代码框架。

    tree -L 3
    .
    ├── Dockerfile
    ├── Makefile
    ├── PROJECT
    ├── bin
    │   └── manager
    ├── config
    │   ├── certmanager
    │   │   ├── certificate.yaml
    │   │   ├── kustomization.yaml
    │   │   └── kustomizeconfig.yaml
    │   ├── default
    │   │   ├── kustomization.yaml
    │   │   ├── manager_auth_proxy_patch.yaml
    │   │   ├── manager_prometheus_metrics_patch.yaml
    │   │   ├── manager_webhook_patch.yaml
    │   │   └── webhookcainjection_patch.yaml
    │   ├── manager
    │   │   ├── kustomization.yaml
    │   │   └── manager.yaml
    │   ├── rbac
    │   │   ├── auth_proxy_role.yaml
    │   │   ├── auth_proxy_role_binding.yaml
    │   │   ├── auth_proxy_service.yaml
    │   │   ├── kustomization.yaml
    │   │   ├── leader_election_role.yaml
    │   │   ├── leader_election_role_binding.yaml
    │   │   └── role_binding.yaml
    │   └── webhook
    │       ├── kustomization.yaml
    │       ├── kustomizeconfig.yaml
    │       └── service.yaml
    ├── go.mod
    ├── go.sum
    ├── hack
    │   └── boilerplate.go.txt
    └── main.go
    8 directories, 28 files
    

    2.3. 新增 API

    kubebuilder create api --group groupa --version v1beta1 --kind ApiExampleA
    Create Resource [y/n]
    y
    Create Controller [y/n]
    y
    ......
    

    KubeBuilder 会将 CRD 和 Controller 添加到工程中。

    tree -L 3
    .
    ├── Dockerfile
    ├── Makefile
    ├── PROJECT # 新增 resources 部分
    ├── api 
    │   └── v1beta1 # 新增 API 描述
    │       ├── apiexamplea_types.go
    │       ├── groupversion_info.go
    │       └── zz_generated.deepcopy.go
    ├── bin
    │   └── manager
    ├── config
    │   ├── certmanager
    │   │   ├── certificate.yaml
    │   │   ├── kustomization.yaml
    │   │   └── kustomizeconfig.yaml
    │   ├── crd # 新增 CRD 定义
    │   │   ├── kustomization.yaml
    │   │   ├── kustomizeconfig.yaml
    │   │   └── patches
    │   ├── default
    │   │   ├── kustomization.yaml
    │   │   ├── manager_auth_proxy_patch.yaml
    │   │   ├── manager_prometheus_metrics_patch.yaml
    │   │   ├── manager_webhook_patch.yaml
    │   │   └── webhookcainjection_patch.yaml
    │   ├── manager
    │   │   ├── kustomization.yaml
    │   │   └── manager.yaml
    │   ├── rbac
    │   │   ├── auth_proxy_role.yaml
    │   │   ├── auth_proxy_role_binding.yaml
    │   │   ├── auth_proxy_service.yaml
    │   │   ├── kustomization.yaml
    │   │   ├── leader_election_role.yaml
    │   │   ├── leader_election_role_binding.yaml
    │   │   └── role_binding.yaml
    │   ├── samples # 新增创建 CRD 对象示例
    │   │   └── groupa_v1beta1_apiexamplea.yaml
    │   └── webhook
    │       ├── kustomization.yaml
    │       ├── kustomizeconfig.yaml
    │       └── service.yaml
    ├── controllers # 新增 Controller
    │   ├── apiexamplea_controller.go
    │   └── suite_test.go
    ├── go.mod # 新增依赖包
    ├── go.sum
    ├── hack
    │   └── boilerplate.go.txt
    └── main.go # 新增处理逻辑
    14 directories, 36 files
    

    2.4 修改配置,适配国内环境

    在容器中进行编译时,由于国内网络问题,会导致拉取不到 Go 依赖包和依赖镜像,需要修改 Dockerfile 文件。

    • 增加 GO 代理
    • 修改镜像源
    git diff Dockerfile 
    
    @@ -7,7 +7,7 @@ COPY go.mod go.mod
     COPY go.sum go.sum
     # cache deps before building and copying source so that we don't need to re-download as much
     # and so that source changes don't invalidate our downloaded layer
    -RUN go mod download
    +RUN GOPROXY=https://gocenter.io go mod download
    
     # Copy the go source
     COPY main.go main.go
    @@ -19,7 +19,7 @@ RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -a -o manager
    
     # Use distroless as minimal base image to package the manager binary
     # Refer to https://github.com/GoogleContainerTools/distroless for more details
    -FROM gcr.io/distroless/static:latest
    +FROM gcr.azk8s.cn/distroless/static:latest
     WORKDIR /
     COPY --from=builder /workspace/manager .
     ENTRYPOINT ["/manager"]
    

    2.5. 构建并推送镜像测试

    1. 本地构建并推送镜像

    由于使用的是远程 Kubernetes 环境,需要借助镜像仓库进行部署。

    • 修改镜像名称
    git diff Makefile
    
    @@ -1,6 +1,6 @@
    
     # Image URL to use all building/pushing image targets
    -IMG ?= controller:latest
    +IMG ?= docker.io/shaowenchen/controller:latest
     # Produce CRDs that work back to Kubernetes 1.11 (no version conversion)
     CRD_OPTIONS ?= "crd:trivialVersions=true"
    
    • 登陆 docker.io 的镜像仓库
    docker login docker.io -u shaowenchen
    Password:
    Login Succeeded
    
    • 构建并推送镜像
    make docker-build & make docker-push
    

    也可以通过,在执行 make docker-build 等命令时,增加 IMG 变量修改镜像名称。

    1. 部署到 Kubernetes
    • 安装 kustomize
    brew install kustomize
    
    • 部署 CRD
    make install
    
    • 查看 CRD
    kubectl get crd
    NAME                                      CREATED AT
    apiexampleas.groupa.k8s.chenshaowen.com   2019-09-24T07:24:45Z
    
    • 部署 Controller
    make deploy
    
    • 查看 deployment
    kubectl get deploy  -n kube-api-system
    NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
    kube-api-controller-manager   1/1     1            1           46s
    
    • 创建 CRD 对象
    kubectl apply -f config/samples/groupa_v1beta1_apiexamplea.yaml
    
    • 查看 CRD 对象
    kubectl get apiexampleas.groupa.k8s.chenshaowen.com
    
    NAME                 AGE
    apiexamplea-sample   61s
    
    kubectl get apiexampleas.groupa.k8s.chenshaowen.com apiexamplea-sample  -o yaml
    
    apiVersion: groupa.k8s.chenshaowen.com/v1beta1
    kind: ApiExampleA
    metadata:
      annotations:
        kubectl.kubernetes.io/last-applied-configuration: |
          {"apiVersion":"groupa.k8s.chenshaowen.com/v1beta1","kind":"ApiExampleA","metadata":{"annotations":{},"name":"apiexamplea-sample","namespace":"default"},"spec":{"foo":"bar"}}
      creationTimestamp: "2019-09-24T07:29:16Z"
      generation: 1
      name: apiexamplea-sample
      namespace: default
      resourceVersion: "635450"
      selfLink: /apis/groupa.k8s.chenshaowen.com/v1beta1/namespaces/default/apiexampleas/apiexamplea-sample
      uid: 05398ab4-7d4a-4f2e-af30-b59e61680c7e
    spec:
      foo: bar
    

    3. 在 Project 中写入逻辑

    通过上面的操作,我们新增了 Kubernetes 对象类型 apiexampleas.groupa.k8s.chenshaowen.comApiExampleA),并实例化对象进行操作。

    这些操作仅仅只是对 etcd 数据的操作,没有触发有效动作。下面,尝试往工程中注入一点自定义逻辑。实现一个简单的功能:给自定义的 CRD 增加两个字段,FirstName、SecondName ;创建对象时,在 Controller 中获取这两个字段,并输出到日志中。

    1. 修改代码
    • 增加 CRD 字段

    api/v1beta1/apiexamplea_types.go 文件 ApiExampleASpec 结构体中增加两个字段:

    type ApiExampleASpec struct {
        // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
        // Important: Run "make" to regenerate code after modifying this file
        FirstName  string `json:"firstname"` // add
        SecondName string `json:"secondname"` // add
    }
    
    • 增加 Controller 逻辑

    首先在 import 中新增 log 包:

    import (
      "log" // add 
      ...
    

    然后在 Reconcile 函数中,处理逻辑

    func (r *ApiExampleAReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
        // _ = context.Background()
        // _ = r.Log.WithValues("apiexamplea", req.NamespacedName)
    
        // // your logic here
    
        // return ctrl.Result{}, nil
        ctx := context.Background()
        _ = r.Log.WithValues("apiexamplea", req.NamespacedName)
    
        obja := &groupav1beta1.ApiExampleA{}
        if err := r.Get(ctx, req.NamespacedName, obja); err != nil {
            log.Println(err, "unable to fetch New Object")
        } else {
            log.Println("fetch New Object:", obja.Spec.FirstName, obja.Spec.SecondName)
        }
    
        return ctrl.Result{}, nil
    }
    
    1. 修改镜像 tag

    修改镜像 tag 是为了远程 Kubernetes 集群远程部署 Deployment 时,能够重新拉取镜像,使用指定的镜像进行部署。

    修改 Makefile 文件中的变量 IMG 为:IMG ?= docker.io/shaowenchen/controller:1

    1. 构建发布
    make & make docker-build & make docker-push
    
    1. 部署
    make deploy
    
    1. 生成一个 CRD 对象实例

    config/samples/groupa_v1beta1_apiexamplea.yaml 文件中修改 name 值,并在 spec 字段新增两个字段:

    apiVersion: groupa.k8s.chenshaowen.com/v1beta1
    kind: ApiExampleA
    metadata:
      name: apiexamplea-sample2
    spec:
      # Add fields here
      # foo: bar
      firstname: shaowen
      secondname: chen
    

    创建 CRD 对象

    kubectl create config/samples/groupa_v1beta1_apiexamplea.yaml
    

    查看 Controller 的 Pod Name

    kubectl get pod -n kube-api-system
    NAME                                           READY   STATUS    RESTARTS   AGE
    kube-api-controller-manager-7d8bb9fc6f-8bmg9   2/2     Running   0          22m
    

    查看创建日志

    kubectl logs kube-api-controller-manager-7d8bb9fc6f-8bmg9 -c manager -n kube-api-system
    2019-09-25T07:09:51.124Z    INFO    controller-runtime.metrics    metrics server is starting to listen    {"addr": "127.0.0.1:8080"}
    2019-09-25T07:09:51.216Z    INFO    controller-runtime.controller    Starting EventSource    {"controller": "apiexamplea", "source": "kind source: /, Kind="}
    2019-09-25T07:09:51.217Z    INFO    setup    starting manager
    2019-09-25T07:09:51.217Z    INFO    controller-runtime.manager    starting metrics server    {"path": "/metrics"}
    2019-09-25T07:10:08.111Z    INFO    controller-runtime.controller    Starting Controller    {"controller": "apiexamplea"}
    2019-09-25T07:10:08.111Z    DEBUG    controller-runtime.manager.events    Normal    {"object": {"kind":"ConfigMap","namespace":"kube-api-system","name":"controller-leader-election-helper","uid":"bf307b9a-829f-478e-9306-68b6c47671fa","apiVersion":"v1","resourceVersion":"871886"}, "reason": "LeaderElection", "message": "kube-api-controller-manager-7d8bb9fc6f-8bmg9_77815cc0-df63-11e9-b8c4-e6f6cfac380f became leader"}
    2019-09-25T07:10:08.211Z    INFO    controller-runtime.controller    Starting workers    {"controller": "apiexamplea", "worker count": 1}
    2019/09/25 07:10:08 fetch New Object: shaowen chen
    

    4. 参考