OpenEBS Certificate Expiration Causes Service Unavailability

1. Installation Method

1
2
kubectl apply -f https://openebs.github.io/charts/openebs-operator.yaml
kubectl patch storageclass openebs-hostpath -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

OpenEBS is mainly used as the default storage for Tekton pipelines. I had also tried Longhorn before, but it could not hold up during peak hours and the pipelines went Pending. And after uninstalling Longhorn there were leftovers, which caused kube-apiserver to keep reporting errors, and in the end it took a great deal of effort to remove.

2. After the Kubernetes Cluster Certificate Expired, OpenEBS Became Unavailable

The Kubernetes cluster and the OpenEBS components were installed on the same day. After the Kubernetes certificate expired, I renewed it quickly with kubeadm certs renew all; the OpenEBS certificate, which I had never paid much attention to, had also expired.

Tekton Controller error

1
2
3
4
{"level":"info","ts":"2022-09-08T07:58:39.882Z",
"logger":"tekton-pipelines-controller.event-broadcaster",
"caller":"record/event.go:282",
"msg":"Event(v1.ObjectReference{Kind:\"PipelineRun\", Namespace:\"qsearch\", Name:\"p-cccq1buj5i3oh0tp2ueg\", UID:\"a1e15eee-4c44-4867-ac0e-decc16a1a0c8\", APIVersion:\"tekton.dev/v1beta1\", ResourceVersion:\"230683178\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' 1 error occurred:\n\t* failed to create PVC pvc-6dc4355ffe: Internal error occurred: failed calling webhook \"admission-webhookopenebs.io\": Post \"https://admission-server-svc.openebs.svc:443/validate?timeout=5s\": x509: certificate has expired or is not yet valid: current time 2022-09-08T07:58:39Z is after 2022-09-08T07:17:40Z\n\n","commit":"7ca5d61"}

OpenEBS Admission Server error

1
2022/09/08 07:53:45 http: TLS handshake error from x.x.x.x:5838: remote error: tls: bad certificate

3. Solution

Back up openebs-validation-webhook-cfg

1
kubectl get ValidatingWebhookConfiguration openebs-validation-webhook-cfg -o yaml > openebs-validation-webhook-cfg.yaml

Delete openebs-validation-webhook-cfg

1
kubectl delete ValidatingWebhookConfiguration openebs-validation-webhook-cfg

This is one solution given in the community Issues, reference link: https://github.com/openebs/openebs/issues/3329 .

It looks like the OpenEBS community has not yet reproduced this problem, nor has it had time to fix it. Recently I had just upgraded OpenEBS to the openebs/admission-server:2.12.1 version.

4. Why

When we delete openebs-validation-webhook-cfg, what exactly are we deleting

Look at the object being deleted:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
cat openebs-validation-webhook-cfg.yaml

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: openebs-validation-webhook-cfg
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    caBundle: xxx

Base64-decode the certificate:

1
echo xxx | base64 -d > openebs.crt

View the certificate details:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
openssl x509 -noout -text -in openebs.crt

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 0 (0x0)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = admission-server-svc-ca
        Validity
            Not Before: Sep  8 07:17:40 2021 GMT
            Not After : Sep  6 07:17:40 2031 GMT
        Subject: CN = admission-server-svc-ca

In fact, the certificate in openebs-validation-webhook-cfg has not expired; deleting openebs-validation-webhook-cfg means that when calling OpenEBS services, no admission control is performed and the validity of the data is not verified. kube-apiserver will not call admission-server-svc.openebs.svc, and so it will not report an error.

Which certificate actually expired

In admission-server-secret, I found two certificates, one is app.crt and the other is ca.crt.

1
2
3
4
5
6
7
kubectl -n openebs get secret admission-server-secret   -o yaml

apiVersion: v1
data:
  app.crt: xxxx
  app.pem: xxxx
  ca.crt: xxx

The ca.crt certificate is the same as the one above, a ten-year certificate. While app.crt is a one-year certificate. View the certificate details:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
openssl x509 -noout -text -in app.crt

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 389184800153601983 (0x566a983852307bf)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = admission-server-svc-ca
        Validity
            Not Before: Sep  8 07:17:40 2021 GMT
            Not After : Sep  8 07:17:40 2022 GMT

How to renew the certificate

In the most recently installed OpenEBS version, the admission-server-secret object can no longer be found, and there is no openebs-validation-webhook-cfg either. At the same time, on the cluster where OpenEBS was upgraded I also could not find anywhere that references the certificate, which is very strange.

1
2
3
4
kubectl -n openebs get all,sa,secret -o yaml|grep admission-server-secret

    name: admission-server-secret
    selfLink: /api/v1/namespaces/openebs/secrets/admission-server-secret

My suspicion is that the OpenEBS installed a year ago did have admission-server-secret used for the ValidatingWebhookConfiguration. The most recent OpenEBS upgrade left behind configuration from the previous version, which caused this strange phenomenon.

Since nothing references this certificate, there is no need to renew it — just back it up and delete it.