Certificates in Kubernetes

I am using Kubernetes 1.15.3; the way different versions handle this may vary.

1. About Certificates

A root certificate is self-signed

A root certificate is issued by itself. Browsers have the CA certificates of common certificate vendors built in. That is why browsers trust the next-level certificates issued by those vendors.

We can also generate a root certificate, but we need to add it to the system’s list of trusted certificates. That way we can issue certificates for ourselves.

Certificates are hierarchical

Certificate issuance is a chain of trust. The root CA issues the sub-CA, and the sub-CA issues to end users. Usually, to reduce the issuance pressure on the root CA, a certain number of levels of intermediate CAs are generated to manage certificate issuance hierarchically.

A certificate is a proof of trust

Certificates are divided into DV, OV, and EV. They use the same encryption algorithms but provide different levels of trust. Trust here does not refer to technology but to the entity being issued to. The higher the grade of a certificate, the stricter the verification of the entity, and the higher the compensation amount if a loss is caused by the certificate.

2. Certificates in Kubernetes

Certificate authentication is divided into one-way and two-way.

One-way authentication only requires the server to prove its identity, for example when a browser accesses a server, while two-way authentication requires the server and the client to prove their identities to each other, for example point-to-point communication in a backend.

The core components of Kubernetes adopt a two-way authentication mechanism: the client and the server both hold certificates.

Common certificate chains of trust in Kubernetes:

/etc/kubernetes/pki/ca

Issues certificates

/etc/kubernetes/pki/apiserver
/etc/kubernetes/pki/apiserver-kubelet-client

/etc/kubernetes/pki/front-proxy-ca

Issues certificates

/etc/kubernetes/pki/front-proxy-client

/etc/kubernetes/pki/etcd/ca

Issues certificates

/etc/kubernetes/pki/etcd/server
/etc/kubernetes/pki/etcd/peer
/etc/kubernetes/pki/etcd/healthcheck-client
/etc/kubernetes/pki/apiserver-etcd-client

/etc/kubernetes/pki/sa

Used for Service Account authentication.

3. Renewing Kubernetes Certificates

In a Kubernetes cluster created by Kubeadm, the certificates of components such as apiserver, controller-manager, and kubelet are valid for only one year by default.

The official recommendation is to upgrade the Kubernetes version with Kubeadm at least once a year, which updates the certificates automatically.

Check the validity period of the root CA certificate, 10 years by default:

1
2
cd /etc/kubernetes/pki
ls | grep ca.crt | xargs -I {} openssl x509 -text -in {} | grep "Not After"

Check the current certificate validity

1
kubeadm alpha certs check-expiration

Re-issue certificates

Renew all certificates

1
kubeadm alpha certs renew all

You can also renew individual ones: apiserver-etcd-client, apiserver-kubelet-client, apiserver, etcd-healthcheck-client, etcd-peer, etcd-server, front-proxy-client

1
kubeadm alpha certs renew apiserver-etcd-client

4. Updating the Domain Names or IP Sources of the apiserver Certificate

If the Master node’s IP has drifted or you want to access the cluster through a specified domain name, you need to update the apiserver certificate. The steps are as follows:

Generate the Kubeadm configuration file

1
kubectl -n kube-system get configmap kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' > kubeadm.yaml

In the kubeadm.yaml file, add a certSANs field under apiServer to specify the IPs and domain names the certificate covers

1
2
3
4
apiServer:
  certSANs:
    - "1.23.45.67"
    - "yourdomain.com"

Back up the certificates

1
mv /etc/kubernetes/pki/apiserver.{crt,key} ~

Generate a new apiserver certificate

1
kubeadm init phase certs apiserver --config kubeadm.yaml

Restart apiserver; simply restarting the Pod will not take effect

First get the apiserver’s CONTAINER_ID, then kill it, and the system will automatically bring apiserver back up.

1
2
docker ps | grep kube-apiserver | grep -v pause
docker kill {CONTAINER_ID}