Kubernetes Cluster Operations in Practice

Compiled from the “Development Tips” series, gathering the common problems and solutions from day-to-day Kubernetes cluster operations.

1. Configuring Multiple Clusters with Kubectl

When doing Kubernetes-related development you usually end up managing several clusters. Kubectl provides multi-cluster context management.

Kubectl’s configuration usually lives in $HOME/.kube/config or /etc/kubernetes/admin.conf. Log into the machine, look at the cluster configuration, and edit it using the format below.

kubeconfig configuration format

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
apiVersion: v1
kind: Config
preferences: {}

clusters:
  - cluster:
      certificate-authority-data: xxx
      server: xxx
    name: { cluster-name1 }
  - cluster:
      certificate-authority-data: xxx
      server: xxx
    name: { cluster-name2 }

users:
  - name: { user-name1 }
    user: xxx
  - name: { user-name2 }
    user: xxx

contexts:
  - context:
      cluster: { cluster-name1 }
      user: { user-name1 }
    name: { context-name1 }
  - context:
      cluster: { cluster-name2 }
      user: { user-name2 }
    name: { context-name2 }

current-context: { context-name1 }

List clusters

1
kubectl config get-contexts

View the config

1
kubectl config view

Switch clusters

1
kubectl config use-context {context-name}

2. helm Reports cannot get resource “namespaces”

Installing an application with helm:

1
2
helm install --name prometheus-operator --namespace=monitoring stable/prometheus-operator
Error: namespaces "monitoring" is forbidden: User "system:serviceaccount:kube-system:default" cannot get resource "namespaces" in API group "" in the namespace "monitoring"

The fix is to add a service account:

1
2
3
kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'

3. Adding a Role to a Node in Kubernetes: worker

1
2
3
4
kubectl get nodes
NAME         STATUS   ROLES    AGE    VERSION
i-6fns0nua   Ready    master   6d3h   v1.15.2
i-m69skuyd   Ready    <none>   6d2h   v1.15.2

1
2
kubectl label node i-m69skuyd  node-role.kubernetes.io/worker=
node/i-m69skuyd labeled

1
2
3
4
kubectl get node
NAME         STATUS   ROLES    AGE    VERSION
i-6fns0nua   Ready    master   6d3h   v1.15.2
i-m69skuyd   Ready    worker   6d2h   v1.15.2

4. Removing a Node from Kubernetes

List the current nodes:

1
2
3
4
kubectl get node
NAME         STATUS   ROLES    AGE    VERSION
i-6fns0nua   Ready    master   6d3h   v1.15.2
i-m69skuyd   Ready    worker   6d2h   v1.15.2

Migrate the Pods by cordoning the node to be deleted:

1
kubectl drain i-m69skuyd --delete-local-data --force --ignore-daemonsets

Delete the node:

1
kubectl delete node i-m69skuyd

5. Installing Prometheus on Kubernetes with Helm

First create the PV; see chapter 1 for reference.

Then run the commands:

1
2
3
4
5
helm install --namespace monitor --name prometheus stable/prometheus \
  --set alertmanager.persistentVolume.storageClass="local" \
  --set server.persistentVolume.storageClass="local"
helm install  --namespace monitor --name grafana stable/grafana \
  --set persistence.storageClassName="local"

For other operations, see Installing Prometheus on a Minikube Cluster.

6. How to Restart a Pod or Service in Kubernetes

Restarting a Pod

If the Pod is managed by a replica controller, just delete the Pod and Kubernetes will recreate it.

List Pods

1
kubectl get pod -n {NAMESPACE}

Delete a Pod

1
kubectl delete pod {POD_NAME} -n {NAMESPACE}

Another approach is to use replace.

1
kubectl replace --force -f pod.yaml

If you do not have a pod.yaml file, you can use the following command directly:

1
kubectl get pod {POD_NAME} -n {NAMESPACE} -o yaml | kubectl replace --force -f -

Restarting a Deployment

Scale the service’s replicas to 0, then restore the original replica count.

List services

1
kubectl get deployment -n {NAMESPACE}

Set replicas to 0

1
kubectl scale deployment {DEPLOYMENT_NAME} --replicas=0 -n {NAMESPACE}

Restore the replica count

1
kubectl scale deployment {DEPLOYMENT_NAME} --replicas={REPLICAS_NUM} -n {NAMESPACE}

7. Exposing a Running Service via NodePort

Before the service has started, you can expose it by editing the yaml configuration. Once it is already running, you can expose it with patch.

List services

1
kubectl get service -n {NAMESPACE}

Expose the port for access

1
kubectl patch service {SERVICE_NAME} -p '{"spec":{"type":"NodePort"}}' -n {NAMESPACE}

8. A NodePort Service Is Reachable Only from Certain Nodes

A service exposed through NodePort can be accessed from outside the cluster using any Kubernetes Node IP plus port. kube-proxy forwards the visiting traffic to each Pod in the service in a round-robin fashion.

However, it turned out that not every Node IP plus port is reachable, only the Node running the Pod is.

The reason is that access via an arbitrary Node IP plus port is implemented through inter-host communication. But Docker 1.13 and later changed the iptables rules and disabled FORWARD by default.

Check the iptables rules:

1
2
3
4
5
iptables -L -n

...
Chain FORWARD (policy DROP)
...

Enable FORWARD globally:

1
iptables -P FORWARD ACCEPT

9. Logging into a Container Terminal in Kubernetes

Command format

1
kubectl exec -it {POD_NAME} -c {CONTAINER_NAME} -n {NAMESPACE_NAME} sh

If the Pod has only one Container, the -c {CONTAINER_NAME} argument can be omitted.

Get the Pod name

1
2
3
kubectl get pod -n monitor

monitor       prometheus-alertmanager-5bc4ccf9df-xmt7c         2/2     Running   6          3d23h

Get the Container name

1
2
3
4
kubectl log prometheus-alertmanager-5bc4ccf9df-xmt7c -n monitor

log is DEPRECATED and will be removed in a future version. Use logs instead.
Error from server (BadRequest): a container name must be specified for pod prometheus-alertmanager-5bc4ccf9df-xmt7c, choose one of: [prometheus-alertmanager prometheus-alertmanager-configmap-reload]

In the message, [prometheus-alertmanager prometheus-alertmanager-configmap-reload] is the full list of containers in the Pod.

Log into the Container terminal

kubectl exec -it prometheus-alertmanager-5bc4ccf9df-xmt7c  -c prometheus-alertmanager -n monitor sh

10. Bulk Deleting PVCs

1
2
kubectl get pvc -A | awk '{print $2}' |grep   {KEYWORD} |xargs kubectl delete pvc -n
{NAME_SPACE}

11. Using StorageClass for Dynamic PV Provisioning

The feature of StorageClass dynamic provisioning is that the administrator only needs to create the storage service and the provisioner associated with it, without specifying the size of the PV. When a user needs storage, they only create a PVC and the Provisioner automatically creates a matching PV.

Set up an NFS service on CentOS, reference link.
Install nfs-client-provisioner

helm2

1
helm install --name nfs-client --set nfs.server=x.x.x.x --set nfs.path=/data stable/nfs-client-provisioner

helm3

1
2
helm repo add stable https://charts.helm.sh/stable
helm install nfs-client stable/nfs-client-provisioner --set nfs.server=x.x.x.x --set nfs.path=/data

But nfs-client-provisioner is no longer maintained and does not support newer Kubernetes versions, so the csi-nfs approach is recommended.

List StorageClasses

1
2
3
kubectl get sc

nfs-client        cluster.local/nfs-client-nfs-client-provisioner   18s

Set the DefaultStorageClass

1
kubectl patch storageclass nfs-client -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Create a PVC to test

Create the file pvc.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc1
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 30Gi

Run the command:

1
kubectl create -f pvc.yaml

List the PVC:

1
2
3
4
kubectl get pvc

NAME   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc1   Bound    pvc-24e4dfc1-cb8b-444b-8e3c-36ec8350df3c   30Gi       RWX            nfs-client     2m

You can see that pvc1 is already Bound, and a new folder default-pvc1-pvc-24e4dfc1-cb8b-444b-8e3c-36ec8350df3c appears under the NFS share directory.

12. Deploying csi-nfs

Install cis-driver-nfs

1
2
helm repo add csi-driver-nfs https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts
helm install csi-driver-nfs csi-driver-nfs/csi-driver-nfs --namespace kube-system --version v4.9.0

Create a StorageClass

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-csi-node1
provisioner: nfs.csi.k8s.io
parameters:
  server: x.x.x.x
  share: /data/nfs
reclaimPolicy: Delete
volumeBindingMode: Immediate
mountOptions:
  - nfsvers=4.1

Some public cloud NFS services may not support nfsvers=4.1; you can try removing this parameter.

Create a test PVC

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-csi-node1-test
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 30Gi
  storageClassName: nfs-csi-node1

13. Adding a New Node to a Kubernetes Cluster

When you run kubeadm init, the Console prints the command for adding a Node. The default token validity is 24h. Once it expires you need to create a new token; run the command:

1
2
kubeadm token create --print-join-command
kubeadm join 192.168.10.2:6443 --token ocyzce.3hv8y7w60lrvulir     --discovery-token-ca-cert-hash sha256:7a86632f54de1004bb3f38124b663f837399d6ba9aa803d58c6707a76c02a6cb

Use the command output by the Console to add the Node to the cluster.

14. Controlling Node Scheduling

Allow Pod scheduling

1
kubectl uncordon {NODE_NAME}

Disallow Pod scheduling

1
kubectl cordon {NODE_NAME}

15. Enabling HTTPS for Ingress

Prepare the certificates, domain.com.crt and domain.com.key

Create a Secret

1
kubectl create secret tls {SECRET_NAME} --key domain.com.key --cert domain.com.crt -n {NAMESPACE}

Update the Ingress configuration

1
2
3
4
5
spec:
  tls:
    - hosts:
        - domain.com
      secretName: { SECRET_NAME }

16. Force Deleting Kubernetes Resources

Delete with --force

1
kubectl delete --force --grace-period=0 {RESOURCE_NAME}

Edit finalizers

Deletion usually fails because the action associated with some finalizer did not complete successfully. If you really must delete it, try the following command:

1
kubectl get namespace myns -o json | tr -d "\n" | sed "s/\"finalizers\": \[[^]]\+\]/\"finalizers\": []/"| kubectl replace /api/v1/namespaces/myns/finalize -f -

Delete from etcd

1
yum install -y etcd

1
ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt  --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key del --prefix=/registry/namespaces/{NAMESPACENAME}

17. Kubernetes Service Available Only on the Node Running the Pod

Normally a Service of type NodePort is reachable at any Node IP plus port. But it is also possible that only the IP plus port of the Node carrying the load is reachable.

First, try configuring the forwarding-related parameters:

1
2
3
4
5
6
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward=1
vm.swappiness=0
EOF

1
sysctl --system

Another possibility is the firewall not exempting the IPENCAP protocol:

The Calico network plugin has two modes, BGP and IPIP. When using IPIP mode, the IPENCAP protocol must be enabled in the firewall.

18. Getting All Images in Use with kubectl

1
kubectl  get pod --all-namespaces -o=jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}{end}{range .spec.initContainers[*]}{.image}{"\n"}{end}{end}' | sort -u

19. Calling Kubernetes Remotely with Telepresence

Telepresence is a project under the CNCF foundation. It works by building a transparent two-way proxy between your local machine and the Kubernetes cluster.

What Telepresence lets you do:

A local service can fully reach other services in the remote cluster.
A local service can directly reach the various resources in Kubernetes, including environment variables, Secrets, ConfigMaps, and so on.
The cluster can directly reach the interfaces exposed locally.

Install and configure kubectl.

Locally, install and configure kubectl so that it can access the Kubernetes cluster normally.

Install Telepresence

1
2
brew cask install osxfuse
brew install datawire/blackbird/telepresence

Connect the local side to the remote branch

1
telepresence

Access a remote cluster service locally

Find the service domain name:

1
kubectl get svc

1
2
3
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
kubernetes   ClusterIP   10.233.0.1     <none>        443/TCP    2h
myservice    ClusterIP   10.233.4.163   <none>        8000/TCP   2m

Open any local Console and you can directly access the in-cluster service:

1
curl http://myservice:8000

1
Hello, world!

Access a local service from the remote cluster

To have the remote cluster reach a local service on port 8080 through port 8000, there are two approaches:

Create a new Deployment

1
telepresence --new-deployment new_deploy_name --expose 8080:8000

Replace an existing Deployment

1
telepresence --swap-deployment existed_deploy_name --expose 8080:8000

20. Restarting a Job in Kubernetes

1
kubectl -n {NAMESPACE} get job {JOB_NAME} -o json | jq 'del(.spec.selector)' | jq 'del(.spec.template.metadata.labels)' | kubectl replace --force -f -

If it reports that the jq command is not found, install jq first with yum install -y jq.

21. Quickly Switching Between Kubernetes Environments on macOS

When doing Kubernetes-related development you often need to switch between clusters. Configuring multi-cluster contexts is one option, but if the clusters are constantly being reset, you can try the following approach:

Define a series of related functions in the ~/.profile file, and to switch you only need to run on_cluster_name.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
# Switch Kubernetes Cluster
function switch_kubeconfig(){
   sudo sed -i "" "/$2/d" /etc/hosts
   sudo echo "$1 $2" >> /etc/hosts

   if test -f ~/.ssh/known_hosts; then
     sed -i "" '/kubernetes.default/d'  ~/.ssh/known_hosts
   fi
   sshpass  -p "your_password" ssh -o StrictHostKeyChecking=no root@$1 "cat /etc/kubernetes/admin.conf" > ~/.kube/config
}
function on_dev1(){
   switch_kubeconfig 10.0.0.1 kubernetes.default
}

22. Attaching a Container to a Pod to Debug the Kubernetes Runtime

In a Service Mesh, to minimize disruption to the existing system, the Sidecar pattern is recommended for design and practice. The Sidecar design pattern adds new capabilities to an application without the configuration and code of an additional third-party component.

With a Sidecar you can work with the environment of a running or non-running container to solve certain operational problems.

Step one: add a Sidecar container that shares storage, network, and so on

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
  Containers:
    - name: sidecar
      image: busybox:1.28.4
      command:
        - sleep
        - "3600"
      imagePullPolicy: IfNotPresent
      resources: {}
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
        - name: db-persistent-storage
          mountPath: /var/lib/mysql

Step two: enter the Sidecar container to troubleshoot the operational failure

1
kubectl exec -it {POD_NAME}  -c sidecar sh

23. Viewing Logs with Journalctl

Systemd is a Linux system tool used to start daemons. journald collects the log information produced by the kernel, initrd, services, and so on.

The single command journalctl is all you need to view all logs.

View all logs

1
journalctl

Follow logs in real time

1
journalctl -f

View logs for a specified service

1
journalctl -u kubelet