Common Commands for Cleaning Up Kubernetes Cluster Resources

Clusters that run for a long time often face various kinds of resource exhaustion. On top of that, when disk space runs low, Kubelet will proactively clean up images on its own, adding another source of uncertainty. This article provides some command snippets for cleanup work.

1. Cleaning Up Basic Kubernetes Objects

Clean up Pods in Evicted state

1
sudo kubectl get pods --all-namespaces -o wide | grep Evicted | awk '{print $1,$2}' | sudo xargs -L1 kubectl delete pod -n

Clean up Pods in Error state

1
sudo kubectl get pods --all-namespaces -o wide | grep Error | awk '{print $1,$2}' | sudo xargs -L1 kubectl delete pod -n

Clean up Pods in Completed state

1
sudo kubectl get pods --all-namespaces -o wide | grep Completed | awk '{print $1,$2}' | sudo xargs -L1 kubectl delete pod -n

Clean up unused PVs

1
sudo kubectl describe -A pvc | grep -E "^Name:.*$|^Namespace:.*$|^Used By:.*$" | grep -B 2 "<none>" | grep -E "^Name:.*$|^Namespace:.*$" | cut -f2 -d: | paste -d " " - - | sudo xargs -n2 bash -c 'kubectl -n ${1} delete pvc ${0}'

Clean up unbound PVCs

1
sudo kubectl get pvc --all-namespaces | tail -n +2 | grep -v Bound | awk '{print $1,$2}' | sudo xargs -L1 kubectl delete pvc -n

Clean up unbound PVs

1
sudo kubectl get pv | tail -n +2 | grep -v Bound | awk '{print $1}' | sudo xargs -L1 kubectl delete pv

2. Linux Cleanup

Check total disk space

1
2
3
4
sudo df -hl /

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       100G   47G   54G  47% /

Check the space used by a given directory

1
2
3
sudo du -sh .

24G	.

Delete folders with a given prefix

1
2
cd /nfsdata
ls | grep archived- |xargs -L1 rm -r

Clean up zombie processes

1
sudo ps -A -ostat,ppid | grep -e '^[Zz]' | awk '{print }' | xargs kill -HUP > /dev/null 2>&1

3. Docker Cleanup

Check disk usage

1
2
3
4
5
6
7
sudo docker system df

TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              361                 23                  178.5GB             173.8GB (97%)
Containers          29                  9                   6.682GB             6.212GB (92%)
Local Volumes       4                   0                   3.139MB             3.139MB (100%)
Build Cache         0                   0                   0B                  0B

Clean up none images

1
sudo docker images | grep none | awk '{print $3}' | sudo xargs docker rmi

Clean up data volumes that are no longer used

1
sudo docker volume rm $(docker volume ls -q)

1
sudo docker volume prune

Clean up build cache

1
sudo docker builder prune

Full cleanup

Deletes stopped containers, unused volumes, unused networks, and dangling images (images with no tag)

1
docker system prune -f

Clean up images matching a regular expression

This cleans up images in the master-8bcf8d7-20211206-111155163 format.

1
sudo docker images |grep -E "([0-9a-z]*[-]){3,}[0-9]{9}" |awk '{print $3}' | sudo xargs docker rmi

4. Setting Up a Schedule

View scheduled tasks

1
sudo crontab -l

Set up a scheduled task

1
sudo crontab -e

Add scheduled tasks in the text

1
2
*/35 */6 * * *  sudo /usr/bin/docker images | grep none | awk '{print $3}' | xargs /usr/bin/docker rmi
45 1 * * * sudo /usr/bin/docker system prune -f

Here the first task runs at the 35th minute of every six hours, and the second task runs at 1:45 every day. Note that the commands must use absolute paths, otherwise they may fail to run.

The cron format

Schedule format: * * * * * shell

First asterisk, minute, values 0-59
Second asterisk, hour, values from 0-23
Third asterisk, day, values from 1-31
Fourth asterisk, month, values from 1-12, or English abbreviations such as Nov, Feb
Fifth asterisk, week, values from 0-6 or English abbreviations such as Wed, Thu, representing the day of the week, where 0 represents the weekend