How to Clean Up Zombie Processes

1. What a Zombie Process Is

How a process is created:

The parent process calls fork() to create a child process
The child process calls exec() to load a new program
The child process finishes executing and calls exit() or returns
The parent process calls wait() or waitpid()

If the parent process never calls wait() or waitpid(), the child stays in the system after it finishes, becoming a zombie process.

2. How to Find Zombie Processes

You can use the ps command to find zombie processes:

1
ps -A -ostat,ppid,pid,cmd | grep -e'^[Zz]'

1
2
3
4
Zl   2062969 2068159 [python] <defunct>
Zl   2062969 2074157 [python] <defunct>
ZNl  2062969 2081477 [ray::WorkerDict] <defunct>
ZNl  2062969 2081478 [ray::WorkerDict] <defunct>

The first column is the process state, the second is the parent process ID (PPID), the third is the zombie process ID (PID), and the fourth is the command.

STAT	Meaning
Z	Zombie process
l	Multi-threaded process
N	Low-priority process

3. How to Clean Up Zombie Processes

3.1 Kill the Parent Process Directly

A zombie process cannot be cleaned up on its own; you can only clean it up by cleaning up its parent process.

The simplest approach is to kill the parent process directly:

1
kill -9 <parent_pid>

You can also clean up the parent processes of all zombie processes in bulk:

1
ps -A -ostat,ppid,pid,cmd | grep -e'^[Zz]' |awk '{print $2}' | xargs kill -9

3.2 Resume the Parent Process to Reap the Zombie Process

Sending SIGCONT (kill -18) to the parent process can resume it from a suspended state, after which it continues to wait() for the child to exit normally.

Check whether the parent process is suspended

1
ps -p <parent_pid> -o stat

If the state is T, the process is suspended and can be resumed with SIGCONT.

Send the SIGCONT signal

1
kill -18 <parent_pid>

3.3 Reap the Container the Process Lives In

Find the container the parent process is in

1
export parent_pid=2062969

1
cat /proc/${parent_pid}/cgroup

1
0::/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28df2390_283c_4962_b6f6_4efaafe529ec.slice/cri-containerd-c94382533ede3ae949b06562b8f347a0656a7446403722d4b026256c19e6aea1.scope

Inspect the container information

Extract the substring after containerd to look up the container.

1
nerdctl -n k8s.io ps -a | grep c94382533

1
c94382533ede    docker.io/3257a037b275eb6ca6f75d86f2bc63b9:v1.0                          "/bin/bash -c bash r…"    14 hours ago    Created             k8s://default/elastic-job-261548-edljob-worker-0/main

Remove the container

1
nerdctl -n k8s.io rm -f c94382533ede

Clean up all abnormal containers

1
nerdctl -n k8s.io ps -a | grep -E 'Exited|Created|Dead|Paused' | awk '{print $1}' | xargs -r nerdctl -n k8s.io rm -f

3.4 The Process Is Stuck on Storage

Inspect the process state

1
export parent_pid=1203391

1
cat /proc/${parent_pid}/status

1
2
3
4
5
6
Name:   python3.10
State:  D (disk sleep)
Tgid:   1203391
Ngid:   1203391
Pid:    1203391
PPid:   1197458

Fix the storage problem

In this case, try running df -h to see whether it hangs. If it does, just clean up the mount point.

1
opscli task -f ~/.ops/tasks/clear-mount.yaml

3.5 The Process Is Stuck on a Device Lock

Inspect the process state

1
export parent_pid=2062969

1
cat /proc/${parent_pid}/status

1
2
3
4
5
6
Name:   bash
State:  S (sleeping)
Tgid:   2062969
Ngid:   0
Pid:    2062969
PPid:   2062945

Inspect the process stack

1
cat /proc/${parent_pid}/stack

1
2
3
4
5
6
7
8
[<0>] do_wait+0x1c7/0x230
[<0>] kernel_wait4+0xaf/0x150
[<0>] zap_pid_ns_processes+0x111/0x1b0
[<0>] do_exit+0xa7c/0xac0
[<0>] do_group_exit+0x47/0xb0
[<0>] __x64_sys_exit_group+0x18/0x20
[<0>] do_syscall_64+0x57/0x190
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

Inspect the child process stack

1
ps -ef --forest | grep ${parent_pid}

1
2
3
4
5
root     2062969       1  0 02:02 ?        00:00:01 [bash]
root     2068159 2062969 12 02:03 ?        01:43:37  \_ [python] <defunct>
root     2074157 2062969  0 02:03 ?        00:00:29  \_ [python] <defunct>
root     2081477 2062969 39 02:04 ?        05:32:22  \_ [ray::WorkerDict] <defunct>
root     2081478 2062969 36 02:04 ?        05:11:06  \_ [ray::WorkerDict] <defunct>

1
export child_pid=2068159

1
cat /proc/${child_pid}/stack

1
2
3
[<0>] rwsem_down_write_slowpath+0x244/0x4d0
[<0>] os_acquire_rwlock_write+0x35/0x40 [nvidia]
[<0>] _nv042313rm+0x10/0x40 [nvidia]

The process is trying to acquire a write lock provided by nvidia, but it is blocked.

Find which processes are using the nvidia devices

1
fuser -v /dev/nvidia*

Kill these processes in bulk

1
fuser -v /dev/nvidia* |awk '{for(i=1;i<=NF;i++)print "kill -9 " $i;}' | sh

1
2
3
                     USER        PID ACCESS COMMAND
/dev/nvidiactl:      root      F.... nvidia-smi
                     root      F.... nvidia-smi

The D state (Uninterruptible sleep) means the process is in a kernel-blocked state, waiting for some resource in a way that cannot be interrupted, and cannot be killed. You can upgrade the driver version and then reboot.