How to Upgrade the cgroup Version of a Kubernetes Node

1. Interface Path Differences Between cgroup v1 and v2

1
2
3
4
/sys/fs/cgroup/cpu/cpu.cfs_quota_us
/sys/fs/cgroup/cpu/cpuacct.usage
/sys/fs/cgroup/memory/memory.limit_in_bytes
/sys/fs/cgroup/memory/memory.usage_in_bytes

1
2
3
4
/sys/fs/cgroup/cpu.max
/sys/fs/cgroup/cpu.stat
/sys/fs/cgroup/memory.max
/sys/fs/cgroup/memory.current

cgroup v2 is the upgraded version of v1, with advantages such as more unified resource hierarchy management and precise resource isolation. But it also means that when writing code, the paths for reading the relevant interface files are different, so compatibility handling is required. Another approach is to unify on a single cgroup version.

Kubernetes supports cgroup v2 by default, but which cgroup a Pod actually uses depends on the cgroup type mounted by the host kernel.

2. Checking the cgroup Configuration of All Nodes

Find a DaemonSet and check the cgroup configuration of the Pods on all nodes.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
for pod in $(kubectl -n monitoring get pod -l k8s-app=node-exporter -o jsonpath='{.items[*].metadata.name}'); do
  node=$(kubectl -n monitoring get pod $pod -o jsonpath='{.spec.nodeName}')
  runtime=$(kubectl get node $node -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}')
  os=$(kubectl get node $node -o jsonpath='{.status.nodeInfo.osImage}')

  echo "===== Pod: $pod | Node: $node | Runtime: $runtime | OS: $os ====="

  if kubectl -n monitoring exec $pod -- test -f /sys/fs/cgroup/cpu.max 2>/dev/null; then
    echo "cgroup v2"
    kubectl -n monitoring exec $pod -- cat /sys/fs/cgroup/cpu.max
  else
    echo "cgroup v1"
    if kubectl -n monitoring exec $pod -- test -f /sys/fs/cgroup/cpu/cpu.cfs_quota_us 2>/dev/null; then
      kubectl -n monitoring exec $pod -- cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
    else
      echo "cpu.cfs_quota_us not found"
    fi
  fi
done

I found that Ubuntu 20 uses cgroup v1 by default, while Ubuntu 22 uses cgroup v2 by default.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
===== Pod: node-exporter-2d7q2 | Node: node-4090-76 | Runtime: containerd://1.6.31 | OS: Ubuntu 20.04.1 LTS =====
cgroup v1
-1
===== Pod: node-exporter-2z9ts | Node: node-4090-53 | Runtime: containerd://1.6.31 | OS: Ubuntu 20.04.1 LTS =====
cgroup v1
-1
===== Pod: node-exporter-462mh | Node: node-4090-ywd-04 | Runtime: containerd://1.6.31 | OS: Ubuntu 22.04.2 LTS =====
cgroup v2
max 100000
===== Pod: node-exporter-4bskd | Node: k8s-cpu-2 | Runtime: containerd://1.6.31 | OS: Ubuntu 22.04.3 LTS =====
cgroup v2
max 100000
===== Pod: node-exporter-4kz4s | Node: node-a800-52 | Runtime: containerd://1.6.31 | OS: Ubuntu 20.04.1 LTS =====
cgroup v1
-1
===== Pod: node-exporter-4lvsm | Node: node-4090-d2r1-25 | Runtime: containerd://1.6.31 | OS: Ubuntu 20.04.1 LTS =====
cgroup v1
-1
===== Pod: node-exporter-592wh | Node: node-a800-gc-09 | Runtime: containerd://1.6.31 | OS: Ubuntu 20.04.1 LTS =====
cgroup v1

3. Upgrading a Node’s cgroup Version

Although cgroup v2 was introduced as early as 4.5, it is recommended that production use be considered only on 5.4 and later.

Check the cgroup version in use

1
stat -fc %T /sys/fs/cgroup/

An output of cgroup2fs means cgroup v2, and an output of tmpfs means cgroup v1.

Check the kernel version

1
uname -r

1
5.4.0-169-generic

Edit /etc/default/grub

1
vim /etc/default/grub

Add the parameter systemd.unified_cgroup_hierarchy=1 inside the quotes on the GRUB_CMDLINE_LINUX line.

Update the grub configuration

1
update-grub

Reboot the system

1
reboot

4. Checking the Containerd cgroup Configuration

1
cat /etc/containerd/config.toml | grep SystemdCgroup

1
            SystemdCgroup = true

When SystemdCgroup is set to true, Containerd uses cgroup v2.

5. Checking the Kubelet cgroup Configuration

1
cat /var/lib/kubelet/config.yaml |grep cgroupDriver

1
cgroupDriver: systemd

When cgroupDriver is set to systemd, Kubelet uses cgroup v2.

6. Problems Encountered

Kernel version too low, cgroup v2 cannot be enabled

1
Warning Failed 6s (x5 over 91s) kubelet Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: open /sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd427dc62_807d_4218_b638_ddf5a6c6f7e7.slice/cri-containerd-node-exporter.scope/cpuset.cpus: no such file or directory: unknown

I hit this error on a host running kernel 4.15; cleaning up Pods and wiping Containerd data both failed to resolve it.