descheduler Secondary Scheduling Makes Kubernetes Load More Balanced

1. Why Secondary Scheduling Is Needed

The job of the Kubernetes scheduler is to bind a Pod to a single best node. To do this, the scheduler runs through a series of filtering and scoring steps.

Kubernetes scheduling is based on Requests, but the actual usage of each Pod changes dynamically. After running for a while, node load becomes uneven. Some nodes are overloaded, while others have very low utilization.

So we need a mechanism that lets Pods be distributed across the cluster’s nodes in a healthier and more balanced way dynamically, rather than being pinned to one host after a single scheduling decision.

2. The Several Ways descheduler Runs

descheduler is a subproject under kubernetes-sigs. First clone the code locally and enter the project directory:

1
2
git clone https://github.com/kubernetes-sigs/descheduler
cd descheduler

One-off Job

Runs only once.

1
2
3
kubectl create -f kubernetes/base/rbac.yaml
kubectl create -f kubernetes/base/configmap.yaml
kubectl create -f kubernetes/job/job.yaml

CronJob

The default is */2 * * * *, running once every 2 minutes.

1
2
3
kubectl create -f kubernetes/base/rbac.yaml
kubectl create -f kubernetes/base/configmap.yaml
kubectl create -f kubernetes/cronjob/cronjob.yaml

Long-running Deployment

The default is --descheduling-interval 5m, running once every 5 minutes.

kubectl create -f kubernetes/base/rbac.yaml
kubectl create -f kubernetes/base/configmap.yaml
kubectl create -f kubernetes/deployment/deployment.yaml

First generate the policy file locally, then run the descheduler command.

1
descheduler -v=3 --evict-local-storage-pods --policy-config-file=pod-life-time.yml

descheduler has a --help flag to view the related help documentation.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
descheduler --help
The descheduler evicts pods which may be bound to less desired nodes

Usage:
  descheduler [flags]
  descheduler [command]

Available Commands:
  completion  generate the autocompletion script for the specified shell
  help        Help about any command
  version     Version of descheduler

3. Testing the Scheduling Effect

Cordon some nodes so that only one node participates in scheduling

1
2
3
4
5
6
7
kubectl get node

NAME    STATUS                     ROLES                         AGE   VERSION
node2   Ready,SchedulingDisabled   worker                        69d   v1.23.0
node3   Ready                      control-plane,master,worker   85d   v1.23.0
node4   Ready,SchedulingDisabled   worker                        69d   v1.23.0
node5   Ready,SchedulingDisabled   worker                        85d   v1.23.0

Run an application with 40 replicas

You can observe that all replicas of this application are on node3.

kubectl get pod -o wide|grep nginx-645dcf64c8|grep node3|wc -l
      40

Deploy descheduler in the cluster

Here the Deployment approach is used.

1
2
3
kubectl -n kube-system get pod |grep descheduler

descheduler-8446895b76-7vq4q               1/1     Running     0              6m9s

Uncordon the nodes

Before scheduling, all replicas are concentrated on node3.

1
2
3
4
5
6
7
kubectl top node

NAME    CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
node2   218m         6%     3013Mi          43%
node3   527m         14%    4430Mi          62%
node4   168m         4%     2027Mi          28%
node5   93m          15%    785Mi           63%

Uncordon the nodes.

1
2
3
4
5
6
7
kubectl get node

NAME    STATUS   ROLES                         AGE   VERSION
node2   Ready    worker                        69d   v1.23.0
node3   Ready    control-plane,master,worker   85d   v1.23.0
node4   Ready    worker                        69d   v1.23.0
node5   Ready    worker                        85d   v1.23.0

Check the descheduler logs

When the scheduled time requirement is met, descheduler starts evicting Pods according to the policy.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
kubectl -n kube-system logs descheduler-8446895b76-7vq4q  -f

I0610 10:00:26.673573       1 event.go:294] "Event occurred" object="default/nginx-645dcf64c8-z9n8k" fieldPath="" kind="Pod" apiVersion="v1" type="Normal" reason="Descheduled" message="pod evicted by sigs.k8s.io/deschedulerLowNodeUtilization"
I0610 10:00:26.798506       1 evictions.go:163] "Evicted pod" pod="default/nginx-645dcf64c8-2qm5c" reason="RemoveDuplicatePods" strategy="RemoveDuplicatePods" node="node3"
I0610 10:00:26.799245       1 event.go:294] "Event occurred" object="default/nginx-645dcf64c8-2qm5c" fieldPath="" kind="Pod" apiVersion="v1" type="Normal" reason="Descheduled" message="pod evicted by sigs.k8s.io/deschedulerRemoveDuplicatePods"
I0610 10:00:26.893932       1 evictions.go:163] "Evicted pod" pod="default/nginx-645dcf64c8-9ps2g" reason="RemoveDuplicatePods" strategy="RemoveDuplicatePods" node="node3"
I0610 10:00:26.894540       1 event.go:294] "Event occurred" object="default/nginx-645dcf64c8-9ps2g" fieldPath="" kind="Pod" apiVersion="v1" type="Normal" reason="Descheduled" message="pod evicted by sigs.k8s.io/deschedulerRemoveDuplicatePods"
I0610 10:00:26.992410       1 evictions.go:163] "Evicted pod" pod="default/nginx-645dcf64c8-kt7zt" reason="RemoveDuplicatePods" strategy="RemoveDuplicatePods" node="node3"
I0610 10:00:26.993064       1 event.go:294] "Event occurred" object="default/nginx-645dcf64c8-kt7zt" fieldPath="" kind="Pod" apiVersion="v1" type="Normal" reason="Descheduled" message="pod evicted by sigs.k8s.io/deschedulerRemoveDuplicatePods"
I0610 10:00:27.122106       1 evictions.go:163] "Evicted pod" pod="default/nginx-645dcf64c8-lk9pd" reason="RemoveDuplicatePods" strategy="RemoveDuplicatePods" node="node3"
I0610 10:00:27.122776       1 event.go:294] "Event occurred" object="default/nginx-645dcf64c8-lk9pd" fieldPath="" kind="Pod" apiVersion="v1" type="Normal" reason="Descheduled" message="pod evicted by sigs.k8s.io/deschedulerRemoveDuplicatePods"
I0610 10:00:27.225304       1 evictions.go:163] "Evicted pod" pod="default/nginx-645dcf64c8-mztjb" reason="RemoveDuplicatePods" strategy="RemoveDuplicatePods" node="node3"

Pod distribution after secondary scheduling

For node load, node3 decreased while the other nodes rose somewhat.

1
2
3
4
5
6
7
kubectl top node

NAME    CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
node2   300m         8%     3158Mi          45%
node3   450m         12%    3991Mi          56%
node4   190m         5%     2331Mi          32%
node5   111m         18%    910Mi           73%

The distribution of Pods across nodes, in a scenario with no affinity or anti-affinity configured.

Node	Pod count (40 replicas total)
node2	11
node3	10
node4	11
node5	8

The Pod count distribution is very balanced, where node2-4 have the same VM configuration and node5 is lower-spec. The figure below is a diagram of the whole process:

4. descheduler Scheduling Strategies

View the default policy configuration recommended by the official repository:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
cat kubernetes/base/configmap.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: descheduler-policy-configmap
  namespace: kube-system
data:
  policy.yaml: |
    apiVersion: "descheduler/v1alpha1"
    kind: "DeschedulerPolicy"
    strategies:
      "RemoveDuplicates":
         enabled: true
      "RemovePodsViolatingInterPodAntiAffinity":
         enabled: true
      "LowNodeUtilization":
         enabled: true
         params:
           nodeResourceUtilizationThresholds:
             thresholds:
               "cpu" : 20
               "memory": 20
               "pods": 20
             targetThresholds:
               "cpu" : 50
               "memory": 50
               "pods": 50

By default the RemoveDuplicates, RemovePodsViolatingInterPodAntiAffinity, and LowNodeUtilization strategies are enabled. We can configure them according to the needs of the actual scenario.

descheduler currently provides the following scheduling strategies:

RemoveDuplicates

Evicts multiple Pods on the same node.

LowNodeUtilization

Finds low-load nodes and evicts Pods from other nodes.

HighNodeUtilization

Finds high-load nodes and evicts the Pods on them.

RemovePodsViolatingInterPodAntiAffinity

Evicts Pods that violate Pod anti-affinity.

RemovePodsViolatingNodeAffinity

Evicts Pods that violate Node anti-affinity.

RemovePodsViolatingNodeTaints

Pods that violate a NoSchedule taint.

RemovePodsViolatingTopologySpreadConstraint

Evicts Pods that violate the topology domain.

RemovePodsHavingTooManyRestarts

Evicts Pods that have restarted too many times.

PodLifeTime

Evicts Pods that have been running longer than a specified time.

RemoveFailedPods

Evicts Pods in a failed state.

5. What Shortcomings descheduler Has

Calculating node load based on Requests does not reflect the real situation

In the source code at https://github.com/kubernetes-sigs/descheduler/blob/028f205e8ccc49440bd52940eb78a737f8f5b824/pkg/descheduler/node/node.go#L253, you can see that descheduler calculates utilization by summing the Request values of the Pods on a Node.

This approach may not be very suitable for real scenarios. It would be more meaningful to use the data directly from metrics-server or Prometheus, because in many cases the Request and Limit settings are inaccurate. Sometimes, to save cost and increase deployment density, the Request is even set to 50m, or even 10m.

Evicting Pods causes application instability

descheduler calculates a series of Pods that meet the criteria through the policy and evicts them. On the good side, descheduler does not evict Pods without a replica controller, does not evict Pods with local storage, and so on, ensuring that eviction does not cause application failure. But when using client.PolicyV1beta1().Evictions to evict a Pod, it deletes the Pod first and then restarts it, rather than doing a rolling update.

For a brief period, there may be no ready Pod on the cluster, or a new Pod may fail to start due to a fault, and the service will then report errors. There are many detailed parameters that need to be tuned.

It depends on Kubernetes’ scheduling strategy

descheduler does not implement a scheduler itself, but depends on the Kubernetes scheduler. This also means that all descheduler can do is evict Pods and let them go through the scheduling process again. If the number of nodes is very small, descheduler may evict Pods frequently.

6. What Scenarios descheduler Is Suitable For

descheduler’s perspective is about dynamics, which includes two aspects: Node and Pod. The dynamic aspect of Node means when a Node’s labels, taints, configuration, count, etc. change. The dynamic aspect of Pod means the distribution of Pods across Nodes, and so on.

Based on these dynamic characteristics, the following suitable scenarios can be summarized:

A new node is added
After a node restarts
After modifying a node’s topology domain or taints, you want existing Pods to also satisfy the topology domain or taints
Pods are not evenly distributed across different nodes

7. References

https://github.com/kubernetes-sigs/descheduler