Affinity in the Kubernetes Scheduler

1. The Scheduler in Kubernetes

kube-scheduler is the component in Kubernetes that decides which Node a Pending Pod runs on; it is called the scheduler.

Kubernetes ships with a large number of built-in scheduling policies and also provides some advanced scheduling policies (nodeAffinity, podAffinity, and so on) for users, which is basically enough to satisfy the vast majority of business requirements.

The earlier document Labels and Selectors in Kubernetes mentioned that Labels and Selectors are very important features in Kubernetes. Labels connect Pods, Deployments, and Services, and they are also used in scheduling policies. Below we will look at how to use Labels to customize scheduling policies.

2. nodeSelector

First, look at what Labels a Node has:

1
2
3
4
5
6
kubectl get nodes --show-labels

NAME    STATUS   ROLES           AGE   VERSION    LABELS
node1   Ready    master,worker   17h   v1.15.12   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node1,kubernetes.io/os=linux,node-role.kubernetes.io/master=
node2   Ready    worker          17h   v1.15.12   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node2,kubernetes.io/os=linux,node-role.kubernetes.io/worker=
node3   Ready    worker          17h   v1.15.12   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node3,kubernetes.io/os=linux,node-role.kubernetes.io/worker=

Let’s review the basic operations on Labels:

Add a label

1
kubectl label node node1 node-role.kubernetes.io/worker=ci

Modify a label

1
kubectl label --overwrite node1 node-role.kubernetes.io/worker=

Delete a label

1
kubectl label node node1 node-role.kubernetes.io/worker-

When using nodeSelector, add nodeSelector to the Pod’s Spec field, which states all the Label conditions the Node must satisfy at the same time. The example below schedules the Pod onto the Node that has the kubernetes.io/hostname=node1 Label.

spec:
  containers:
  - ...
  nodeSelector:
    kubernetes.io/hostname: node1

After version 1.2, Kubernetes introduced nodeAffinity, which is functionally similar to nodeSelector; nodeSelector will be deprecated in later versions.

3. nodeAffinity

nodeAffinity is mainly used to control which Node a Pod should run on. There are two ways to schedule by affinity:

Soft policy, satisfy if possible
Hard policy, must satisfy

These policies are decided by Label matching, and Kubernetes provides several operators:

In, the Label is in some list
NotIn, the Label is not in some list
Gt, the Label is greater than some value
Lt, the Label is less than some value
Exists, the Label exists
DoesNotExist, the Label does not exist

With these operators and Labels, we can customize our own scheduling policies. Below is an official example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
spec:
  containers:
    - ...
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution: # 硬策略，强制满足
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/e2e-az-name
              operator: In
              values:
                - e2e-az1
                - e2e-az2
    preferredDuringSchedulingIgnoredDuringExecution: # 软策略，尽量满足
      - weight: 1
        preference:
          matchExpressions:
            - key: another-node-label-key
              operator: In
              values:
                - another-node-label-value

Add a nodeAffinity field to the Pod’s Spec to describe this.

If multiple nodeSelectorTerms are specified at the same time, the Node only needs to satisfy one of them to be schedulable. If multiple matchExpressions are specified, the Node must satisfy all of them to be schedulable.

4. podAffinity

podAffinity is similar to nodeAffinity, except that nodeAffinity describes a Pod’s selection of a Node, while podAffinity describes a Pod’s selection of a Pod.

podAffinity has an extra topologyKey (topology domain), which is equivalent to adding a dimension for choosing a Node to the Pod’s scheduling policy. First the Node’s Label must satisfy the topologyKey requirement, and then we examine whether the Labels carried by the running Pods satisfy the affinity requirement.

The example below requires the Pod’s scheduling to satisfy:

The Node’s Label must have failure-domain.beta.kubernetes.io/zone
The Pods running on the Node must have the Label security=S1
Try not to schedule onto a Node whose Label has kubernetes.io/hostname and whose Pod Label has security=S2

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
spec:
  containers:
    - ...
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution: #硬策略，强制满足
        - labelSelector:
            matchExpressions:
              - key: security
                operator: In
                values:
                  - S1
          topologyKey: failure-domain.beta.kubernetes.io/zone
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution: #软策略，尽量满足
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchExpressions:
                - key: security
                  operator: In
                  values:
                    - S2
            topologyKey: kubernetes.io/hostname

5. taints, tolerations

taints target Nodes, and tolerations target Pods. If a Node is marked with a taint, then that Node will not be scheduled to, unless the Pod is given tolerations that tolerate this taint. taints and tolerations are usually used for scheduling onto some special Nodes, such as a master, a Node with GPUs, a Node with SSDs, a Node with a lot of memory, and so on.

The format of a taint is: <key>=<value>:<effect>.

Here key and value (both may be empty) are used for tolerations matching, while effect has three values:

- PreferNoSchedule , try not to schedule
- NoSchedule , cannot schedule
- NoExecute , cannot schedule, and evict existing Pods at the same time

Add a taint to a Node

1
kubectl taint nodes node1 key1=value1:NoSchedule

View a Node’s taint

1
kubectl describe nodes node1

Remove a taint from a Node

1
kubectl taint nodes node1 key1:NoSchedule-

Have a Pod tolerate a taint

In the Pod’s Spec field, add tolerations to describe the taint it tolerates. The example below happens to tolerate the taint applied above:

1
2
3
4
5
6
7
8
spec:
  containers:
    - ...
  tolerations:
    - key: "key1"
      operator: "Equal"
      value: "value1"
      effect: "NoSchedule"