How to Run a Pod on a Specified Node

1. Specify a Node via nodeSelector When Creating a Workload

Add a label to the node

1
kubectl label node node2 project=A

Create the workload with the specified nodeSelector

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
cat <<EOF | kubectl apply -f -

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-nodeselector
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-nodeselector
  template:
    metadata:
      labels:
        app: nginx-nodeselector
    spec:
      nodeSelector:
        project: A
      containers:
      - name: nginx
        image: nginx
EOF

Check the workload

1
2
3
4
kubectl get pod  -o wide

NAME                                  READY   STATUS    RESTARTS   AGE   IP              NODE    NOMINATED NODE   READINESS GATES
nginx-nodeselector-7bb75b7687-7r5xk   1/1     Running   0          19s   10.233.96.60    node2   <none>           <none>

As expected, the Pod runs on the specified node node2.

Clean up the environment

1
2
kubectl delete deployments nginx-nodeselector
kubectl label node node2 project-

In fact, there is another node selection parameter, nodeName, which directly specifies the node name. But this setting is too rigid, and it bypasses Kubernetes’ own scheduling mechanism, so it is rarely used in production.

2. Bind a Namespace to a Node via Admission Control

Specifying a nodeSelector when creating a workload lets you set the node a Pod runs on. But if you want to bind all Pods under a namespace to run on a specified node, that approach falls short. Using the kube-apiserver’s admission control can achieve this goal — a feature that entered the alpha stage back in Kubernetes 1.5.

2.1 Modify kube-apiserver Parameters

Edit the kube-apiserver file:

1
vim /etc/kubernetes/manifests/kube-apiserver.yaml

Add PodNodeSelector to admission-plugins:

1
- --enable-admission-plugins=NodeRestriction,PodNodeSelector

Here NodeRestriction is enabled by default. For a highly available cluster, you need to modify every kube-apiserver. After the modification, wait a moment for kube-apiserver to finish restarting.

2.2 Add an Annotation to the Namespace

Edit the namespace and add an annotation:

1
kubectl edit ns default

1
2
3
4
5
6
apiVersion: v1
kind: Namespace
metadata:
  name: default
  annotations:
    scheduler.alpha.kubernetes.io/node-selector: project=A

scheduler.alpha.kubernetes.io/node-selector can be a node name or a label key-value pair.

2.3 Add the Specified Label to the Node

Label the node3 node with project=A:

1
kubectl label node node3 project=A

This binds the workloads in the default namespace to the node node3.

2.4 Create a Workload

Create a workload for testing

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
cat <<EOF | kubectl apply -f -

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-scheduler
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx-scheduler
  template:
    metadata:
      labels:
        app: nginx-scheduler
    spec:
      containers:
      - name: nginx
        image: nginx
EOF

Check the workload distribution

1
2
3
4
5
6
kubectl get pod -o wide

NAME                               READY   STATUS    RESTARTS   AGE   IP             NODE    NOMINATED NODE   READINESS GATES
nginx-scheduler-6478998698-brkzn   1/1     Running   0          84s   10.233.92.52   node3   <none>           <none>
nginx-scheduler-6478998698-m422x   1/1     Running   0          84s   10.233.92.51   node3   <none>           <none>
nginx-scheduler-6478998698-mnf4d   1/1     Running   0          84s   10.233.92.50   node3   <none>           <none>

As you can see, although the cluster has 4 available nodes, all workloads in the default namespace run under the node3 node.

2.5 Clean Up the Environment

Clean up the label

1
kubectl label node node3 project-

Clean up the workload

1
kubectl delete deployments nginx-scheduler

Clean up the annotation

kubectl edit ns default

Note that if a namespace already has scheduler.alpha.kubernetes.io/node-selector enabled but the nodes do not have the relevant label, then the Pod will remain in the Pending state and cannot be scheduled until a Node matching the label appears.

3. Group Nodes Using Topology Domains

As shown in the figure below, with the kube-apiserver’s admission control plugin we can establish a model in which each project has one namespace, and each namespace contains specified nodes. This can satisfy the requirements of business isolation and cost accounting. But as a cluster grows larger, a project needs to divide several availability zones within the cluster to guarantee business availability.

Topology domains mainly solve the problem of Pod distribution across a cluster, and can be used to implement the requirement of Pods targeting specific nodes. The topology domain feature of the Kubernetes cluster scheduler entered the Alpha stage in 1.16 and the Beta stage in 1.18. Let us run some experiments below:

Divide nodes into different topology domains

Here we assign node2 to zone a, and node3 and node4 to zone b.

1
kubectl label node node2 zone=a

1
kubectl label node node3 node4 zone=b

Create a workload

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
cat <<EOF | kubectl apply -f -

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-topology
spec:
  replicas: 20
  selector:
    matchLabels:
      app: nginx-topology
  template:
    metadata:
      labels:
        app: nginx-topology
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: nginx-topology
      containers:
      - name: nginx
        image: nginx
EOF

Here topologyKey specifies the Key used to divide topology domains, maxSkew means the difference in the number of Pods between zone=a and zone=b cannot exceed 1, and whenUnsatisfiable: DoNotSchedule means that when the condition is not met, no scheduling is performed.

Check the Pod distribution

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
kubectl get pod  -o wide

NAME                              READY   STATUS    RESTARTS   AGE    IP              NODE    NOMINATED NODE   READINESS GATES
nginx-topology-7d8698544d-2srcj   1/1     Running   0          3m3s   10.233.92.63    node3   <none>           <none>
nginx-topology-7d8698544d-2wxkp   1/1     Running   0          3m3s   10.233.96.53    node2   <none>           <none>
nginx-topology-7d8698544d-4db5b   1/1     Running   0          3m3s   10.233.105.43   node4   <none>           <none>
nginx-topology-7d8698544d-9tqvn   1/1     Running   0          3m3s   10.233.96.58    node2   <none>           <none>
nginx-topology-7d8698544d-9zll5   1/1     Running   0          3m3s   10.233.105.45   node4   <none>           <none>
nginx-topology-7d8698544d-d6nbm   1/1     Running   0          3m3s   10.233.105.44   node4   <none>           <none>
nginx-topology-7d8698544d-f4nw9   1/1     Running   0          3m3s   10.233.96.54    node2   <none>           <none>
nginx-topology-7d8698544d-ggfgv   1/1     Running   0          3m3s   10.233.92.66    node3   <none>           <none>
nginx-topology-7d8698544d-gj4pg   1/1     Running   0          3m3s   10.233.92.61    node3   <none>           <none>
nginx-topology-7d8698544d-jc2xt   1/1     Running   0          3m3s   10.233.92.62    node3   <none>           <none>
nginx-topology-7d8698544d-jmmcx   1/1     Running   0          3m3s   10.233.96.56    node2   <none>           <none>
nginx-topology-7d8698544d-l45qj   1/1     Running   0          3m3s   10.233.92.65    node3   <none>           <none>
nginx-topology-7d8698544d-lwp8m   1/1     Running   0          3m3s   10.233.92.64    node3   <none>           <none>
nginx-topology-7d8698544d-m65rx   1/1     Running   0          3m3s   10.233.96.57    node2   <none>           <none>
nginx-topology-7d8698544d-pzrzs   1/1     Running   0          3m3s   10.233.96.55    node2   <none>           <none>
nginx-topology-7d8698544d-tslxx   1/1     Running   0          3m3s   10.233.92.60    node3   <none>           <none>
nginx-topology-7d8698544d-v4cqx   1/1     Running   0          3m3s   10.233.96.50    node2   <none>           <none>
nginx-topology-7d8698544d-w4r86   1/1     Running   0          3m3s   10.233.96.52    node2   <none>           <none>
nginx-topology-7d8698544d-wwn95   1/1     Running   0          3m3s   10.233.96.51    node2   <none>           <none>
nginx-topology-7d8698544d-xffpx   1/1     Running   0          3m3s   10.233.96.59    node2   <none>           <none>

Among them, the node2 node has 10 Pods, the node3 node has 7, and the node4 node has 3. As you can see, the Pods are evenly distributed across zone=a and zone=b.

Clean up the environment

1
2
kubectl label node node2 node3 node4 zone-
kubectl delete deployments nginx-topology

4. Summary

As a cluster grows larger, problems such as isolation between businesses and exclusive use of nodes by a business begin to surface. Usually, each business has its own separate namespace, so we can bind the namespace to nodes.

This article mainly presents two methods. One is to set nodeSelector directly when creating a workload — a clever trick is to use the namespace value as the value. The other is to rely on the admission control plugin provided by kube-apiserver and, by means of annotations, filter the specified nodes by label when creating workloads in the namespace, thereby completing the binding between the namespace and the nodes.

Thinking further, if the number of nodes is very large and availability zones need to be divided to spread the load, then we can rely on topology domains to achieve this. Through topology domains, we can make workloads distribute evenly across the specified availability zones and racks according to the configured policy.