Tekton Optimization: A Custom Cluster Scheduler

1. A Constrained Build Environment Cannot Meet Build Requirements

Tekton is a CICD engine based on a Kubernetes cluster, and it is more cloud-native than Jenkins. In plain terms, that means it is easier to develop plugins for, easier to scale, easier to observe, and more fun.

Because code can only be stored on the company intranet, the build cluster can only be deployed on the office intranet. This led to a lot of limitations:

Hardware resources: no elastic scaling capability
Network restrictions: accessing github.com, docker.io, and dl-cdn.alpinelinux.org is very slow
Reliability restrictions: the data center’s stable environment cannot be guaranteed, and the hardware failure rate is high
Operations restrictions: maintaining the system requires first connecting to the company intranet

But these are all CICD R&D matters that nobody pays attention to. I just had to bear it silently and find a way to solve it — who told me to be a wage earner.

Until the business R&D team, unable to take it any longer, started publicly complaining: “Why is the CDN upload task so slow? It never used to be like this.” That was because I had not joined the company back then; once I arrived, it had long since slowed down.

There was no way around it — the business R&D team are the users of the build system, and as a platform developer I could only think of another approach, so I looked into a cloud connection solution. For 10k RMB a month, you could connect to Huawei Cloud’s overseas VPN dedicated line, with a direct connection overseas.

But paying money is something the business side would never agree to. When it comes to programmers’ needs, why would they be willing to spend money?

2. A Cross-Region Kubernetes Build Cluster

Interestingly, the company I work for has offices in multiple regions. The intranets of the various regions are interconnected, but their egress networks are different. One of the regions has many overseas business lines, and its egress network quality is exceptionally good.

This kind of network environment happens to be perfect for building a cross-region Kubernetes cluster for builds. As shown below:

In the region with good overseas access quality, add several build nodes.

As shown in the figure above, we plan to schedule CICD tasks that access overseas resources onto the Good to Google nodes for execution. Therefore, we need a Kubernetes cluster scheduler that can customize scheduling policies based on different tasks.

3. Several Common Ways to Extend a Scheduler

default-scheduler[1]

Directly hard-code modifications on top of the scheduler source code, then recompile kube-scheduler and replace the original kube-scheduler.

custom scheduler

When creating a resource, you can set the spec.schedulerName field to specify which scheduler handles it. In this way, multiple schedulers coexist in one cluster, and each scheduler has a different scheduler name.

scheduler extender[2]

As shown in the figure above, the scheduler extender provides several extension points. When the kube-scheduler scheduling flow enters that extension stage, it sends an HTTP request to the scheduler extender to handle custom logic.

When deploying, you can use a Deployment to deploy the scheduler extender, and simply modify the kube-scheduler startup parameters to point at the scheduler extender’s address.

scheduler framework

As shown in the figure above, the scheduler framework also provides several extension points; depending on the stage being handled, you implement the corresponding interface.

When deploying, you simply replace the kube-scheduler image and add a configuration file stating which plugins to enable.

4. A Custom Cluster Scheduler

Of the four approaches above, the first is too hardcore with its hard-coding, and the second suits multi-tenant scenarios. Both the third and fourth are based on extension points, but the third requires deploying an extra component, while the fourth directly replaces the kube-scheduler image.

The third approach needs to make HTTP requests during scheduling and create a cluster Client to maintain an Informer; the fourth should be more efficient, and it is newer with more of a future. There are plenty of detailed tutorials online, so here I mainly record the pitfalls I ran into.

4.1 How to Create a New Project

Open https://github.com/kubernetes-sigs/scheduler-plugins/ and switch to the tag corresponding to your cluster. The build cluster is Kubernetes 1.21, so choose v0.21.6.

This project has many plugins you can reference; you can use them directly, or modify and customize them according to your own needs.

Copy the replace section of the go.mod file and create your own Go project.

The purpose of doing this is:

To keep the scheduler code’s dependency versions consistent with the cluster version; otherwise, spanning too many versions will cause parameter incompatibility issues
To avoid dependency errors during compilation, which are very time-consuming to deal with

4.2 Quick Design

How to direct namespaces and tasks to specific nodes

1
2
3
kubectl annotate namespace kube-system com.chenshaowen.scheduler-plugin.filterimage=cdn
kubectl annotate namespace kube-system com.chenshaowen.scheduler-plugin.nodes=node2
kubectl annotate namespace kube-system com.chenshaowen.scheduler-plugin.ns=default

Since this is global configuration, the configuration information is written directly into the annotation of a certain namespace, using it as a database.

Whether a task is a CDN task is identified through the Pod’s image. But not every project’s CDN task needs to go through an overseas node; there are also CDN tasks that upload domestically.

With the data above, the final effect should be: within the default namespace, allow Pods whose image contains the string cdn to be preferentially scheduled to node2; forbid Pods whose image does not contain the string cdn from being scheduled to node2; and forbid other namespaces from being scheduled to node2.

Which extensions are needed

Filter and Score, these two extension points, are enough. If you are unsure which extensions to use, you can directly look at the Plugin interface definitions to check the interface parameters and return values.

Filter needs to forbid Pods from non-specified namespaces from being scheduled to the specified node, while letting through the scheduling of special Pods from the specified namespace.

Score needs to preferentially schedule special Pods from the specified namespace to the specified node.

main

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
func main() {
	rand.Seed(time.Now().UnixNano())
	command := app.NewSchedulerCommand(
		app.WithPlugin(image.Name, image.New),
	)
	if err := command.Execute(); err != nil {
		println(os.Stderr, "%v\n", err)
		os.Exit(1)
	}
}

main starts up using the default-scheduler code, then registers its own plugin; you can register multiple.

Define the plugin ImageNode

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
type ImageNode struct {
	handle framework.Handle
}

// var _ = framework.PreFilterPlugin(&ImageNode{})
var _ = framework.FilterPlugin(&ImageNode{})
var _ = framework.ScorePlugin(&ImageNode{})

const Name = "ImageNode"

func (i *ImageNode) Name() string {
	return Name
}

func New(_ runtime.Object, h framework.Handle) (framework.Plugin, error) {
	return &ImageNode{
		handle: h,
	}, nil
}

This format is fixed; var _ = framework.FilterPlugin(&ImageNode{}) is there to ensure ImageNode must implement the FilterPlugin interface, otherwise it will not compile. Here you need to implement the FilterPlugin and ScorePlugin interfaces.

Implement the FilterPlugin interface

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
func (i *ImageNode) Filter(ctx context.Context, state *framework.CycleState, pod *v1.Pod, nodeInfo *framework.NodeInfo) *framework.Status {
	if pod == nil {
		return framework.NewStatus(framework.Error, "pod is nil")
	}
	node := nodeInfo.Node()
	if node == nil {
		klog.Infof("node is nil")
		return framework.NewStatus(framework.Error, "node is nil")
	}
	nodes, nss, filterImage := isSpecialNS(i.handle.ClientSet(), pod.Namespace)

	if len(nodes) == 0 || len(nss) == 0 || len(filterImage) == 0 {
		klog.Infof("nodes, nss, filterImage is nil")
		return framework.NewStatus(framework.Success, "default")
	}

	workload := ""
	if len(pod.ObjectMeta.OwnerReferences) > 0 {
		workload = pod.ObjectMeta.OwnerReferences[0].Kind
	}
	if workload == "DaemonSet" {
		klog.Info("DaemonSet pass")
		return framework.NewStatus(framework.Success, "DaemonSet pass")
	}

	if isStringInList(node.Name, nodes) {
		if isStringInList(pod.Namespace, nss) && isSpecialImage(pod, filterImage) {
			klog.Infof("plugin hit")
			return framework.NewStatus(framework.Success, "plugin hit")
		} else {
			klog.Info(fmt.Printf("plugin disable pod %s special node %s", pod.Name, node.Name))
			return framework.NewStatus(framework.Unschedulable, "plugin disable special node")
		}
	}
	klog.Info("default pass")
	return framework.NewStatus(framework.Success, "default")
}

There is a special piece of logic here: let DaemonSet Pods through, otherwise DaemonSet Pods would be unable to run on the marked nodes.

Implement the ScorePlugin interface

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
func (i *ImageNode) Score(ctx context.Context, cycleState *framework.CycleState, pod *v1.Pod, nodeName string) (int64, *framework.Status) {
	nodes, nss, filterImage := isSpecialNS(i.handle.ClientSet(), pod.Namespace)
	if len(nodes) > 0 && len(filterImage) > 0 && len(nss) > 0 {
		if isStringInList(nodeName, nodes) && isSpecialImage(pod, filterImage) {
			retScore := framework.MaxNodeScore - rand.Int63n(10)
			klog.Infof("special node score %d", retScore)
			return retScore, framework.NewStatus(framework.Success, "special node score")
		}
	}
	retScore := rand.Int63n(framework.MaxNodeScore - 50)
	klog.Infof("rand node score %d", retScore)
	return retScore, framework.NewStatus(framework.Success, "rand node score")
}

func (i *ImageNode) ScoreExtensions() framework.ScoreExtensions {
	return i
}

func (i *ImageNode) NormalizeScore(ctx context.Context, state *framework.CycleState, pod *v1.Pod, scores framework.NodeScoreList) *framework.Status {
	// Find highest and lowest scores.
	var highest int64 = -math.MaxInt64
	var lowest int64 = math.MaxInt64
	for _, nodeScore := range scores {
		if nodeScore.Score > highest {
			highest = nodeScore.Score
		}
		if nodeScore.Score < lowest {
			lowest = nodeScore.Score
		}
	}

	// Transform the highest to lowest score range to fit the framework's min to max node score range.
	oldRange := highest - lowest
	newRange := framework.MaxNodeScore - framework.MinNodeScore
	for i, nodeScore := range scores {
		if oldRange == 0 {
			scores[i].Score = framework.MinNodeScore
		} else {
			scores[i].Score = ((nodeScore.Score - lowest) * newRange / oldRange) + framework.MinNodeScore
		}
	}

	return nil
}

Score is simply scoring the nodes; the higher the score, the higher the scheduling priority. The handling here is rather crude, because the requests of CICD tasks are all small, and default-scheduler’s scoring is inherently inaccurate, so it just hands out random scores.

To avoid CDN tasks always being scheduled to the same marked node, a certain amount of fluctuation is introduced, randomly subtracting a certain amount from the nodes’ scores.

4.4 Debugging and Deployment

This is where a beginner spends the most time when getting hands-on. Reading one or two documents makes it easy to understand, but in actual practice you often run into all sorts of problems.

Create a local file scheduler-plugin.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
apiVersion: kubescheduler.config.k8s.io/v1beta1
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: "/Users/shaowenchen/.kube/config"
profiles:
  - schedulerName: default-scheduler
    plugins:
      filter:
        enabled:
          - name: ImageNode
      score:
        enabled:
          - name: ImageNode

apiVersion needs to be adjusted according to the cluster version. kubeconfig points to the local kubeconfig file. Both filter and score need the ImageNode plugin enabled.

Start kube-scheduler

1
2
3
4
5
6
7
8
go run main.go \
    --leader-elect=true \
    --feature-gates=RotateKubeletServerCertificate=true,TTLAfterFinished=true,ExpandCSIVolumes=true,CSIStorageCapacity=true \
    --authentication-kubeconfig=/Users/shaowenchen/.kube/config \
    --authorization-kubeconfig=/Users/shaowenchen/.kube/config \
    --kubeconfig=/Users/shaowenchen/.kube/config \
    --config=/Volumes/Data/Code/Github/demo/scheduler-plugin/scheduler-plugin.yaml \
    --v=5

The startup parameters may differ across cluster versions; you need to check them on the cluster. Adding --v=5 lets you see more detailed logs.

Build the image

The Dockerfile is as follows

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
FROM golang:1.19 AS build

WORKDIR /go/src/kube-scheduler
COPY . .

RUN CGO_ENABLED=0 GOOS=linux go build -o kube-scheduler .

FROM alpine:3.14

COPY --from=build /go/src/kube-scheduler/kube-scheduler /usr/bin/kube-scheduler

CMD ["/usr/bin/kube-scheduler"]

Run the build command

1
docker build -t shaowenchen/scheduler-plugin:latest .

Deploy [on every Master node]

Create the file /etc/kubernetes/scheduler-plugin.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
apiVersion: kubescheduler.config.k8s.io/v1beta1
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: "/etc/kubernetes/scheduler.conf"
profiles:
  - schedulerName: default-scheduler
    plugins:
      filter:
        enabled:
          - name: ImageNode
      score:
        enabled:
          - name: ImageNode

At this point the kubeconfig should point to the cluster’s kubeconfig file.

Edit /etc/kubernetes/manifests/kube-scheduler.yaml and add the following content

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
apiVersion: v1
kind: Pod
metadata:
  name: kube-scheduler
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-scheduler
    ...
    - --config=/etc/kubernetes/scheduler-plugin.yaml
    image: shaowenchen/scheduler-plugin:latest
    imagePullPolicy: Always
    volumeMounts:
    - mountPath: /etc/kubernetes/scheduler-plugin.yaml
      name: scheduler-plugin
      readOnly: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/scheduler-plugin.yaml
      type: File
    name: scheduler-plugin

--config=/etc/kubernetes/scheduler-plugin.yaml specifies the configuration file, and volumeMounts and volumes are used to mount the configuration file.

Note that kube-scheduler is a static pod, so you need to edit the file to actually restart it. Using kubectl -n kube-system delete pod kube-scheduler-master1 only restarts the container and will not use the latest image.

5. Summary

This article optimizes a Tekton-based CICD system from the perspective of Kubernetes cluster scheduling, scheduling specified tasks onto specified nodes to make better use of existing network resources.

The main content is as follows:

Given the business requirements, we planned to use nodes in multiple regions for builds, so we needed a custom scheduler
We surveyed four common schedulers and ultimately chose the scheduler framework extension approach
Using the scheduler framework approach, we implemented a simple scheduler plugin
Local debugging and deployment

But there is still much more that can be customized here, for example:

Choosing different nodes based on task priority
Choosing different nodes based on host load
Choosing different nodes based on task resource requirements
…

This further expands the room for customization and optimization available to a CICD system.