1. Background

1.1 Problems with the Current Use of Jenkins

• The orchestration engine is unstable

Jenkins is an orchestration engine written in Java, and it stops the world (STW) during a full GC. Under large-scale builds, STW can prevent Jenkins from handling new requests.

• Heavy builds stall

Jenkins stores data in disk files; every pipeline and every build occupies a file directory, producing a huge number of files. The number of pipelines is usually limited, but once builds reach the 10,000+ level, you start to feel the impact of IO on Jenkins.

• Developing plugins is expensive

Although Jenkins already has many plugins, the CICD system still needs to develop plugins of its own to face the company’s myriad large internal systems. Developing a Jenkins plugin requires mastering the Java language and learning Jenkins’ plugin mechanism. Developing a plugin means using Jenkins’ lifecycle as the entry point and extending it. First, based on the feature you want to extend, find the extension class in the Jenkins Packages documentation. Then, in the plugin’s main class, extend that class and implement your own business logic.

• Poor concurrency performance

Due to its own limitations, Jenkins cannot run multiple replicas on Kubernetes. With Kubernetes-based Jenkins, a clear bottleneck appears once build concurrency reaches around 400; going further requires major architectural optimization and upgrading.

1.2 Requirements for CICD

• Cross-network

Services move to the cloud, but the code cannot leave the company. We need to assemble in the cloud while building container images on the internal network.

• Pipelines can be executed at scale

CICD provides a one-shot runtime. CICD is an automation system: the more times it runs, the more human time it saves. In the future, CICD will carry more and more scenarios. Cluster installation, certificate inspection…

• Zero-downtime operations

Previously, orchestration engine maintenance was mostly scheduled in the early morning hours, because every restart of Jenkins takes several minutes, and during that window the CICD system cannot provide service.

• Deliver in less time, iterate continuously

Designing a huge, fully-formed system was not the original intent. We wanted to validate the idea quickly, put it into use, and then keep iterating rapidly to optimize and refine the system.

2. Comparing the Options

2.1 What Makes a Good CICD

A good CICD tool should have the following characteristics:

• Outer DSL simple and easy to grasp - User
• Inner DSL efficient and easy to maintain - Developer
• Ecosystem with many reusable atoms - Ecosystem

UDE lets you score a CICD tool. Below is a comparison of several common CI tools:

• Jenkins

Outer is the Jenkinsfile written in Groovy, Inner is Jenkins written in Java. Neither U nor D is good; Jenkins is hard to maintain, but its huge plugin ecosystem gives E a big boost.

• GitLab CI

Outer is the .gitlab-ci.yml description file written in Yaml, Inner is the parsing engine written in Ruby, with a Runner written in Go. U is very good and quick to pick up — I have written some documentation for GitLab before. D is not great: Ruby’s performance is mediocre and fewer and fewer people know it. E is rather poor: although there are templates similar to Jenkins shared libraries providing atom-level reuse, cross-team reuse is very low, which is bad for building a community ecosystem.

• Tekton

Outer is the PipelineRun description written in Yaml, Inner is the Controller written in Go, continuously executing orchestration flows on Kubernetes Pods. As for plugins, the Tekton community currently provides more than a hundred plugins for reuse.

2.2 Tekton vs Jenkins

In orchestration engine market share surveys of recent years, Jenkins has held over half the usage for many consecutive years — the result of nearly 20 years of accumulation and the siphon effect of being the leader. But after entering the cloud-native era, the infrastructure changed and Jenkins did not keep up well; Jenkins X abandoned Jenkins and switched to Tekton as its default orchestration engine. Building an orchestration engine in-house is too costly, so here we mainly compare Jenkins with Tekton:

3. A Tekton-Based Solution

3.1 What Components Tekton Includes

• Pipeline

The foundational module of CI/CD workflows, used to create tasks and pipelines.

• Triggers

The event trigger for CI/CD workflows, used to automatically trigger pipelines based on events.

• CLI

A command-line tool for managing CICD workflows.

• Dashboard

A general-purpose web management tool for pipelines.

3.2 What a Tekton Pipeline Is Made Of

The image above is a schematic of a Pipeline. A Pipeline usually consists of multiple Tasks, and a Task has its own independent Pod runtime environment. These Tasks run serially or in parallel. Within each Task there are several Steps, and Steps execute serially. A Step has its own independent Container runtime environment.
Below is an example of a pipeline that runs a simple script:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: test-demo
spec:
  pipelineSpec:
    tasks:
      - name: test-1
        taskSpec:
          steps:
            - name: run
              image: alpine
              script: |
                                echo "Hello World."

3.3 Supporting Multi-Cluster Builds

Supporting multiple clusters significantly enhances the scalability and maintainability of CICD. Tekton has abstracted all resources as CRDs, which means that with open-source components such as KubeFed v2, Karmada, and Open Cluster Management, we can very easily distribute pipeline resources across multiple clusters.

As shown above, we create all pipeline resources in the host cluster for metadata management. Then, through an open-source multi-cluster management solution, we distribute resources to different clusters. Each cluster is a separate build environment, which effectively spreads out the load pressure brought by CICD pipelines.

Under the current resource plan, the company’s internal network has very limited server resources, so we need to use cloud resources for assembly as much as possible. The host cluster and some worker clusters are on the public network, while the clusters that execute CI builds must be on the internal network.

This requires network connectivity between the host cluster and the worker clusters. Punching through the network with a tunnel is an operation that harms internal network security, so implementing it with open-source multi-cluster components is not feasible. But we did not stop there; this design gave us inspiration.

3.4 The Implemented Architecture

The image above shows an architecture we have currently implemented and plan to keep optimizing. It is mainly divided into three parts:

• web

Provides the user interface for editing and describing pipelines graphically.

• apiserver

Provides web API endpoints and the endpoint for workers to pull tasks.

• worker

Pulls pipeline tasks for the current cluster, executes them, and pushes the results.

A user creates a pipeline in the web UI; it is saved to the DB through the Apiserver, and a sync event is produced at the same time. Workers poll the Apiserver to pull sync tasks from the message queue, then execute them in the current cluster. After execution finishes, the results and related logs are pushed to the Apiserver and saved to the DB. Ultimately, users can view the execution results directly from the DB on the page.

It is worth noting that a cluster can run multiple worker services, and the only requirement on the cluster is connectivity to the Apiserver, so worker clusters can be brought in for builds from either the public or internal network.

4. Summary and Outlook

4.1 Rich Features, Optimized Performance

Good products are continually polished under the pressure of requirements. We will keep collecting everyone’s requirements for CICD and improving the CICD system.

• Approval feature

Flow control is one of the essential features of CICD. With runAfter, you can control the execution order and dependencies between tasks, but the Tekton community provides no solution for approvals. We have two approaches for approvals, currently being designed and implemented.

• Integrating physical machine builds

Since we currently mainly serve image builds for web and backend projects, we do not yet provide physical machine integration. But we have already considered the approach and are only waiting on user demand.

• Sub-pipelines

Sub-pipelines allow one pipeline to be split into several; different sub-pipelines can execute on different worker clusters, while giving better control over the flow.

• Pipeline cluster management

The current pipeline backend supports multiple clusters, but the frontend does not yet provide a settings entry point. Multi-cluster build support is one of the highlights of this design, and we hope to provide self-service onboarding, self-service management, and self-service use for users as soon as possible.

4.2 A CICD System That Carries More Capabilities

Beyond the concrete design and implementation this time, I also want to talk about my understanding of CICD systems. We usually think of a CICD system as being only for building and releasing. But in fact, CICD provides a kind of runtime, corresponding to Serverless. This runtime can carry many application scenarios, and even replace some SaaS. Here are two scenarios:

• Delivery

Under a Kubernetes cluster, we can use Helm to deliver applications. But how do you deliver Kubernetes itself? How do you deliver services aimed at VMs/bare-metal servers? The answer is pipelines.

Service providers package their respective services as Task plugins and offer them to integrators. Integrators then orchestrate various services through pipelines, providing customers with delivery solutions.

• Automated operations

  1. Incident handling 2. Adding/removing nodes 3. Requesting resources 4. Service changes
     …