A Universal Pipeline Design

1. Decoupling the Engine to Unlock Pipeline Capabilities

When designing a system, we often face a dilemma. Do we contain the complexity and offer a single, easy-to-use capability to the outside; or do we release the complexity and return flexibility to the user? This is a real test of product ability.

When designing a CICD system, we could simply throw concepts like Jenkinsfile and PipelineRun directly at users, letting them learn the relevant domain knowledge first and then use the product. Of course, we could also keep abstracting and build a model between the person and the system, achieving the conversion of intent into instructions. We want a more usable product, so we chose to hide the underlying concepts and continue abstracting and modeling.

From Jenkins and GitLab CI to GitHub Actions and Tekton, new infrastructure always brings a variety of new building blocks. We want to reduce the cost of switching, to be able to move between engines. Technology keeps changing, but we want to stay consistent for the user.

Although pipeline-related technology evolves quickly, it is ultimately people who carry it out. Human knowledge is inherited: however technology changes, the community that builds pipeline engines is relatively stable and overlapping. This makes it possible to decouple the engine and design a universal pipeline.

2. The Pipeline Data Model

Similar concepts can be found across many CICD engines.

Jenkins

A pipeline contains many Stages, and the steps within a Stage contain multiple scripts executed serially.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
pipeline {
    agent any

    stages {
        stage('Build') {
            steps {
                echo 'Building-1..'
                echo 'Building-2..'
            }
        }
        stage('Test') {
            steps {
                echo 'Testing..'
            }
        }
        stage('Deploy') {
            steps {
                echo 'Deploying....'
            }
        }
    }
}

GitLab CI

The pipeline contains two serial Stages, build and test, and each Stage contains several Jobs executed in parallel.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
stages:
  - build
  - test

build-code-job:
  stage: build
  script:
    - echo "Check the ruby version, then build some Ruby project files:"
    - ruby -v
    - rake

test-code-job1:
  stage: test
  script:
    - echo "If the files are built successfully, test some files with one command:"
    - rake test1

test-code-job2:
  stage: test
  script:
    - echo "If the files are built successfully, test other files with a different command:"
    - rake test2

GitHub Actions

A pipeline is defined by jobs. A pipeline has many possible jobs (constituted by the build job in the example), and each job contains many serial steps.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
name: Octo Organization CI

on:
  push:
    branches: [$default-branch]
  pull_request:
    branches: [$default-branch]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2

      - name: Run a one-line script
        run: echo Hello from Octo Organization

Based on the examples above, we abstract the Pipeline.

1
2
3
4
5
6
7
- pipeline
  - stage-1
  - step-1-1
  - step-1-2
  - stage-2
  - step-2-1
  - ...

As shown in the figure below, a pipeline contains several Stages, which can be parallel or serial. A Stage contains several Steps that execute scripts serially. In Tekton, a Stage corresponds to a Task.

The runtime of a pipeline can be a Kubernetes cluster, a physical machine, a Container environment, and so on.

The runtime of a Stage may be a Pod, a physical machine, a Container environment, and so on.

A Step has a workspace, and then executes a Shell Script.

A pipeline does not need a complex definition; even a few simple scripts can orchestrate complex logic. But abstracting and modeling the pipeline is good for plugin (Step) extensibility and for the development and maintenance of the pipeline product itself.

A pipeline goes through a series of Managers, which associate it with a specific runtime, execution engine, credentials, and so on. Finally it renders the pipeline description that the engine accepts, such as Jenkins’s Jenkinsfile or Tekton’s Yaml.

3. The Code-Level Data Model

The main fields of the core data structures are given below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
// 定义运行时
type Runtime struct {
	Name        string
	Provider    interface{}
}

// 定义引擎
type Engine string

// 定义作用域，分为三个层级。
type Scope string

const (
	ScopePipeline Scope = "Pipeline"
	ScopeStage    Scope = "Stage"
	ScopeStep     Scope = "Step"
)

// 定义参数结构
type ParamSpec struct {
	Name string `json:"name"`
	Scope ParamType `json:"type,omitempty"`
	Default interface{} `json:"default,omitempty"`
}

// 定义插件模板
type Step struct {
	Name        string      `json:"name"`
	Params      []ParamSpec `json:"params,omitempty"`
	Script      string      `json:"script,omitempty"`
  Engine      string      `json:"engine"`
	Workspace   string        `json:"workspace,omitempty"`
}

// 定义组装流水线之后，插件(Step)相关字段
type PipelineStep struct {
	Name    string  `json:"name,omitempty"`
	StepRef string  `json:"StageRef,omitempty"`
	Params  []Param `json:"params,omitempty"`
	Status  string  `json:"status,omitempty"`
	Workspace string  `json:"workspace,omitempty"`
}

// 定义 Stage ，主要是一系列的 Steps
type Stage struct {
	ObjectMeta `json:"metadata"`
	Spec       StageSpec `json:"spec"`
}

type StageSpec struct {
	Description string         `json:"description,omitempty"`
	Params      []ParamSpec    `json:"params,omitempty"`
	Steps       []PipelineStep `json:"steps,omitempty"`
	Workspace   string         `json:"workspace,omitempty"`
}

// 定义组装流水线之后，阶段(Stage)相关字段
type PipelineStage struct {
	Name      string        `json:"name,omitempty"`
	StageRef  Stage         `json:"stageRef,omitempty"`
	Params    []Param       `json:"params,omitempty"`
	Workspace string        `json:"workspace,omitempty"`
	Status    string        `json:"status,omitempty"`
	Runtime   *Runtime
}

// 定义流水线的结构
type Pipeline struct {
	ObjectMeta `json:"metadata"`
	Spec       PipelineSpec `json:"spec"`
}

type PipelineSpec struct {
	Params      []ParamSpec     `json:"params,omitempty"`
	Stages      []PipelineStage `json:"stages,omitempty"`
	Workspace string `json:"workspace,omitempty"`
}

// 定义运行一条流水线相关的字段
type PipelineRun struct {
	ObjectMeta `json:"metadata,omitempty"`
	Spec       PipelineRunSpec `json:"spec,omitempty"`
	Status     string          `json:"status,omitempty"`
}

type PipelineRunSpec struct {
	PipelineRef string         `json:"pipelineRef,omitempty"`
	Params      []Param        `json:"params,omitempty"`
	Runtime     *Runtime
}

At the code level, two things need attention:

Template and instance. A template is a framework or fragment built into the system that relates to the engine, such as Step, Stage, Pipeline; an instance is a template or fragment after personalized parameters are filled in, such as PipelienStep, PipelineStage, PipelineRun.
Scope. Parameters have a field Scope, used to indicate the range in which the parameter is visible. In fact, it is the Step that really uses the parameters, but after assembly the parameters in the Stage scope are promoted into the Stage. Likewise, parameters in the Pipeline scope are promoted into the Pipeline. Parameters serve different purposes at different levels: extracted from the inside out, injected from the outside in.

4. Data and Interaction at Pipeline Runtime

The above is the execution flow of a pipeline; you can follow each step, so it is not repeated here. Below we describe the pipeline’s operation mainly from the perspective of different roles.

4.1 Built-in Step Plugin Templates

First, some commonly used plugin Scripts need to be built into the system.

For example, Jenkins’s Git Clone plugin

1
git(url: '${param.git_repo}', credentialsId: '${param.ssh-key}', branch: '${param.branch}', changelog: true, poll: false)

Building and pushing an image with Jenkins.

1
2
3
4
5
withCredentials([usernamePassword(passwordVariable : 'DOCKER_PASSWORD' ,usernameVariable : 'DOCKER_USERNAME' ,credentialsId : "${param.credential_id}" ,)]) {
          sh 'echo "$DOCKER_PASSWORD" | docker login ${param.registry_server} -u "$DOCKER_USERNAME" --password-stdin'
          sh 'docker push  ${param.image_name}'
          sh 'docker logout'
        }

Running a script with Jenkins.

1
sh '${param.script_content}'

It could of course be a plugin fragment from another engine, but it is mainly a script fragment plus parameter injection, so we will not list more.

4.2 Creating a Pipeline: The User’s Perspective

As shown in the figure above, the user first gets a list of Step templates based on the engine they choose. Then, through orchestration, they assemble the Steps into a pipeline.

Here Step-1, 2, 3 represent instances of selecting a template Step and initializing its parameters. The assembled Pipeline data structure is then stored in the backend.

If creation is done from a template, then it is only necessary to initialize the Pipeline data structure for the user in advance.

4.3 Creating a Pipeline: The Developer’s Perspective

Frontend development

After requesting the Step template list, assemble the Pipeline structure from the instantiation parameters entered by the user.

Backend development

After saving the user’s Pipeline data, render the Pipeline into the pipeline description the engine needs, based on the Step template information. For example, generate a Jenkinsfile and sync it to Jenkins to create the pipeline.

4.4 Runtime: Data Flow and Interaction

Executing a pipeline

The frontend calls the backend API to get the Pipeline definition and pops up a dialog for the Pipeline-level parameters, letting the user enter the relevant values. After clicking confirm, a PipelineRun object is created.

Based on the PipelineRun object, the backend triggers the engine’s execution API, passing the customized parameters from the PipelineRun into the pipeline execution.

Viewing a pipeline

The PipelineRun is the pipeline’s execution history; the pipeline status needs to be queried from the engine and written into the PipelineRun object.

Re-run, pause, resume, approve

The PipelineRun records the full record of a given execution, including the parameters and the Pipeline definition. Therefore, as long as the engine supports the above features, they can all be implemented.