1. What Problem We Hit

A Jenkins build log threw an error:

1
2
3
4
5
6

Started by user admin
Lightweight checkout support not available, falling back to full checkout.
Checking out git https://github.com/shaowenchen/pipeline-test.git into /var/jenkins_home/workspace/abc@script to read Jenkinsfile
...
...
Unable to access '.git/index.lock': File exists.

Cause analysis:

To briefly explain the deployment situation: Jenkins is deployed on Kubernetes using a Helm Chart, and builds run on dynamic Kubernetes Pods. Jenkins’ /var/jenkins_home is mounted to a PV for persistence.

Concurrent pipelines preempting the same workspace caused the build to fail. This is a good question, and working through the pipeline execution flow helps deepen your understanding of Jenkins. The fix: you can simply enter the workspace directory and delete index.lock, or you can try to solve the Lightweight failure problem.

2. Common Project Types in Jenkins

First, let’s look at the common project types in Jenkins. There are mainly three:

• Freestyle project

The Freestyle project type offers the ability to edit the flow directly on the Jenkins page.

• Pipeline

The Pipeline type hands the orchestration of the execution part to the user, offering a more flexible way to execute.

• Multibranch Pipeline

The Multibranch Pipeline type provides richer multi-branch use cases.

As shown above, this is the log of a multibranch pipeline scan. You can see that the multi-branch scan consumes a large amount of disk space under /var/jenkins_home/cache, so you should not reduce Jenkins’ disk space just because you are using dynamic Pods and external object storage for archiving.

3. Jenkinsfile Syntax Types

Besides editing directly with a Freestyle project, the Jenkinsfile is also a common way to define a pipeline. Jenkinsfile follows the Groovy DSL, and there are mainly two types of syntax:

• Scripted
• Declarative

3.1 Scripted Pipeline

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

node {
    stage('Build') {
        //
    }
    stage('Test') {
        //
    }
    stage('Deploy') {
        //
    }
}

3.2 Declarative Pipeline

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                sh 'make'
            }
        }
        stage('Test'){
            steps {
                sh 'make check'
                junit 'reports/**/*.xml'
            }
        }
        stage('Deploy') {
            steps {
                sh 'make publish'
            }
        }
    }
}

4. Jenkinsfile Source Types

When using a Jenkinsfile to define a pipeline, there are mainly two ways:

• Pipeline script

As shown above, the pipeline flow is edited directly on the page.

• Pipeline script from SCM

Since configuration is also part of what needs to be managed in DevOps practice, the Jenkinsfile is usually added to the repository and maintained along with the version history. As shown above, you can specify the repository and path directly and select the Jenkinsfile.

The Branches to build and Lightweight checkout settings here deserve special attention, and they are discussed below.

5. The SCM Pipeline Execution Flow

5.1 Fetching the Jenkinsfile from the Repository

If Lightweight checkout is enabled and takes effect, Jenkins fetches the Jenkinsfile directly. There are two specific ways:

• Via the API, where you will see the following log

1
2
3
4

Branch indexing
05:05:03 Connecting to https://api.github.com using /******
Obtained Jenkinsfile from e216c1ce3bed549462c2bb6762b6dbce32f220e6
Running in Durability level: MAX_SURVIVABILITY

• Via Git, where you will see the following log

1
2
3
4
5

Branch indexing
 > git rev-parse --is-inside-work-tree # timeout=10
Setting origin to https://github.com/shaowenchen/pipeline-test.git
 > git config remote.origin.url https://github.com/shaowenchen/pipeline-test.git # timeout=10
Fetching origin...

If Lightweight checkout is not enabled or Lightweight checkout fails, you will see the following log.

1
2
3

Started by user admin
Lightweight checkout support not available, falling back to full checkout.
Checking out git https://github.com/shaowenchen/pipeline-test.git into /var/jenkins_home/workspace/abc@script to read Jenkinsfile

There are many possible reasons for Lightweight checkout to fail:

• An old Jenkins version
• An old Git Server version
• The Branches to build parameter has multiple or incorrect selections

Testing different versions of the components makes it easy to detect the relevant problem. Here we focus on Branches to build.

The Branches to build parameter specifies which branches the Jenkinsfile can be fetched from. If it is set to */master, then only the Jenkinsfile on the master branch is valid. If it is set to */*, it means the Jenkinsfile on all branches is valid. But in that case Jenkins also cannot tell which branch’s pipeline it should request.

Lightweight can only fail, degrading to checking out the entire repository into the /var/jenkins_home/workspaces directory, and using the Jenkinsfile on the branch of the last commit among all branches matched by the regular expression as this run’s pipeline definition.

The problem mentioned at the beginning is obvious: a multibranch pipeline scenario used the pipeline type instead. If the Branches to build configuration causes Lightweight to degrade, and there are many concurrent builds at the same time, SCM pipelines will share a single /var/jenkins_home/workspaces/{pipeline_name}, causing concurrency problems.

When a multibranch pipeline’s Lightweight degrades, it checks out the repository code into the /var/jenkins_home/workspaces/{pipeline_name_branch_name} directory, isolating different branches, which reduces concurrency conflicts.

5.2 Building Based on the Jenkinsfile Content

After obtaining the Jenkinsfile, things are simple: dynamically create a Pod based on the definition.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

Obtained Jenkinsfile from ce0b611f3c443d423dcc499575eb560630625712
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Agent base-zrz3f is provisioned from template Kubernetes Pod Template
---
apiVersion: "v1"
kind: "Pod"
metadata:
  annotations: {}
  labels:
    jenkins: "slave"
    jenkins/base: "true"
  name: "base-zrz3f"
spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - preference:
          matchExpressions:
          - key: "node-role.kubernetes.io/worker"
            operator: "In"
            values:
            - "ci"
        weight: 1

The pipeline build runs on the dynamic Pod.

1

Running on base-zrz3f in /home/jenkins/agent/workspace/hotfix_bugfix_REQ-12956_20200618