Building an Automated Test System with Kubernetes and Jenkins

1. Test Layering

The purpose of testing is to verify expected behavior and uncover latent defects. Testing strengthens confidence that a qualified product can be delivered, and it is what makes agile iteration possible. You could say that testing determines the development pace of a product.

The network model has the seven-layer OSI and the four-layer TCP, while development patterns include MTV, MVC, MVP, MVVM, and so on. High cohesion, low coupling; dividing responsibilities, splitting into modules, and layering. Then structuring and standardization, as the technology gradually matures.

Testing, too, is divided into UI testing, API testing, and unit testing. Testing is not a new technology; it is more a balance between output and cost.

As shown above, this is a test pyramid. The higher you go, the higher the cost, the higher the demands on the environment, the longer the execution time, and the more troublesome the maintenance — but the closer you get to the end user’s scenario. In How Google Tests Software, according to Google’s experience, the ratio of test cases at each layer is 70:20:10, that is, 70% unit tests, 20% API tests, and 10% UI tests.

This post is mainly about how to use Jenkins to run automated tests on a Kubernetes cluster. Previous articles have already touched on parts of testing, such as unit tests, agile development, and Robotframework, so I will not elaborate on those here.

2. Unit Testing

Unit tests run very frequently, and every code commit should trigger a run. Unit tests have few dependencies and usually only need a container runtime.

Below is an example of running unit tests with golang:latest.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
pipeline {
  agent {
    kubernetes {
      label 'unittest'
      yaml '''apiVersion: v1
kind: Pod
spec:
  containers:
  - name: golang
    image: golang:latest
    command: [\'cat\']
    tty: true
    volumeMounts:
    - name: dockersock
      mountPath: /var/run/docker.sock
    - name: dockerbin
      mountPath: /usr/bin/docker
  volumes:
  - name: dockersock
    hostPath:
      path: /var/run/docker.sock
  - name: dockerbin
    hostPath:
      path: /usr/bin/docker
      '''
      defaultContainer 'golang'
    }
  }

  stages {
    stage('testing') {
      steps {
          sh '''
          git clone https://github.com/etcd-io/etcd.git
          cd etcd
          make test
          '''
      }
    }
  }
}

Execution log:

For other languages and frameworks, unit tests can also be run conveniently on Kubernetes by installing a few packages and mocking the relevant services. What is more worth exploring is the technique of writing unit tests, not the runtime or the unit-testing approach.

3. API Testing

If a team’s automated testing is just getting started, API automated testing is a very good entry point.

Unit tests are mainly written by the R&D team. In the process of rapid iteration, experienced engineers will not forget to write unit tests. The faster the refactoring and changes, the less of a burden testing becomes — on the contrary, it becomes more important. If unit tests are not written, you can only say that they are not valued. Pushing forward something that the executors do not value and whose benefits managers can hardly see is very difficult.

UI automated testing, meanwhile, is often replaced by manual testing. At the same time, maintaining UI automated tests is costly, and during rapid iteration you should not do too much UI automated testing.

The advantage of API testing is that, under a front-end/back-end separation architecture, the API documentation and materials are relatively complete, and team members are relatively familiar with the APIs, which is conducive to testing.

Below is an example of API automated testing using Postman:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
pipeline {
  agent {
    kubernetes {
      label 'apitest'
      yaml '''apiVersion: v1
kind: Pod
spec:
  containers:
  - name: newman
    image: postman/newman_alpine33
    command: [\'cat\']
    tty: true
    volumeMounts:
    - name: dockersock
      mountPath: /var/run/docker.sock
    - name: dockerbin
      mountPath: /usr/bin/docker
  volumes:
  - name: dockersock
    hostPath:
      path: /var/run/docker.sock
  - name: dockerbin
    hostPath:
      path: /usr/bin/docker
      '''
      defaultContainer 'newman'
    }
  }

  parameters {
    string(name: 'HOST', defaultValue: '10.0.0.10', description: '')
    string(name: 'PORT', defaultValue: '8000', description: '')
    string(name: 'USERNAME', defaultValue: 'admin', description: '')
    string(name: 'PASSWORD', defaultValue: 'password', description: '')

  }

  stages {
    stage('testing') {
      steps {
          sh '''
          apk add --no-cache bash git openssh
          git clone https://yourdomain.com/ns/ks-api-test.git

          cd ks-api-test

          sed -i "s/__HOST__/$HOST/g" postman_environment.json
          sed -i "s/__PORT__/$PORT/g" postman_environment.json
          sed -i "s/__USERNAME__/$USERNAME/g" postman_environment.json
          sed -i "s/__PASSWORD__/$PASSWORD/g" postman_environment.json

          npm install -g newman-reporter-htmlextra
          newman run iam/postman_collection.json -e postman_environment.json -r htmlextra
          '''
      }
    }
  }
  post {
    always {
        archiveArtifacts 'ks-api-test/newman/*'
    }
  }
}

The archive after execution:

View the report:

An API automated testing framework is easy to build; it only needs a few features:

Interface requests
Response assertions
Request orchestration
Report generation

But you must choose a suitable approach based on the team’s API testing and delivery habits. You can develop your own or use existing tools. The choice above is the Postman + Newman approach, because the team generally uses Postman for API testing.

What remains is how to organize everyone to test — you can have each person commit files to a common repository, or use the paid version of Postman to share data and test centrally.

4. UI Testing

UI automated testing is costly for several reasons:

Test cases are hard to maintain. Front-end styles change, product logic changes.
It is difficult to provide a stable runtime environment. All kinds of timeouts and dirty data lead to a high failure rate.

The UI automated testing here uses the Robotframework framework that I am familiar with, using keywords for automated testing. For related documentation, see How to Package a Docker Image of Robot Framework
.

Below is an example of UI automated testing using Robotframework:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
pipeline {
  agent {
    kubernetes {
      label 'robotframework'
      yaml '''apiVersion: v1
kind: Pod
spec:
  containers:
  - name: robotframework
    image: shaowenchen/docker-robotframework:latest
    tty: true
    volumeMounts:
    - name: dockersock
      mountPath: /var/run/docker.sock
    - name: dockerbin
      mountPath: /usr/bin/docker
  volumes:
  - name: dockersock
    hostPath:
      path: /var/run/docker.sock
  - name: dockerbin
    hostPath:
      path: /usr/bin/docker
      '''
      defaultContainer 'robotframework'
    }
  }

  parameters {
    string(name: 'HOST', defaultValue: '10.0.0.10', description: '')
    string(name: 'PORT', defaultValue: '8080', description: '')
    string(name: 'USERNAME', defaultValue: 'admin', description: '')
    string(name: 'PASSWORD', defaultValue: 'password', description: '')
  }

  stages {
    stage('testing') {
      steps {
          sh '''
          curl -s -L https://raw.githubusercontent.com/shaowenchen/scripts/master/kubesphere/preinstall.sh | bash
          git clone https://yourdomain.com/ns/ks-ui-test.git

          cd ks-ui-test

          sed -i "s/__USERNAME__/$USERNAME/g" tests/common.robot
          sed -i "s/__PASSWORD__/$PASSWORD/g" tests/common.robot

          echo "\nTestEnv  http://$HOST:$PORT" >> tests/api.robot
          echo "\nTestEnv  http://$HOST:$PORT" >> tests/devops.robot
          ./start.sh'''
      }
    }
  }

  post {
    always {
        sh 'tar cvf report-$BUILD_NUMBER.tar ks-ui-test/tests/report'
        archiveArtifacts '*.tar'
    }
  }
}

Execution log:

Test report: