Dial Testing Services with Jenkins

1. Why Dial Testing

For some of the critical services in a system, we usually set up monitoring. When a failure occurs, it can be detected as quickly as possible and a notification sent to the people who care. Once a failure has occurred, the failure process can be traced effectively.

Dial testing is an important part of a monitoring system: it can check the network quality of a service and provide real-time alerts.

On public clouds, cloud vendors provide dial testing services through widely distributed dial testing nodes. So how do you implement dial testing on Jenkins?

2. Setting Up a Mail Server

The mail service chosen here is Poste, with the mail domain mail.dev.chenshaowen.com.

Domain configuration

mail.dev.chenshaowen.com -> A record -> host IP
mail.dev.chenshaowen.com -> TXT record -> v=spf1 +all
_dmarc.mail.dev.chenshaowen.com -> TXT record -> v=DMARC1;p=none

Running the service

Create the data storage directory

1
mkdir /maildata

Run poste.io in the background

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
docker run -d \
    --security-opt apparmor=unconfined \
    --security-opt seccomp=unconfined \
    -p 25:25 \
    -p 80:80 \
    -p 110:110 \
    -p 143:143 \
    -p 443:443 \
    -p 587:587 \
    -p 993:993 \
    -p 995:995 \
    -v /etc/localtime:/etc/localtime:ro \
    -v /maildata:/data \
    --name "mailserver" \
    -h "mail.dev.chenshaowen.com" \
    -t analogic/poste.io

Service configuration, set the IP whitelist

Open the page https://mail.dev.chenshaowen.com:443 and create a mail account. Once you enter the Poste home page, you will see the following page:

Here you need to add the IP of the host running Jenkins to the whitelist, otherwise you will get a 550 error.

In addition, under the /webmail/ route, users can use the mailbox features normally.

3. Dial Testing with Jenkins

For a quick Jenkins deployment, see the Docker Compose script.

The main features used are:

Email notification
Creating a pipeline
Scheduled builds

3.1 Enabling Email Notification

In Jenkins, go to [Manage Jenkins] -> [Configure System] -> [E-mail Notification] and configure the email notification service, as shown below:

After filling in the service information, it is recommended to send a test email to make sure email works. Finally, just save.

3.2 Creating a Pipeline

The dial testing here is divided into two parts: the service status code and the response time, each forming one of two parallel stages.

The testing principle is to use the curl command to get the status code and response time of the service link, then compare them with the expected values. If the trigger conditions are met, an exception notification email is sent.

Below is the content of the Jenkinsfile:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
pipeline {
  agent any
  parameters {
    string(name: 'LINK', defaultValue: 'https://www.chenshaowen.com/', description: '待拨测链接')
    string(name: 'HTTP_CODE', defaultValue: '200', description: '预期状态码')
    string(name: 'TIME_TOTAL', defaultValue: '1', description: '超时时间，单位：秒')
    string(name: 'EMAIL', defaultValue: 'admin@mail.dev.chenshaowen.com', description: '异常时，邮件通知人')
  }
  stages {
    stage('default') {
        parallel {
            stage('状态测试') {
                steps {
                    script {
                        try{
                            sh '''
                            export _HTTP_CODE=$(curl --connect-timeout 120 -s -o /dev/null -w \'%{http_code}\' $LINK)
                            if [ "$_HTTP_CODE" != "$HTTP_CODE" ]
                            then
                                exit -1
                            fi
                            '''
                        }catch(err){
                            currentBuild.result = 'FAILURE'
                            mail(subject: "$LINK 访问状态码错误", body: "消息来自 DevOps 流水线，请检查相关服务是否异常。", to: "$EMAIL")
                        }
                    }
                }
            }
            stage('超时测试') {
                steps {
                    script {
                        try{
                            sh '''
                            _TIME_TOTAL=$(curl --connect-timeout 120 -s -o /dev/null -w \'%{time_total}\' $LINK)
                            TIME_TOTAL=`echo $TIME_TOTAL| awk '{print int($0)}'`
                            _TIME_TOTAL=`echo $_TIME_TOTAL| awk '{print int($0)}'`
                            if [ $_TIME_TOTAL -ge  $TIME_TOTAL ]
                            then
                                exit -1
                            fi
                            '''
                        }catch(err){
                            currentBuild.result = 'FAILURE'
                            mail(subject: "$LINK 访问超时", body: "消息来自 DevOps 流水线，请检查相关服务是否异常。", to: "$EMAIL")
                        }
                    }
                }
            }
        }
    }
  }
}

Create a [Pipeline] job, click [Configure], paste the Jenkinsfile content above into the script content of [Pipeline], and save.

3.3 Scheduled Builds

After the dial testing pipeline is created successfully, it can only be triggered manually. What dial testing needs is uninterrupted monitoring 24 hours a day. At this point, you need the scheduled build feature.

On the pipeline page, click [Configure] and find [Build Triggers]. Check scheduled build, set the pipeline to trigger every 5 minutes, and fill in the parameter: