How to Build an Automated Deployment Pipeline for Django with Jenkins, Docker, and GitLab

One of the illusions that programmers at big companies easily fall into is mistaking platform capabilities for their own abilities. In a large team, we should not focus only on our own little patch of ground; we need to understand every part of the platform. On one hand, this helps us make better use of the platform’s features; on the other, it helps our own technical growth. This article describes how to build an automated development and deployment pipeline for Django using Jenkins, Docker, and GitLab. The relevant tools are all open source and ready to use out of the box.

1. Development Workflow

In the production environment, the web application is deployed with multiple K8S instances, and the stateful services MySQL and RabbitMQ are deployed as clusters. Monitoring, log collection, and log search and other surrounding facilities were also set up.

Compared with the production environment, for the development workflow here I want to simulate production as closely as possible, but it does not need to be too polished. After all, personal time and energy are limited, and improving things step by step as needs arise is a good choice.

Here GitLab is used as the development repository, Jenkins as the automation engine, and Docker images for deployment.

Below is a simple deployment flow:

When the trigger condition is met, Jenkins automatically pulls the code from GitLab, builds a Docker image, and finally runs the Django instance on the server. This is basically enough to simulate the entire deployment flow.

2. GitLab Configuration

GitLab was chosen because it allows creating private repositories.

Create a repository

First, create a GitLab repository, for example: ProjectA

Add an SSH key for remote access

Run the following command locally to generate the SSH key needed for remote access

1
ssh-keygen -o -t rsa -b 4096 -C "mail@chenshaowen.com"

On the https://gitlab.com/profile page, find [SSH Keys] and add the key generated above.

Generate a token for accessing your personal repository

On the https://gitlab.com/profile page, find [Access Tokens], fill in the information, and click generate to get a PersonToken.

3. Jenkins Configuration

Jenkins can expose an API directly and also supports plugin extension. For teams familiar with Java, Jenkins is very appealing. With Jenkins you can satisfy all kinds of CI and CD needs.

The Jenkins here is mainly used to deploy services. By receiving the commit information sent by GitLab, it pulls the latest code, runs a script, and completes the deployment.

Install the GitLab-related plugins

The Jenkins plugins needed here are mainly:

Gitlab Authentication plugin
Gitlab Hook Plugin
Gitlab Plugin

In [Jenkins] -> [Plugin Manager], search for and install the plugins, then restart Jenkins for them to take effect.

Add GitLab access credentials

In [Jenkins] -> [Credentials] -> [System] -> [Global credentials (unrestricted)], click [Add Credentials], choose [Gitlab API token] as the type, and use the PersonToken generated in chapter 2 as the API token.

Create a pipeline and configure the repository

Create a [Build a free-style software project] and fill in the repository address of ProjectA as shown above. Click to add an SSH key access credential.

Configure the Jenkins trigger rule

As shown above, under [Build Triggers] check [Build when a change is pushed to GitLab. GitLab webhook URL:] to get the GitLab Webhook address. Click [Advanced] to generate a Token.

Configure the Webhook in GitLab

In the previous step two values were obtained: the GitLab Webhook and the Token.

As shown above, fill in the relevant information under [Settings] -> [Integrations] in the GitLab project repository. If your Webhook is not an https link, you also need to uncheck [Enable SSL verification].

Jenkins build configuration

The build configuration is in fact the script command that Jenkins executes after pulling the repository code. Here you can simply run the start.sh script under the project.

4. Building the Docker Image

Deploying with Docker helps package environment dependencies and horizontally scale the service. In a production environment, deployment is usually done with multiple instances plus a cluster to ensure high availability of the service.

Here docker-compose is mainly used to build images and orchestrate the containers the Django runtime needs.

The image above shows the directory structure of the whole repository, divided into four parts.

4.1 Django Project Code

Django uses the default directory structure, and there are two things to note:

Distinguish environments via an environment variable

At startup, an environment variable needs to be passed in to let Django distinguish environments. In the settings.py file:

1
2
3
4
if os.getenv('Env') == 'Production':
    DEBUG = False
else:
    DEBUG = True

In Django’s DEBUG=False mode, static files cannot be forwarded, so WhiteNoise needs to be configured

1
2
3
4
5
6
7
8
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfile')
#
STATICFILES_DIRS = (
    os.path.join(BASE_DIR, 'static'),
)
WHITENOISE_STATIC_PREFIX = '/static/'
MIDDLEWARE.append('whitenoise.middleware.WhiteNoiseMiddleware')
STATICFILES_STORAGE = 'whitenoise.storage.CompressedStaticFilesStorage'

4.2 Data Storage

By mounting the data directory into Docker as a volume, you can preserve the running state and avoid losing data when a container restarts.

4.3 Image Configuration

The Python image needs the basic dependency packages installed.

Dockerfile:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
FROM python:3.7-alpine

RUN apk update \
    && apk add --no-cache --virtual bash \
    && apk add gcc \
    && apk add musl-dev \
    && apk add linux-headers \
    && apk add jpeg-dev \
    && apk add zlib-dev \
    && apk add mariadb-dev \
    && apk add libffi-dev

COPY requirements.txt /requirements.txt
RUN pip install --upgrade pip \
    && pip install -r requirements.txt \
    && rm /usr/bin/mysql*

RUN mkdir /code
WORKDIR /code

requirements.txt file

1
2
3
4
5
6
7
8
9
django==2.1.2
gunicorn==19.9.0
mysqlclient==1.3.13
pymysql==0.9.2
whitenoise==4.1.2
celery==4.2.1
django-celery-results==1.0.4
django-celery-beat==1.3.0
redis==2.10.6

MySQL image, used to provide the DB access service.

Dockerfile:

1
2
FROM mysql:5.7
COPY my.cnf /etc/mysql/conf.d/my.cnf

my.cnf file:

1
2
3
4
[mysqld]
character-set-server=utf8
[client]
default-character-set=utf8

Container orchestration

Here the Python image is reused to provide the django and celery container environments. Django is started via gunicorn.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
version: '2'
services:
  python:
    build: ./docker/python
    container_name: django
    ports:
      - 7900:7900
    volumes:
      - ./code:/code
    command: >
      bash -c "pip install -r requirements.txt
      && python manage.py migrate
      && python manage.py collectstatic --no-input
      && gunicorn news.wsgi -b 0.0.0.0:7900"
    environment:
      - Env=Production
    depends_on:
      - mysql
      - redis
      - rabbitmq
      - celery
      - mongo
    networks:
      - django-networks

  mysql:
    build: ./docker/mysql
    container_name: mysql
    ports:
      - 3306:3306
    volumes:
      - ./data/mysql:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=root
      - MYSQL_DATABASE=news
    networks:
      - django-networks

  redis:
    image: redis:latest
    container_name: redis
    expose:
      - "6379"
    networks:
      - django-networks

  rabbitmq:
    image: rabbitmq:3-management
    container_name: rabbitmq
    environment:
        - RABBITMQ_DEFAULT_USER=guest
        - RABBITMQ_DEFAULT_PASS=guest
    ports:
        - "5673:5673"
    networks:
      - django-networks

  celery:
    build: ./docker/python
    container_name: celery
    environment:
      - Env=Production
    depends_on:
        - rabbitmq
        - mysql
    volumes:
        - ./code:/code
    command: >
      bash -c "pip install -r requirements.txt
      && celery -A news.celery worker -l INFO
      && celery -A news.celery beat -l INFO --scheduler django_celery_beat.schedulers:DatabaseScheduler"
    networks:
        - django-networks

  mongo:
    image: mongo:latest
    container_name: mongo
    ports:
      - "27018:27017"
    volumes:
      - ./data/mongo:/data/db
    networks:
      - django-networks

networks:
  django-networks:
    driver: "bridge"

The start.sh script, used to build the image and restart the containers.

1
2
3
4
#!/bin/bash
docker-compose build
docker-compose stop
docker-compose up -d

5. Running Tests

After committing code to the GitLab repository, the Jenkins pipeline is triggered automatically.

At this point the service is accessible on port 7900. If you need to bind a domain, add an Nginx Server configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
server {
     listen 80;
     server_name yourdomain.com;

     location / {
         proxy_pass http://127.0.0.1:7900;
         proxy_set_header Host $host;
         proxy_set_header X-Real-IP $remote_addr;
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
     }
}

By triggering a Celery background task, you can confirm that Celery, RabbitMQ, and MySQL all provide their services normally.

6. Comparison with the Production Environment

High availability

The key to high concurrency is being stateless, using clusters to provide high-performance, highly available services for state. Here MySQL, RabbitMQ, and Redis are all single instances; a production environment needs cluster deployment.

On the other hand, single-machine single-instance deployment is very unreliable; it is best to use multi-machine multi-instance deployment.

Runtime logs

For online services, logs are important information for auditing and troubleshooting errors. Using ELK + Filebeat to collect logs at different stages of the chain and at different levels, and to string them together in chronological order, is very necessary.

Runtime isolation

A single host in a production environment may run many instances, and the resources each instance uses — CPU, memory, IO, and so on — need to be isolated to avoid them affecting one another.

Service registration

Here we add a service by adding an Nginx Server configuration. You can automate this process with Etcd + Confd; see a previous article. If you use K8S, Ingress can achieve a similar effect.

Monitoring

A production environment, of course, also cannot do without monitoring and alerting on the status of various services. You can use open-source monitoring tools such as Prometheus + Grafana to quickly build a monitoring system.