PaaS Deployment with Buildpack

I have been learning Go recently, and the internal PaaS platform we commonly use happens to support Go as well as related Web frameworks. A PaaS system that supports multiple languages cannot do without the buildpack mechanism. Although PaaS platforms keep being upgraded, the buildpack mechanism has always been retained. This article is mainly a collection of buildpack materials and some hands-on practice.

1. How PaaS Deploys Applications

Whether a PaaS platform is based on native Docker or on Kubernetes, it only solves the problem of resource isolation; it does not prescribe how an App runs. For an App to run, besides the code that developers care about, there is the configuration information that operations staff care about. This configuration information includes:

Account configuration, such as MySQL and Redis account passwords.
Service configuration, such as service name, port number, and third-party service addresses like Ceph.
Runtime dependencies, such as Python 2, Python 3, Golang, and Nodejs, along with related dependency packages such as gunicorn.

In fact, besides providing the runtime, before running an App, PaaS also needs to execute a series of scripts to complete some necessary runtime environment setup. Usually these scripts are called buildpacks and are managed in a shared Git repository.

2. buildpack

buildpack is Heroku’s deployment mechanism. Heroku is a PaaS platform that supports multiple languages, including Ruby, Java, Node.js, Scala, Clojure, Python, PHP, Perl, and more. On Github, Heroku open-sourced buildpacks. You can pick them up and use them right away for your project, and you can also customize and develop your own buildpacks.

CloudFoundry and Heroku buildpacks are compatible: the same buildpack can be deployed on either Heroku or CloudFoundry. Many PaaS platforms use buildpacks to deploy applications. buildpack has become the de facto standard for PaaS application deployment.

3. Cloud Foundry Deployment Principles

Cloud Foundry is the industry’s first open-source PaaS cloud platform, supporting a wide range of frameworks, languages, runtime environments, cloud platforms, and application services. Cloud Foundry has had a very important influence on the way PaaS platforms were later built. Many teams refer to Cloud Foundry when building their own PaaS.

The diagram below shows the complete flow of deploying an App on Cloud Foundry.

The user uploads the application with the CF PUSH command
The CLI tells CCNG to create an application
CCNG adds a record for the application in the database, for example the application name and which buildpack to use
The CLI uploads the program
CCNG stores the program
The CLI starts the application
Since the application has not been deployed yet, CCNG finds a DEA (Droplet Execution Agent) and runs the buildpack inside that DEA to deploy the application
The DEA outputs information about running the buildpack
When the buildpack finishes, it outputs a Droplet file (the compiled and packaged result), and the DEA stores that file
The DEA reports the packaging result to CCNG
CCNG picks a DEA to deploy the application
The application runs in the DEA and its output is sent to CCNG

As you can see, both the buildpack and the App run in the same environment (the DEA). A buildpack is very simple — it needs only three scripts:

bin/detect, detects whether the buildpack supports this application
bin/compile, the compilation script
bin/release, the packaging script

4. Creating an Application with the Heroku Buildpack

Cloud Foundry is a private cloud PaaS solution, while Heroku is a public cloud PaaS solution. Cloud Foundry has built a very good ecosystem by developing partners for integration and training. Heroku offers paid application deployment services on the public internet, but also provides a certain free quota.

For simplicity, I did not set up Cloud Foundry here, and instead used the deployment service Heroku provides directly.

4.1 Preparing Heroku

Step one: register a Heroku account at https://www.heroku.com/.

Step two: install the Heroku Toolbelt client.

Toolbelt is Heroku’s command-line tool, which lets you manage Heroku applications from the command line. Download link.

4.2 Preparing the Django Project

Step one: create a Django project for testing the deployment.

1
2
3
django-admin startproject herokupro
ls herokupro/
herokupro  manage.py

Step two: following the buildpack conventions, create two new files in the project.

requirements.txt, the third-party packages the App depends on
Procfile, the command executed when the App starts

1
2
3
4
5
6
cat herokupro/requirements.txt
django==1.8.3
gunicorn==19.7.1
whitenoise==3.3.0
cat herokupro/Procfile
web: gunicorn herokupro.wsgi

Step three: modify the Django project to suit gunicorn deployment

Add to herokupro/wsgi.py:

1
2
3
from whitenoise.django import DjangoWhiteNoise

application = DjangoWhiteNoise(application)

Add to herokupro/settings.py

1
2
3
4
5
6
7
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
STATIC_ROOT = os.path.join(BASE_DIR, 'assets')
STATIC_URL = '/static/'
STATICFILES_STORAGE = 'whitenoise.django.GzipManifestStaticFilesStorage'
STATICFILES_DIRS = (
    os.path.join(BASE_DIR, 'static'),
)

Create the static folder

1
touch herokupro/static/keep

Step four (optional): create a runtime.txt file to specify the Python version.

You can find the available Python versions on the Github page for heroku-buildpack-python.

1
2
cat herokupro/runtime.txt
python-3.7.6

The final directory structure:

1
2
ls herokupro/
Procfile  herokupro  manage.py  requirements.txt  runtime.txt  static

4.3 Creating the Heroku Application

Step one: log in to Heroku.

heroku login

Step two: create the App.

1
2
3
heroku create heroku-django-app-hello
Creating ⬢ heroku-django-app-hello... done
https://heroku-django-app-hello.herokuapp.com/ | https://git.heroku.com/heroku-django-app-hello.git

Heroku assigns the App two addresses:

The access address, https://heroku-django-app-hello.herokuapp.com/
The git repository address, https://git.heroku.com/heroku-django-app-hello.git

Step three: commit the code and build and deploy the application.

1
2
3
4
git init
git remote add origin https://git.heroku.com/heroku-django-app-hello.git
git add .
git commit -m "init"

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
git push -u origin master

Enumerating objects: 22, done.
Counting objects: 100% (22/22), done.
Delta compression using up to 4 threads
Compressing objects: 100% (18/18), done.
Writing objects: 100% (22/22), 5.23 KiB | 595.00 KiB/s, done.
Total 22 (delta 5), reused 0 (delta 0)
remote: Compressing source files... done.
remote: Building source:
remote:
remote: -----> Python app detected
remote: -----> Installing python-3.6.7
remote: -----> Installing pip
remote: -----> Installing SQLite3
remote: -----> Installing requirements with pip
remote:        Collecting django==1.8.3 (from -r /tmp/build_a54c15c00f10fa5ae49c62b5f9169306/requirements.txt (line 1))
remote:          Downloading https://files.pythonhosted.org/packages/a3/e1/0f3c17b1caa559ba69513ff72e250377c268d5bd3e8ad2b22809c7e2e907/Django-1.8.3-py2.py3-none-any.whl (6.2MB)
remote:        Installing collected packages: django
remote:        Successfully installed django-1.8.3
remote:
remote: -----> python manage.py collectstatic --noinput
remote:        63 static files copied to '/tmp/build_a54c15c00f10fa5ae49c62b5f9169306/staticfiles'.
remote:
remote: -----> Discovering process types
remote:        Procfile declares types -> web
remote:
remote: -----> Compressing...
remote:        Done: 48.8M
remote: -----> Launching...
remote:        Released v5
remote:        https://heroku-django-app-hello.herokuapp.com/ deployed to Heroku
remote:
remote: Verifying deploy... done.
To https://git.heroku.com/heroku-django-app-hello.git
 * [new branch]      master -> master

Visit the App address Heroku provides, https://heroku-django-app-hello.herokuapp.com/:

A deployment may not succeed on the first try. You can debug by viewing the logs with this command:

1
heroku logs --tail --app heroku-django-app-hello