In a non-staged build scenario, when we use a container to build, we can mount the cache directory inside the container onto the build host, run the build task, and then copy the artifacts into the runtime image to produce the application image. But with a staged build, the build image and the runtime image live in the same Dockerfile, which makes it harder to optimize the caching of third-party dependencies.
1. Create a Vue Example Project
1
| npm install -g @vue/cli --force
|
- Initialize the example project
Using the default configuration, create the example project: hello-world
At this point the project already contains all its dependencies, so you can run it directly:
Dependency packages are usually not committed to the code repository. To better simulate the build scenario, remove the dependencies here and then build.
- Add a Dockerfile to the project
Enter the project directory:
Edit and save the Dockerfile:
1
2
3
4
5
6
7
8
9
10
11
12
13
| vim Dockerfile
FROM node:lts-alpine as builder
WORKDIR /
COPY package.json /
RUN npm install
COPY . .
RUN npm run build
FROM nginx:alpine
COPY --from=builder /dist/ /usr/share/nginx/html/
EXPOSE 80
|
Run the command:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
| docker build --no-cache -t shaowenchen/hello-world:v1 -f Dockerfile .
[+] Building 139.2s (13/13) FINISHED
=> [internal] load build definition from Dockerfile 2.6s
=> => transferring dockerfile: 228B 0.2s
=> [internal] load .dockerignore 3.4s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/nginx:alpine 4.2s
=> [internal] load metadata for docker.io/library/node:lts-alpine 4.3s
=> CACHED [builder 1/6] FROM docker.io/library/node:lts-alpine@sha256:2c6c59cf4d34d4f937ddfcf33bab9d8bbad8658d1b9de7b97622566a52167f2b 0.0s
=> [internal] load build context 1.8s
=> => transferring context: 5.03kB 0.4s
=> CACHED [stage-1 1/2] FROM docker.io/library/nginx:alpine@sha256:da9c94bec1da829ebd52431a84502ec471c8e548ffb2cedbf36260fd9bd1d4d3 0.0s
=> [builder 2/6] COPY package.json / 5.3s
=> [builder 3/6] RUN npm install 93.1s
=> [builder 4/6] COPY . . 5.9s
=> [builder 5/6] RUN npm run build 13.6s
=> [stage-1 2/2] COPY --from=builder /dist/ /usr/share/nginx/html/ 4.0s
=> exporting to image 4.0s
=> => exporting layers 2.3s
=> => writing image sha256:dc0f72b655eb95235b51d8fb30c430c3c1803c2d538d9948941f3e7afd23ab56 0.2s
=> => naming to docker.io/shaowenchen/hello-world:v1 0.2s
|
Run the command to create a container:
1
| docker run --rm -it -p 80:80 shaowenchen/hello-world:v1
|
Open http://localhost locally and you will see the page.
2. Optimizing by Mounting Cache with Buildkit
The idea behind this approach is to store the third-party packages separately in a cache image, and when building the application image, mount the files from the cache image into the build environment.
2.1 Enabling Buildkit
Buildkit is disabled by default. There are two ways to enable it:
- The first: add a buildkit setting to
/etc/docker/daemon.json — { "features": { "buildkit": true }} enables the buildkit feature by default. - The second: add the environment variable
DOCKER_BUILDKIT=1 every time you run a docker command.
2.2 Mounting the Cache with Bind
- Prepare the Dockerfile for the cache image
Create the Dockerfile:
1
2
3
4
5
6
7
| vim Dockerfile-Cache
FROM node:lts-alpine as builder
WORKDIR /
COPY . .
RUN npm install
RUN npm run build
|
There is a small detail here: you need to compile the third-party packages with npm run build. Caching only the third-party packages does not give you a very good speedup. At the same time, precompiling reduces CPU and memory consumption.
- Build the cache image containing the third-party packages
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
| docker build --no-cache -t shaowenchen/hello-world:cache -f Dockerfile-Cache .
[+] Building 111.9s (9/9) FINISHED
=> [internal] load build definition from Dockerfile-Cache 1.8s
=> => transferring dockerfile: 132B 0.0s
=> [internal] load .dockerignore 2.9s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/node:lts-alpine 4.2s
=> [internal] load build context 1.7s
=> => transferring context: 4.57kB 0.2s
=> CACHED [1/5] FROM docker.io/library/node:lts-alpine@sha256:2c6c59cf4d34d4f937ddfcf33bab9d8bbad8658d1b9de7b97622566a52167f2b 0.0s
=> [2/5] COPY . . 3.6s
=> [3/5] RUN npm install 69.2s
=> [4/5] RUN npm run build 14.5s
=> exporting to image 13.9s
=> => exporting layers 13.0s
=> => writing image sha256:e6ba7406f5d0c33d446ecc9a3c8e35fa593176ec9dedd899d39a1c00a14a5179 0.2s
=> => naming to docker.io/shaowenchen/hello-world:cache 0.2s
|
- Prepare the build Dockerfile for the application
1
2
3
4
5
6
7
8
9
10
11
12
13
| vim Dockerfile-Bind
FROM node:lts-alpine as builder
WORKDIR /
COPY . .
RUN --mount=type=bind,from=shaowenchen/hello-world:cache,source=/node_modules,target=/node_modules \
--mount=type=bind,from=shaowenchen/hello-world:cache,source=/root/.npm,target=/root/.npm npm install
RUN --mount=type=bind,from=shaowenchen/hello-world:cache,source=/node_modules,target=/node_modules \
--mount=type=bind,from=shaowenchen/hello-world:cache,source=/root/.npm,target=/root/.npm npm run build
FROM nginx:alpine
COPY --from=builder /dist/ /usr/share/nginx/html/
EXPOSE 80
|
Every command that uses the cache needs --mount added in front of it.
- Build the application image
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
| docker build --no-cache -t shaowenchen/hello-world:v1-bind -f Dockerfile-Bind .
[+] Building 55.3s (13/13) FINISHED
=> [internal] load build definition from Dockerfile-Bind 2.5s
=> => transferring dockerfile: 42B 0.0s
=> [internal] load .dockerignore 3.4s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/nginx:alpine 4.0s
=> [internal] load metadata for docker.io/library/node:lts-alpine 3.8s
=> [internal] load build context 2.4s
=> => transferring context: 4.47kB 0.2s
=> CACHED FROM docker.io/shaowenchen/hello-world:cache 0.3s
=> CACHED [stage-1 1/2] FROM docker.io/library/nginx:alpine@sha256:da9c94bec1da829ebd52431a84502ec471c8e548ffb2cedbf36260fd9bd1d4d3 0.0s
=> CACHED [builder 1/5] FROM docker.io/library/node:lts-alpine@sha256:2c6c59cf4d34d4f937ddfcf33bab9d8bbad8658d1b9de7b97622566a52167f2b 0.0s
=> [builder 2/5] COPY . . 4.2s
=> [builder 3/5] RUN --mount=type=bind,from=shaowenchen/hello-world:cache,source=/node_modules,target=/node_modules --mount=type=bind,from=shaowenchen/hello-world:cache 16.8s
=> [builder 4/5] RUN --mount=type=bind,from=shaowenchen/hello-world:cache,source=/node_modules,target=/node_modules --mount=type=bind,from=shaowenchen/hello-world:cache 13.2s
=> [stage-1 2/2] COPY --from=builder /dist/ /usr/share/nginx/html/ 3.7s
=> exporting to image 4.8s
=> => exporting layers 3.1s
=> => writing image sha256:de18663c5752a41cd61c23fb2cbbc1ac9c4c79cf5fdbe15ca16e806d0ce18d9d 0.2s
=> => naming to docker.io/shaowenchen/hello-world:v1-bind 0.1s
|
As you can see, with the cache in place, the total time to run install and build dropped from over 100 seconds to under 30 seconds.
3. Optimizing by Storing the Cache in S3
3.1 Quickly Deploy a minio
Reference file: Build Artifacts and Cache in Jenkins
Create the credential file .s3cfg under the hello-world directory.
host_base = 1.1.1.1:9000
host_bucket = 1.1.1.1:9000
use_https = False
access_key = minio
secret_key = minio123
signature_v2 = False
3.3 Rework the Dockerfile to Fit S3 Caching
The main work items here are:
- Install s3cmd
- Fetch and extract the cache, ignoring errors (empty the first time)
- … install dependencies and build
- Compress and upload the cache
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
| vim Dockerfile-S3
FROM node:lts-alpine as builder
ARG BUCKETNAME
ENV BUCKETNAME=$BUCKETNAME
RUN apk add python3 && ln -sf python3 /usr/bin/python && apk add py3-pip
RUN wget https://sourceforge.net/projects/s3tools/files/s3cmd/2.2.0/s3cmd-2.2.0.tar.gz \
&& mkdir -p /usr/local/s3cmd && tar -zxf s3cmd-2.2.0.tar.gz -C /usr/local/s3cmd \
&& ln -s /usr/local/s3cmd/s3cmd-2.2.0/s3cmd /usr/bin/s3cmd && pip3 install python-dateutil
WORKDIR /
# Get Cache
COPY .s3cfg /root/
RUN s3cmd get s3://$BUCKETNAME/node_modules.tar.gz && tar xf node_modules.tar.gz || exit 0
RUN s3cmd get s3://$BUCKETNAME/npm.tar.gz && tar xf npm.tar.gz || exit 0
COPY . .
RUN npm install
RUN npm run build
# Uploda Cache
RUN s3cmd del s3://$BUCKETNAME/node_modules.tar.gz || exit 0
RUN s3cmd del s3://$BUCKETNAME/npm.tar.gz || exit 0
RUN tar cvfz node_modules.tar.gz node_modules
RUN tar cvfz npm.tar.gz ~/.npm
RUN s3cmd put node_modules.tar.gz s3://$BUCKETNAME/
RUN s3cmd put npm.tar.gz s3://$BUCKETNAME/
FROM nginx:alpine
COPY --from=builder /dist/ /usr/share/nginx/html/
EXPOSE 80
|
- Building the application image with the S3 cache for the first time
Before building, you need to create a bucket named hello-world ahead of time.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
| docker build --no-cache --build-arg BUCKETNAME="hello-world" -t shaowenchen/hello-world:v1-s3 -f Dockerfile-S3 .
[+] Building 244.7s (23/23) FINISHED
=> [internal] load build definition from Dockerfile-S3 1.7s
=> => transferring dockerfile: 40B 0.1s
=> [internal] load .dockerignore 2.6s
=> => transferring context: 2B 0.1s
=> [internal] load metadata for docker.io/library/nginx:alpine 2.6s
=> [internal] load metadata for docker.io/library/node:lts-alpine 0.0s
=> CACHED [builder 1/16] FROM docker.io/library/node:lts-alpine 0.0s
=> [internal] load build context 2.2s
=> => transferring context: 4.53kB 0.1s
=> CACHED [stage-1 1/2] FROM docker.io/library/nginx:alpine@sha256:da9c94bec1da829ebd52431a84502ec471c8e548ffb2cedbf36260fd9bd1d4d3 0.0s
=> [builder 2/16] RUN apk add python3 && ln -sf python3 /usr/bin/python && apk add py3-pip 32.3s
=> [builder 3/16] RUN wget https://sourceforge.net/projects/s3tools/files/s3cmd/2.2.0/s3cmd-2.2.0.tar.gz && mkdir -p /usr/local/s3cmd && tar -zxf s3cmd-2.2.0.tar.g 12.8s
=> [builder 4/16] COPY .s3cfg /root/ 5.8s
=> [builder 5/16] RUN s3cmd get s3://hello-world/node_modules.tar.gz && tar xf node_modules.tar.gz || exit 0 6.7s
=> [builder 6/16] RUN s3cmd get s3://hello-world/npm.tar.gz && tar xf npm.tar.gz || exit 0 7.3s
=> [builder 7/16] COPY . . 5.7s
=> [builder 8/16] RUN npm install 71.3s
=> [builder 9/16] RUN npm run build 14.4s
=> [builder 10/16] RUN s3cmd del s3://hello-world/node_modules.tar.gz || exit 0 7.5s
=> [builder 11/16] RUN s3cmd del s3://hello-world/npm.tar.gz || exit 0 6.9s
=> [builder 12/16] RUN tar cvfz node_modules.tar.gz node_modules 11.3s
=> [builder 13/16] RUN tar cvfz npm.tar.gz ~/.npm 9.4s
=> [builder 14/16] RUN s3cmd put node_modules.tar.gz s3://hello-world/ 14.8s
=> [builder 15/16] RUN s3cmd put npm.tar.gz s3://hello-world/ 15.9s
=> [stage-1 2/2] COPY --from=builder /dist/ /usr/share/nginx/html/ 4.5s
=> exporting to image 3.9s
=> => exporting layers 2.5s
=> => writing image sha256:dceead698b2c5f3980bf17f246078fe967dda2d9b009c30d9fdb0c60263146e5 0.1s
=> => naming to docker.io/shaowenchen/hello-world:v1-s3 0.2s
|
You can see the relevant cache files in the Minio UI:
- Building the application image with the S3 cache again
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
| docker build --no-cache --build-arg BUCKETNAME="hello-world" -t shaowenchen/hello-world:v1-s3 -f Dockerfile-S3 .
[+] Building 213.8s (23/23) FINISHED
=> [internal] load build definition from Dockerfile-S3 2.0s
=> => transferring dockerfile: 40B 0.0s
=> [internal] load .dockerignore 2.7s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/nginx:alpine 4.6s
=> [internal] load metadata for docker.io/library/node:lts-alpine 0.0s
=> CACHED [builder 1/16] FROM docker.io/library/node:lts-alpine 0.0s
=> CACHED [stage-1 1/2] FROM docker.io/library/nginx:alpine@sha256:da9c94bec1da829ebd52431a84502ec471c8e548ffb2cedbf36260fd9bd1d4d3 0.0s
=> [internal] load build context 1.9s
=> => transferring context: 4.53kB 0.1s
=> [builder 2/16] RUN apk add python3 && ln -sf python3 /usr/bin/python && apk add py3-pip 30.9s
=> [builder 3/16] RUN wget https://sourceforge.net/projects/s3tools/files/s3cmd/2.2.0/s3cmd-2.2.0.tar.gz && mkdir -p /usr/local/s3cmd && tar -zxf s3cmd-2.2.0.tar.g 13.2s
=> [builder 4/16] COPY .s3cfg /root/ 5.5s
=> [builder 5/16] RUN s3cmd get s3://hello-world/node_modules.tar.gz && tar xf node_modules.tar.gz || exit 0 16.7s
=> [builder 6/16] RUN s3cmd get s3://hello-world/npm.tar.gz && tar xf npm.tar.gz || exit 0 15.3s
=> [builder 7/16] COPY . . 4.7s
=> [builder 8/16] RUN npm install 18.4s
=> [builder 9/16] RUN npm run build 13.6s
=> [builder 10/16] RUN s3cmd del s3://hello-world/node_modules.tar.gz || exit 0 7.4s
=> [builder 11/16] RUN s3cmd del s3://hello-world/npm.tar.gz || exit 0 7.9s
=> [builder 12/16] RUN tar cvfz node_modules.tar.gz node_modules 10.8s
=> [builder 13/16] RUN tar cvfz npm.tar.gz ~/.npm 10.0s
=> [builder 14/16] RUN s3cmd put node_modules.tar.gz s3://hello-world/ 17.9s
=> [builder 15/16] RUN s3cmd put npm.tar.gz s3://hello-world/ 16.3s
=> [stage-1 2/2] COPY --from=builder /dist/ /usr/share/nginx/html/ 5.0s
=> exporting to image 3.8s
=> => exporting layers 2.5s
=> => writing image sha256:a9c46eef6073b3ef8e6c4cd33cc1ed11c94dcebdb0883c89283883d9434de331 0.2s
=> => naming to docker.io/shaowenchen/hello-world:v1-s3 0.2s
|
As you can see, the install and build commands take about 80 seconds, but the S3 cache-related operations take about 50 seconds.
Of those 80 seconds, another place that can be optimized is that the network between the build environment and the S3 service is rate-limited to 1.2 MB/S, which makes pulling and pushing take too long and leaves plenty of room for improvement. I think under 30 seconds would be reasonable.
4. Summary
Cache acceleration is a hard problem for CI products. Everyone uses things differently, and what we can do is provide solutions for the user’s scenario rather than forcibly change their habits. In the CI product I developed previously, the main approach was to mount the cache from the host into the build environment to speed things up, which does not apply to the staged build scenario. Here I have mainly provided two approaches:
The first is to enable the Buildkit feature and store the third-party dependency packages in a cache image. The cache image can be updated on a schedule according to a policy. When building the image, mount the third-party packages from the cache image.
The second is to use S3 to store the third-party dependency packages, and use the s3cmd command to manage the cache at build time.
Neither of these two approaches is particularly good. The main reason is that both require modifying the Dockerfile, which is quite invasive to the business.
5. References
