In principle, Jenkins can use Podman to build images in any Kubernetes cluster; this article uses Containerd as the example.
1. Dropping Docker Brings New Challenges to CICD
In CICD scenarios, we often need to build and push images within a pipeline.
In an earlier document, Creating Jenkins Slaves Dynamically on Kubernetes, I described how mounting the /var/run/docker.sock file allows images to be built and pushed in a Docker-driven Kubernetes cluster. In How to Use Docker in Docker, I explained this in more detail; the principle is sharing the host’s Docker Daemon.
After version 1.20, the Kubernetes community abandoned its support for Docker. Other communities later took it over, but it still cast a faint shadow over Docker. Against that backdrop, we started to consider how to practice CICD in a non-Docker environment.
A non-Docker environment means the previous approach of mounting /var/run/docker.sock no longer works, and we need to find a new solution.
2. Test Cluster Environment
2.1 Kubernetes - 1.17.9
Run the following command to check the Kubernetes version:
1
2
3
4
| kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.9", GitCommit:"4fb7ed12476d57b8437ada90b4f93b17ffaeed99", GitTreeState:"clean", BuildDate:"2020-07-15T16:18:16Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.9", GitCommit:"4fb7ed12476d57b8437ada90b4f93b17ffaeed99", GitTreeState:"clean", BuildDate:"2020-07-15T16:10:45Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
|
2.2 Containerd - 1.4.3
Run the following command to check the containerd version:
1
2
3
| containerd --version
containerd github.com/containerd/containerd v1.4.3 269548fa27e0089a8b8278fc4fc781d7f65a939b
|
Because Containerd does not support the Docker API, common commands such as docker build and docker push are unavailable in a Containerd environment. We therefore need an image build and push tool that does not depend on Docker and targets the OCI standard.
3.1 Podman in Brief
Podman is a container and image management tool that implements the OCI standard. It is also daemonless — no daemon process is required — and it supports use by unprivileged users. Podman offers functionality similar to the Docker CLI, and in most cases you can run alias docker=podman to replace Docker with Podman without any problems.
3.2 Installing Podman
For installation instructions, see the Podman installation guide. CentOS 7 is used as the example here:
1
2
| curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/CentOS_7/devel:kubic:libcontainers:stable.repo
yum -y install podman
|
1
2
3
| podman --version
podman version 3.0.1
|
- View the command arguments
The full help output is reproduced here for easy reference.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
| podman --help
manage pods and images
Usage:
podman [flags]
podman [command]
Available Commands:
attach Attach to a running container
build Build an image using instructions from Containerfiles
commit Create new image based on the changed container
container Manage Containers
cp Copy files/folders between a container and the local filesystem
create Create but do not start a container
diff Inspect changes on container's file systems
events Show podman events
exec Run a process in a running container
export Export container's filesystem contents as a tar archive
generate Generated structured data
healthcheck Manage Healthcheck
help Help about any command
history Show history of a specified image
image Manage images
images List images in local storage
import Import a tarball to create a filesystem image
info Display podman system information
init Initialize one or more containers
inspect Display the configuration of a container or image
kill Kill one or more running containers with a specific signal
load Load an image from container archive
login Login to a container registry
logout Logout of a container registry
logs Fetch the logs of a container
mount Mount a working container's root filesystem
network Manage Networks
pause Pause all the processes in one or more containers
play Play a pod
pod Manage pods
port List port mappings or a specific mapping for the container
ps List containers
pull Pull an image from a registry
push Push an image to a specified destination
restart Restart one or more containers
rm Remove one or more containers
rmi Removes one or more images from local storage
run Run a command in a new container
save Save image to an archive
search Search registry for image
start Start one or more containers
stats Display a live stream of container resource usage statistics
stop Stop one or more containers
system Manage podman
tag Add an additional name to a local image
top Display the running processes of a container
umount Unmounts working container's root filesystem
unpause Unpause the processes in one or more containers
unshare Run a command in a modified user namespace
varlink Run varlink interface
version Display the Podman Version Information
volume Manage volumes
wait Block on one or more containers
Flags:
--cgroup-manager string Cgroup manager to use (cgroupfs or systemd) (default "systemd")
--cni-config-dir string Path of the configuration directory for CNI networks
--config string Path of a libpod config file detailing container server configuration options
--conmon string Path of the conmon binary
--cpu-profile string Path for the cpu profiling results
--events-backend string Events backend to use
--help Help for podman
--hooks-dir strings Set the OCI hooks directory path (may be set multiple times)
--log-level string Log messages above specified level: debug, info, warn, error, fatal or panic (default "error")
--namespace string Set the libpod namespace, used to create separate views of the containers and pods on the system
--network-cmd-path string Path to the command for configuring the network
--root string Path to the root directory in which data, including images, is stored
--runroot string Path to the 'run directory' where all state information is stored
--runtime string Path to the OCI-compatible binary used to run containers, default is /usr/bin/runc
--storage-driver string Select which storage driver is used to manage storage of images and containers (default is overlay)
--storage-opt stringArray Used to pass an option to the storage driver
--syslog Output logging information to syslog as well as the console
--tmpdir string Path to the tmp directory
--trace Enable opentracing output
-v, --version Version of podman
Use "podman [command] --help" for more information about a command.
|
While covering the Docker commands, Podman adds support for Pod operations.
3.3 Building and Pushing Images on the Host
In practice you can simply replace the docker command with podman.
1
2
3
4
5
6
7
8
9
10
11
12
13
| echo -e 'FROM busybox\nRUN echo "hello world"' | podman build -t docker.io/shaowenchen/myimage:latest -
STEP 1: FROM busybox
Getting image source signatures
Copying blob 5c4213be9af9 done
Copying config 491198851f done
Writing manifest to image destination
Storing signatures
STEP 2: RUN echo "hello world"
hello world
STEP 3: COMMIT
4c8794086d9de80f71d182457b6d2cb18b9d61975b98bcd4cb167bdcabae5b2c
4c8794086d9de80f71d182457b6d2cb18b9d61975b98bcd4cb167bdcabae5b2c
|
1
2
| podman images |grep shaowenchen
docker.io/shaowenchen/myimage latest 4c8794086d9d 4 minutes ago 1.46 MB
|
1
2
3
4
| podman login docker.io -u shaowenchen
Password:
Login Succeeded!
|
1
2
3
4
5
6
7
8
9
| podman push docker.io/shaowenchen/myimage:latest
Getting image source signatures
Copying blob 2893437c336c done
Copying blob 84009204da3f done
Copying config 4c8794086d done
Writing manifest to image destination
Storing signatures
|
4. Building Images with Podman in Jenkins
4.1 Key Configuration
- Use hostPath to mount
/var/lib/containers onto the host
A PVC can also be used, though the PVC may need extra parameters — see below.
Otherwise you will get the following error:
1
| Error: 'overlay' is not supported over overlayfs, a mount_program is required: backing file system is unsupported for this graph driver
|
Otherwise you will get the following error:
1
| Error: kernel does not support overlay fs: 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay": backing file system is unsupported for this graph driver
|
- The Podman argument
--cgroup-manager=cgroupfs
This setting needs to be considered when using a PVC as the storage directory. The kernel isolates a group of resources through the Cgroup Driver; the available options are cgroupfs and systemd, and they must be kept consistent with the cluster environment, because they share a single kernel. My test environment uses cgroupfs.
Otherwise you will get the following error:
1
| systemd cgroup flag passed, but systemd support for managing cgroups is not available
|
- The Podman argument
--events-backend=file
This setting usually does not block the execution flow; add it if you want to keep the logs cleaner.
Otherwise you will get the following error:
1
| unable to write system event: "write unixgram @0011c->/run/systemd/journal/socket: sendmsg: no such file or directory
|
4.2 Example 1: Explicitly Using a yaml Template in the Jenkinsfile
Here the container’s /var/lib/containers is mounted to the host directory /var/lib/containers, though it could equally be mounted to the host’s /tmp; there is no hard requirement. The host directory merely provides a place to store the data.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
| pipeline {
agent {
kubernetes {
yaml """
apiVersion: v1
kind: Pod
spec:
containers:
- name: centos
image: centos:7
command:
- cat
tty: true
securityContext:
privileged: true
volumeMounts:
- name: storage
mountPath: /var/lib/containers
volumes:
- name: storage
hostPath:
path: /var/lib/containers
"""
}}
stages {
stage('Hello') {
steps {
container('centos') {
sh '''
curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_7/devel:kubic:libcontainers:stable.repo
yum -y install podman
echo -e 'FROM busybox\nRUN echo "hello world"' | podman --events-backend=file build -t docker.io/shaowenchen1/myimage:latest -
podman --events-backend=file images |grep shaowenchen1
'''
}
}
}
}
}
|
The Jenkins execution log:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| ···
Dependency Updated:
systemd.x86_64 0:219-78.el7_9.3 systemd-libs.x86_64 0:219-78.el7_9.3
Complete!
+ podman --events-backend=file build -t docker.io/shaowenchen1/myimage:latest -
+ echo -e 'FROM busybox
RUN echo "hello world"'
STEP 1: FROM busybox
STEP 2: RUN echo "hello world"
--> Using cache 4c8794086d9de80f71d182457b6d2cb18b9d61975b98bcd4cb167bdcabae5b2c
STEP 3: COMMIT docker.io/shaowenchen1/myimage:latest
--> 4c8794086d9d
4c8794086d9de80f71d182457b6d2cb18b9d61975b98bcd4cb167bdcabae5b2c
+ podman --events-backend=file images
+ grep shaowenchen1
docker.io/shaowenchen1/myimage latest 4c8794086d9d 19 hours ago 1.46 MB
|
4.3 Example 2: Using a PVC to Mount the /var/lib/containers Directory
When using a PVC to store Podman data, the cluster storage must be prepared in advance.
- Check whether the cluster has a default StorageClass
1
2
3
4
5
6
7
| kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
openebs-device openebs.io/local Delete WaitForFirstConsumer false 19d
openebs-hostpath (default) openebs.io/local Delete WaitForFirstConsumer false 19d
openebs-jiva-default openebs.io/provisioner-iscsi Delete Immediate false 19d
openebs-snapshot-promoter volumesnapshot.external-storage.k8s.io/snapshot-promoter Delete Immediate false 19d
|
The namespace here must match the namespace where the dynamic agents in Jenkins run.
1
2
3
4
5
6
7
8
9
10
11
12
13
| cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: storage
namespace: default
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 30Gi
EOF
|
1
2
3
4
| kubectl -n default get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES
storage Pending openebs-hostpath 11s
|
Because the WaitForFirstConsumer mode is used, the PV is bound only once a Pod uses the PVC.
- Create and run the Jenkins pipeline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
| pipeline {
agent {
kubernetes {
yaml """
apiVersion: v1
kind: Pod
spec:
containers:
- name: centos
image: centos:7
command:
- cat
tty: true
securityContext:
privileged: true
volumeMounts:
- name: storage
mountPath: /var/lib/containers
volumes:
- name: storage
persistentVolumeClaim:
claimName: storage
"""
}}
stages {
stage('Hello') {
steps {
container('centos') {
sh '''
curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_7/devel:kubic:libcontainers:stable.repo
yum -y install podman
echo -e 'FROM busybox\nRUN echo "hello world"' | podman --events-backend=file build -t docker.io/shaowenchen2/myimage:latest -
podman --events-backend=file images |grep shaowenchen2
'''
}
}
}
}
}
|
The Jenkins execution log:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| ···
Dependency Updated:
systemd.x86_64 0:219-78.el7_9.3 systemd-libs.x86_64 0:219-78.el7_9.3
Complete!
+ echo -e 'FROM busybox
RUN echo "hello world"'
+ podman --events-backend=file build -t docker.io/shaowenchen2/myimage:latest -
STEP 1: FROM busybox
STEP 2: RUN echo "hello world"
--> Using cache f4676f5b5e47a78970f2d97f4a5b77423f381e9742faae06d8c1a2d93bdb27c2
STEP 3: COMMIT docker.io/shaowenchen2/myimage:latest
--> f4676f5b5e4
f4676f5b5e47a78970f2d97f4a5b77423f381e9742faae06d8c1a2d93bdb27c2
+ podman --events-backend=file images
+ grep shaowenchen2
docker.io/shaowenchen2/myimage latest f4676f5b5e47 2 hours ago 1.46 MB
|
5. Summary
This article mainly offers an approach to building CICD images in a Kubernetes cluster that is not driven by Docker, by replacing Docker with Podman. Podman was chosen because its usage is closer to Docker, whereas Buildah requires users to change their image build commands, since Buildah uses buildah bud.
In production practice, we need to package the Podman commands into the base image of the CI Agent. Through alias docker=podman, pipelines built on Docker commands can be swapped over.
A brief summary of the key points for using Podman:
- Cache support. By mounting the
/var/lib/containers directory, images can be cached, and they can be divided into different directories by business. - Seamless replacement for Docker. Wherever hooks exist, the switch can be made without users noticing.
- More universal. It implements the OCI standard and does not depend on a specific component.
- Privileged mode. Running Podman inside a container requires privileged mode. Container-in-container is a mode that is hard to escape.
6. References
