Unlike on CentOS or Ubuntu, where we feel that mv is faster than cp, when you build an image with a Dockerfile, using Run cp is faster than Run mv. This post presents the test and verification data behind that.
1. Test Setup
Ubuntu 20.04.1 LTS
32C
125Gi
Since this is a production machine, it carries some load, so the tests will have variance. I run the test multiple times and sample the results once they stabilize.
1
2
3
| ls
main lib i100 cc
|
1
2
3
4
5
6
7
| du -h --max-depth=1
2.7M ./i100
47M ./main
808K ./cc
424M ./lib
474M .
|
- Total number of files and directories
1
2
3
| ls -lR| wc -l
42978
|
2. Building with the Run mv Command
Dockerfile contents
1
2
3
4
5
6
7
| FROM golang:1.13
COPY ./ /go/src/code
RUN mkdir /a && mv /go/src/code/cc/* /a/ \
&& mkdir /b && mv /go/src/code/lib/* /b/ \
&& mkdir /c && mv /go/src/code/i100/resource/* /c/ \
&& mkdir /d && mv /go/src/code/main/* /d/
|
Build the image
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| DOCKER_BUILDKIT=1 docker build --no-cache -t test:v1 -f ./Dockerfile1 .
[+] Building 78.0s (8/8) FINISHED
=> [internal] load build definition from Dockerfile1 0.0s
=> => transferring dockerfile: 334B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for golang:1.13 0.0s
=> CACHED [1/3] FROM golang:1.13 0.0s
=> [internal] load build context 2.0s
=> => transferring context: 3.05MB 1.9s
=> [2/3] COPY ./ /go/src/code 4.7s
=> [3/3] RUN mkdir /a && mv /go/src/code/cc/* /a/ && mkdir /b && mv /go/src/code/lib/* /b/ && mkdir /c && mv /go/src/code/i100/resource/* /c/ && mkdir /d && mv /go/src/code/main/* /d/ 57.6s
=> exporting to image 13.6s
=> => exporting layers 13.6s
=> => writing image sha256:973b97d407a6403132d279f2c8ac713268ada69fe067e355700efa650ff65d8b 0.0s
=> => naming to docker.io/library/test:v1 0.0s
|
Run mv took 57.6s.
3. Building with the Run cp Command
Dockerfile contents
1
2
3
4
5
6
7
| FROM golang:1.13
COPY ./ /go/src/code
RUN mkdir /a && cp -R /go/src/code/cc/* /a/ \
&& mkdir /b && cp -R /go/src/code/lib/* /b/ \
&& mkdir /c && cp -R /go/src/code/i100/resource/* /c/ \
&& mkdir /d && cp -R /go/src/code/main/* /d/
|
Build the image
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| DOCKER_BUILDKIT=1 docker build --no-cache -t test:v1 -f ./Dockerfile2 .
[+] Building 26.2s (8/8) FINISHED
=> [internal] load build definition from Dockerfile2 0.0s
=> => transferring dockerfile: 282B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for golang:1.13 0.0s
=> [internal] load build context 2.0s
=> => transferring context: 3.05MB 2.0s
=> CACHED [1/3] FROM golang:1.13 0.0s
=> [2/3] COPY ./ /go/src/code 5.4s
=> [3/3] RUN cp -R /go/src/code/cc / && cp -R /go/src/code/lib / && cp -R /go/src/code/i100/resource / && cp -R /go/src/code/main / 5.1s
=> exporting to image 13.5s
=> => exporting layers 13.4s
=> => writing image sha256:bd3c53ac40006a79ec009b6112fdcfec85e0adef6d0fcf6aa65d3ee02b2e202a 0.0s
=> => naming to docker.io/library/test:v1 0.0s
|
Run cp took 5.1s.
4. Tracing the mv and cp Commands with strace
The statistics after a large number of runs are not given above, but the result reproduces reliably across repeated runs: with the same project, the RUN cp command is far more efficient at building the image than the RUN mv command.
- On Ubuntu, when the source and destination are on the same filesystem
The mv command first tries to call rename for a fast move, and only falls back to cp-style copying after that fails. Below are some of the system calls traced with strace.
1
2
3
4
5
6
7
8
| strace cp -R main main1
read(3, "# -*- coding: utf-8 -*-\n\n\"\"\"\n@ve"..., 131072) = 8425
write(4, "# -*- coding: utf-8 -*-\n\n\"\"\"\n@ve"..., 8425) = 8425
mkdir("main1/scripts/update_oauth", 0755) = 0
lstat("main1/scripts/update_oauth", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
openat(AT_FDCWD, "main/scripts/update_oauth", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
fstat(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
|
1
2
3
4
5
| strace mv main1 main2
ioctl(0, TCGETS, {B9600 opost isig icanon echo ...}) = 0
renameat2(AT_FDCWD, "main1", AT_FDCWD, "main2", RENAME_NOREPLACE) = 0
lseek(0, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
|
You can see that mv calls renameat2 directly, skipping the file-copy operation.
- On Ubuntu, when the source and destination are not on the same filesystem
In this case the mv command not only has to copy the files, it also has to unlink (delete) them, which is one more step than cp. If there are many files, the unlink calls become very expensive.
/data is mounted on a separate disk
1
2
3
4
5
| strace cp -R main /data/main1
lstat("main/upgrade/1.0.1/README.MD", {st_mode=S_IFREG|0644, st_size=1599, ...}) = 0
openat(AT_FDCWD, "main/upgrade/1.0.1/README.MD", O_RDONLY|O_NOFOLLOW) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=1599, ...}) = 0
|
1
2
3
4
5
6
7
| strace mv main /data/main2
renameat2(AT_FDCWD, "main/upgrade/1.0.7/README.MD", AT_FDCWD, "/data/main2/upgrade/1.0.7/README.MD", RENAME_NOREPLACE) = -1 EXDEV (Invalid cross-device link)
lstat("main/upgrade/1.0.7/README.MD", {st_mode=S_IFREG|0644, st_size=1280, ...}) = 0
newfstatat(AT_FDCWD, "/data/main2/upgrade/1.0.7/README.MD", 0x7fff09ac3810, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
unlink("/data/main2/upgrade/1.0.7/README.MD") = -1 ENOENT (No such file or directory)
|
mv tries rename, but it fails, so it can only fall back to cp mode.
- The Run mv command in a Dockerfile
Since the base image above does not have the strace command, here we install strace on top of the test:v1 image and commit it again; the specific steps are omitted.
Dockerfile contents
1
2
3
4
5
6
| FROM test:v1
RUN strace mv /a /a1 \
&& strace mv /b /b1 \
&& strace mv /c /c1 \
&& strace mv /d /d1
|
Build the image
1
2
3
4
5
6
| DOCKER_BUILDKIT=1 docker build --no-cache -t test:v2 -f ./Dockerfile . --progress=plain
#5 0.435 renameat2(AT_FDCWD, "/a", AT_FDCWD, "/a1", RENAME_NOREPLACE) = -1 EXDEV (Invalid cross-device link)
#5 0.436 newfstatat(AT_FDCWD, "/a/CHANGES.txt", {st_mode=S_IFREG|0644, st_size=435, ...}, AT_SYMLINK_NOFOLLOW) = 0
#5 0.436 newfstatat(AT_FDCWD, "/a1/CHANGES.txt", 0x7ffeb191e6d0, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
#5 0.436 unlink("/a1/CHANGES.txt") = -1 ENOENT (No such file or directory)
|
Invalid cross-device link shows that the Run mv call inside a Dockerfile cannot make rename succeed. In other words, Run mv in a Dockerfile = Run cp + Run unlink.
5. Summary
In our production CICD system, some pipelines were very slow to build when they executed Run mv, and we could not find the reason. This post mainly analyzes that problem and gives a solution: Run cp can replace Run mv. For build projects with a large number of files, this produces a significant speedup. If the number of files is small, this optimization can be ignored. The specific conclusions are:
- On the same filesystem, mv is faster than cp
- On different filesystems, cp is faster than mv
- In a Dockerfile, Run cp is faster than mv; in the example above it dropped from 57s to 5s
That raises the question: how does the Docker Daemon handle a Dockerfile? And do other tools, such as Kaniko, have a similar problem? Why is unlink inside a Dockerfile more expensive than unlink on the host?
In this case, a second optimization is to replace Run mv with the Copy and Add commands, avoiding Run-based file operations in the Dockerfile; a third optimization is to note that a project rarely reaches 40k files, so you can use .dockerignore to keep unneeded files out of the build context — for example .git, node_modules, vendor, .m2, and so on — to reduce unlink time.
6. References
