1. Picking Up from Last Time: Shared Storage Optimizes Pulling Images Overseas

Building on An Image Management and Distribution Case Based on Harbor and Registry, I recently made another optimization.

The previous approach was to deploy a Mirror Cache in each region on a machine with low specs and a large disk to cache images. This introduced a problem: every region needed to pull the image. With N regions, publishing one application meant pulling N times, and the dedicated-line bandwidth used during publishing was multiplied by N.

What was overlooked here is that overseas cloud vendors generally have very fast network speeds. We can partition the overseas regions by connectivity quality and deploy a Mirror Cache in each zone.

Another operational problem is how to expand the disk and clean up images on the host running Mirror Cache. Although Registry lets you set a cache period, the community reports that the setting does not take effect; and the disk can only be expanded up to 1T, after which you have to migrate to a data disk. So a different approach was adopted here: sharing backend storage among multiple Mirror Caches. As shown below:

The advantages of using the same OBS backend storage:

• Publishing a multi-region application requires only one pull, instead of one pull per region
• Object storage such as OBS and S3 scales and stays stable better than local disks
• Even without CDN acceleration, the object storage services of the big cloud vendors are not slow anywhere in the world
• Object storage supports lifecycle management, which can be used to automatically clean up image data

Since all image data is cached in OBS object storage, Mirror Cache is in effect a stateless service. Going a step further, we can merge it into a single instance placed in the region with good connectivity to all the other zones.

Now on to the topic of this article: one experience of failing to pull a large image.

2. Failing When Pulling a Large Image

The image is about 10G in size. The pull path is as follows:

Docker Client -> Docker Daemon -> Mirror Cache -> LB -> Harbor

The specific errors from each component are as follows:

• While pulling the image, the Docker Client kept retrying the large single-layer image

1

2a57ddc33014: Downloading [=======================>                           ]  646.6MB/1.378GB

When the progress bar reached close to 70%-80%, this image layer started downloading again:

1

2a57ddc33014: Retrying in 2 seconds

• Docker Daemon error log on the pulling host

1
2
3

Sep 08 01:12:08 x.x.x.x dockerd[8434]: time="2022-09-08T01:12:08.681917626Z" level=info msg="Download failed, retrying (2/5): expected HTTP 206 from byte range request"
Sep 08 01:19:17 x.x.x.x dockerd[8434]: time="2022-09-08T01:19:17.196665587Z" level=info msg="Download failed, retrying (3/5): unexpected EOF"
Sep 08 01:19:20 x.x.x.x dockerd[8434]: time="2022-09-08T01:19:20.197654060Z" level=error msg="Not continuing with pull after error: context canceled"

• Error log of the image cache service Mirror Cache

1
2
3

linux arch/amd64 UpstreamClient(Docker-Client/20.10.13 \\(linux\\))"
time="2022-09-08T02:30:50.580118978Z" level=error msg="Error committing to storage: stream error: stream ID 17; INTERNAL_ERROR; received from peer" auth.user.name=x go.version=go1.16.15 http.request.host=harbor.chenshaowen.com http.request.id=d51a853a-447a-44bf-9e0e-f015fb4c9bd8 http.request.method=GET http.request.remoteaddr=x.x.x.x http.request.uri="/v2/x/x/blobs/sha256:2c618d17e28be23c5c38b084bfb3f288371bbed89181e57f7cf43f535261e354" http.request.useragent="docker/18.09.7 go/go1.10.8 git-commit/2d0083d kernel/3.10.0-1160.53.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.13 \(linux\))" vars.digest="sha256:2c618d17e28be23c5c38b084bfb3f288371bbed89181e57f7cf43f535261e354" vars.name="x/x"
time="2022-09-08T02:30:50.580404499Z" level=error msg="response completed with error" auth.user.name=x err.code=unknown err.detail="stream error: stream ID 15; INTERNAL_ERROR; received from peer" err.message="unknown error" go.version=go1.16.15 http.request.host=x http.request.id=d51a853a-447a-44bf-9e0e-f015fb4c9bd8 http.request.method=GET http.request.remoteaddr=x.x.x.x http.request.uri="/v2/x/x/blobs/sha256:2c618d17e28be23c5c38b084bfb3f288371bbed89181e57f7cf43f535261e354" http.request.useragent="docker/18.09.7 go/go1.10.8 git-commit/2d0083d kernel/3.10.0-1160.53.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.13 \(linux\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=3m30.111855949s http.response.status=500 http.response.written=1091061683 vars.digest="sha256:2c618d17e28be23c5c38b084bfb3f288371bbed89181e57f7cf43f535261e354" vars.name="x/x"

• LB error

1

(104: Connection reset by peer) while reading upstream

Characteristics of a large image that cannot be pulled:

• Smaller image layers pull successfully; only a few large single-layer images fail
• Pulling the large image layer keeps retrying
• Other small images pull successfully

Combined with the logs above, my judgment was that a timeout configured on some component was causing the connection to drop after a while. There are plenty of articles online about the LB error log, and the way to solve (104: Connection reset by peer) is to increase the buffer or the timeout.

The timeout here fits that judgment exactly. The LB’s proxy_connect_timeout, proxy_send_timeout, and proxy_read_timeout were all 60s, but for various reasons the LB service could not have these parameters changed, so I had to find another way. If you run into a similar situation, you can try changing these parameter values on the LB.

In the end, I skipped the LB entirely and connected Mirror Cache directly to Harbor. The pull path is as follows:

Docker Client -> Docker Daemon -> Mirror Cache -> Harbor

Without LB forwarding, this not only saved the related costs but simplified the architecture and reduced operational overhead. More importantly, large images could now be pulled, and the endless retrying never happened again.

3. Reference

• https://docs.docker.com/registry/configuration/