1. DockerHub Limits

In the end, there is no getting around this error:

1

Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Starting November 2, 2020, DockerHub officially began limiting the pull rate for non-paying users:

• Anonymous users: only 100 pulls every 6 hours

• Logged-in users: only 200 pulls every 6 hours

Alright, under normal circumstances this would be the end of the friendship — if you won’t let me use it, I just won’t. But take another look at this diagram:

For some teams, DockerHub is more than image storage — more importantly, it serves as a distribution center. Every time an image is built, it is pushed straight to DockerHub, and then other places sync the image from there.

So the DockerHub pull limit had to be dealt with.

2. How to Test

2.1 How Images Are Pulled

Before testing, let’s first understand how an image is pulled. The diagram below shows the structure of an image: one image corresponds to one Manifest, i.e. the JSON structure shown here.

To make this more concrete, let’s enable Docker’s experimental feature and inspect the manifest of the nginx image.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

export DOCKER_CLI_EXPERIMENTAL=enabled
docker manifest inspect --verbose nginx

```json
[
	{
		"Ref": "docker.io/library/nginx:latest@sha256:99d0a53e3718cef59443558607d1e100b325d6a2b678cd2a48b05e5e22ffeb49",
		"SchemaV2Manifest": {
			"schemaVersion": 2,
			"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
			"config": {
				"mediaType": "application/vnd.docker.container.image.v1+json",
				"size": 7480,
				"digest": "sha256:bc9a0695f5712dcaaa09a5adc415a3936ccba13fc2587dfd76b1b8aeea3f221c"
			},
			"layers": [
				{
					"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
					"size": 27105484,
					"digest": "sha256:852e50cd189dfeb54d97680d9fa6bed21a6d7d18cfb56d6abfe2de9d7f173795"
				},
...
]

As you can see, the Manifest records a lot of fingerprint data for the Layers; the Layer is the actual image layer data, while the Manifest only records metadata. Pulling an image happens in two steps:

1. Pull the Manifest
2. Pull the layer data according to the description in the Manifest. If a local cache exists, no request is made.

2.2 Test Script

1. Check the current DockerHub pull quota

If you are a logged-in user, you can add the --user 'username:password' argument after curl.

1
2
3
4
5
6

TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:shaowenchen/dockerhub-ratelimit:pull" | jq -r .token)
curl --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/shaowenchen/dockerhub-ratelimit/manifests/1

ratelimit-limit: 100;w=21600
ratelimit-remaining: 99;w=21600
docker-ratelimit-source: 1.2.3.4

docker-ratelimit-source indicates the restricted IP.
ratelimit-remaining is the remaining pull count within 21600 seconds, i.e. 6 hours.

Only pulling image layer data consumes the quota, so pulling an image you have already pulled does not consume it.

2. Exhaust the quota with a script

1
2
3
4
5

number=1 ; \
while [[ $number -le 200 ]] ; do \
    docker pull shaowenchen/dockerhub-ratelimit:$number ; \
    ((number = number + 1)) ; \
done

1
2
3
4
5

number=1 ; \
while [[ $number -le 200 ]] ; do \
    docker rmi shaowenchen/dockerhub-ratelimit:$number ; \
    ((number = number + 1)) ; \
done

3. After configuring, continue pulling images

There are mainly three categories of images here:

• The first is an image that exists locally

1

docker pull shaowenchen/docker-robotframework

• The second is a public image (not present locally)

1

docker pull python

• The third is an image you built yourself (not present locally)

1

docker pull shaowenchen/s2ipy

3. Configure an Image Mirror to Lift the Pull Limit

3.1 Available Mirrors and Test Results

Overall, using the NetEase or Tencent mirror is a good choice.

3.2 Configuration

• Edit Docker’s daemon file

1

vim /etc/docker/daemon.json

• Add the NetEase mirror

1
2
3
4
5
6

{
  "registry-mirrors": [
    "https://hub-mirror.c.163.com"
  ],
  "live-restore": true
}

• Restart the Docker service

1

systemctl daemon-reload && systemctl restart docker