Open in app

Sign in

Write

Sign in

Mitigate Docker Hub rate limit for free

Ángel Barrera Sánchez
7 min readNov 23, 2020

Some weeks ago, Docker Inc announced its new service limits to its container registry: Docker Hub. It was a surprise for most of the vendors and final users.

These limits hit to all Docker Hub consumers, from developers, CI systems, and Kubernetes clusters.

There are multiple alternatives to mitigate these limits. The easiest way to work around them is to pay to Docker Inc a monthly subscription then using your credentials to pull images.

If you don’t want to pay to Docker Inc, you can still use a free account, which increases 2x the rate limit, to pull images from Docker Hub.

Meanwhile, Google Cloud has a free cache mirror for Docker Hub. Please take a look at the following link to know more details about it: https://cloud.google.com/container-registry/docs/pulling-cached-images

By mixing both strategies, it is possible to mitigate the new limits Docker is setting to its public registry.

Hands-on

The main goal is to mix both options: a working Docker Hub mirror (mirror.gcr.io) plus having a bigger pull rate limit due to the authenticated pulls requests to Docker Hub.

Create Docker Hub credentials

The first action to execute is to create a free Docker Hub account. Once created, navigate to https://hub.docker.com/settings/security then generate an access token:

Save the generated credentials in a safe location; you will need them later.

This step is not entirely required but highly recommended to have an easy way to invalidate old credentials.

Configure docker registry mirrors

To configure your Docker daemon to pull images from the Google Cloud Container Registry cache configure the daemon in one of the following ways:

  • To configure the Docker daemon automatically on startup, set the next value in /etc/docker/daemon.json
{
 “registry-mirrors”: [“https://mirror.gcr.io"]
}
  • When you start the daemon, pass in the Container Registry hostname:
dockerd — registry-mirror=https://mirror.gcr.io
  • Add the following line to your /etc/default/docker file:
DOCKER_OPTS=”${DOCKER_OPTS} — registry-mirror=https://mirror.gcr.io"

Finally, restart the Docker daemon.

sudo service docker restart

or

sudo service docker stop && sudo service docker start

Verify the configuration by running:

$ docker system info | tail -4
WARNING: No swap limit support
 Registry Mirrors:
  https://mirror.gcr.io/
 Live Restore Enabled: false

Test it downloading a commonly used container image:

$ docker pull python:3.8
3.8: Pulling from library/python
756975cb9c7e: Pull complete
d77915b4e630: Pull complete
5f37a0a41b6b: Pull complete
96b2c1e36db5: Pull complete
c495e8de12d2: Pull complete
33382189822a: Pull complete
b208f1fbe418: Pull complete
eff0ed295004: Pull complete
ee9028950cff: Pull complete
Digest: sha256:8e7dd58a4cb8a8b703ae2fe90cd67e79576524ff388426331151b50a5850b359
Status: Downloaded newer image for python:3.8
docker.io/library/python:3.8

Then pull a more specific python container image:

$ docker pull python:3.7.9-slim-stretch
3.7.9-slim-stretch: Pulling from library/python
4297e0229558: Pull complete
17bfea12ab8f: Pull complete
7bc9a29233a0: Pull complete
f8498c38db7f: Pull complete
d7c705bcba75: Pull complete
Digest: sha256:9e4ddd6a005934552b21f188541a880f6bf22d2d6b08eb7f4082a9effc4d166f
Status: Downloaded newer image for python:3.7.9-slim-stretch
docker.io/library/python:3.7.9-slim-stretch

We can smell a small difference. It takes longer to start the download than before. Why? If we look at the logs in the system journals’:

$ journalctl -u docker -n 1 --no-pager
-- Logs begin at Sat 2020-10-17 13:00:48 CEST, end at Thu 2020-11-19 15:30:01 CET. --
nov 19 15:27:41 angel-nuc-i3 dockerd[236668]: time="2020-11-19T15:27:41.045133927+01:00" level=info msg="Attempting next endpoint for pull after error: manifest unknown: Failed to fetch \"3.7.9-slim-stretch\" from request \"/v2/library/python/manifests/3.7.9-slim-stretch\"."

It means that the image was not cached at the Google Cloud mirror and it tries to download using the original Docker Hub URI.

Note that the python:3.8 (which is most used than python:3.7.9-slim-stretch) got downloaded directly from the Google Cloud cache registry witch does not count against Docker Hub rate limits.

Configure Docker Hub credentials

Currently, 100 pull request to the Docker Hub registry per 6 hours is allowed for unauthenticated clients if you authenticate your pull requests, the limit increase up to 200 pull request for 6 hours.

source: https://www.docker.com/pricing

So, to increase the rate limit established for the unauthenticated pull request to Docker Hub is highly recommended to log into your Docker Hub account.

$ docker login docker.io -u <THE_USERNAME> -p <THE_PASSWORD>
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded
$ cat ~/.docker/config.json
{
 "auths": {
  "https://index.docker.io/v1/": {
   "auth": "YW5nZWxiYXJyZXJhOTI6bm90LXRoZS1yZWFsLXBhc3N3b3JkICA6KQo="
  }
 },
 "HttpHeaders": {
  "User-Agent": "Docker-Client/19.03.13 (linux)"
 }
}

Then every pull request to Docker Hub is authenticated.

Test the setup

Let’s pull another standard container image to test everything continues working:

$ docker pull node:14
14: Pulling from library/node
7919f5b7d602: Pull complete
0e107167dcc5: Pull complete
66a456bba435: Pull complete
5435318a0426: Pull complete
8494dd328465: Pull complete
3b01939c6506: Pull complete
1ce5dd1bc774: Pull complete
1184d3e0bcb0: Pull complete
1e0791c658e9: Pull complete
Digest: sha256:922f9cbc7d960750d3db015e7520f41957130caf8bd16138b29f518aa4ba43a7
Status: Downloaded newer image for node:14
docker.io/library/node:14

Wait a minute; it takes longer than expected to start the download; let’s see the system journal again:

$ journalctl -u docker -n 1 --no-pager
-- Logs begin at Sat 2020-10-17 13:00:48 CEST, end at Thu 2020-11-19 16:52:06 CET. --
nov 19 16:52:06 angel-nuc-i3 dockerd[236668]: time="2020-11-19T16:52:06.628385170+01:00" level=info msg="Attempting next endpoint for pull after error: Get https://mirror.gcr.io/v2/library/node/manifests/14: unauthorized: Not Authorized."

The docker cli is attempting to authenticate also the mirror pull request. It fails, so it ends up hitting the rate limit at Docker Hub.

Photo by Alexas Fotos from Pexels

Fix the setup

To fix this weird situation being benefit of having authenticated requests against Docker Hub (with 200 pull request per 6 hours instead of 100) and the Google Cloud cache mirror working, create a local reverse proxy to Google Cloud cache in your instance.

First, create the following configuration file:

http://:2020 {
    log {
        level info
    }reverse_proxy https://mirror.gcr.io {
        header_up Host mirror.gcr.io
        header_up -X-Forwarded-For
        header_up -X-Forwarded-Proto
        header_up -Authorization
        header_up -Www-Authenticate
        header_down -Www-Authenticate
    }
}

The configuration removes any authentication/proxy related header as the docker cli send it, and the google cloud cache registry doesn’t need them.

Then run a Caddy container locally with the above configuration file (Caddyfile):

$ docker run --name registry-mirror-proxy --restart always -d -p 2020:2020 -v $(pwd)/Caddyfile:/etc/caddy/Caddyfile caddy
Unable to find image 'caddy:latest' locally
latest: Pulling from library/caddy
188c0c94c7c5: Pull complete
da2e2d825895: Pull complete
2aaf2e3dee2e: Pull complete
b2215cf93250: Pull complete
060365e49ba5: Pull complete
Digest: sha256:e46050590a9ac1a411b5da0349f735b5977e9dcbe6ab91e9a95c9a731533aba8
Status: Downloaded newer image for caddy:latest
87dfd3e69c57508c6ff7f410dea3401cb20e2219c889b229f594e5bb0eef2480
$ docker logs -f registry-mirror-proxy
{"level":"info","ts":1605801934.784401,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"info","ts":1605801934.787342,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["[::1]:2019","127.0.0.1:2019","localhost:2019"]}
{"level":"info","ts":1605801934.7879956,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003ccd20"}
{"level":"info","ts":1605801934.7882853,"logger":"tls","msg":"cleaned up storage units"}
{"level":"info","ts":1605801934.7888107,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1605801934.7888331,"msg":"serving initial configuration"}

The proxy is configured. You can check if it works querying an example image metadata (alpine/git):

$ curl localhost:2020/v2/alpine/git/manifests/latest
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 947,
         "digest": "sha256:8715680f27333935bb384a678256faf8e8832a5f2a0d4a00c9d481111c5a29c0",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 947,
         "digest": "sha256:402c8dc1266551b05bfe63c44efc6cd129cce1193058feb8ca1766617c1af911",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 946,
         "digest": "sha256:88e06aad911902679d00ae319225d23eac98319a35f9c92d74b5f6d1aad63c1e",
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      }
   ]
}

Finally, reconfigure the docker registry mirrors by changing https://mirror.gcr.io with http://localhost:2020 in the /etc/docker/daemon.json configuration file:

{
  "registry-mirrors": ["http://localhost:2020"]
}

To apply the configuration, don’t forget to restart the docker daemon by running:

sudo service docker restart

Test the setup by downloading another high demanded container image:

$ docker pull alpine:3.12
3.12: Pulling from library/alpine
188c0c94c7c5: Already exists
Digest: sha256:c0e9560cda118f9ec63ddefb4a173a2b2a0347082d7dff7dc14272e7841a5b5a
Status: Downloaded newer image for alpine:3.12
docker.io/library/alpine:3.12

Check the journal:

$ journalctl -u  docker -n 1 --no-pager
-- Logs begin at Sat 2020-10-17 13:00:48 CEST, end at Thu 2020-11-19 17:17:01 CET. --
nov 19 17:16:40 angel-nuc-i3 systemd[1]: Started Docker Application Container Engine.

No issue found. Check the proxy logs:

$ docker logs --tail 1 registry-mirror-proxy
{"level":"info","ts":1605802631.0372276,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"172.17.0.1:60812","proto":"HTTP/1.1","method":"GET","host":"localhost:2020","uri":"/v2/library/alpine/blobs/sha256:d6e46aa2470df1d32034c6707c8041158b652f38d2a9ae3d7ad7e7532d22ebe0","headers":{"User-Agent":["docker/19.03.13 go/go1.13.15 git-commit/4484c46d9d kernel/5.4.0-53-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.13 \\(linux\\))"],"Accept-Encoding":["identity"],"Connection":["close"]}},"common_log":"172.17.0.1 - - [19/Nov/2020:16:17:11 +0000] \"GET /v2/library/alpine/blobs/sha256:d6e46aa2470df1d32034c6707c8041158b652f38d2a9ae3d7ad7e7532d22ebe0 HTTP/1.1\" 302 13","duration":0.052763726,"size":13,"status":302,"resp_headers":{"Server":["Caddy","Docker Registry"],"Date":["Thu, 19 Nov 2020 16:17:11 GMT"],"Cache-Control":["private"],"Vary":["Accept-Encoding"],"X-Frame-Options":["SAMEORIGIN"],"Accept-Ranges":["none"],"Docker-Distribution-Api-Version":["registry/2.0"],"Content-Type":["application/json"],"X-Xss-Protection":["0"],"Location":["https://storage.googleapis.com/eu.artifacts.cloud-containers-mirror.appspot.com/containers/images/sha256:d6e46aa2470df1d32034c6707c8041158b652f38d2a9ae3d7ad7e7532d22ebe0"]}}

There we can see the request flowing through it without credentials. We can assume everything is working now.

Conclusion

Docker inc designed these rate limit carefully to allow developers works without being affected, meanwhile most of the current CI systems are affected by these limits as they need to pull images to perform continuous integration actions.

By combining the usage of the free Google Cloud registry mirror along with authenticated pull request to Docker Hub, we can confidently run any continuous integration system.

Please, have in mind the following statements:

  • Only the most used container images are cached in the Google Cloud mirror registry.
  • Paying 5$ per month to Docker inc makes it possible for you to forget about mirror configuration. (still requires to set up credentials)
  • The setup described in this blog post does not guarantee you are not going to hit the limit at Docker Hub.
Docker
Dockerhub
Kubernetes
Cicd
Limits

--

--

Ángel Barrera Sánchez

Written by Ángel Barrera Sánchez

59 Followers

eps1.0_hellofriend.mov

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams