Docker Hub Rate Limiting FAQ

Beginning November 1, 2020, Docker Hub will enable rate limits based on the originating IP address. However, CircleCI has partnered with Docker to ensure that our users can continue to access Docker Hub without rate limits. On November 1st, with few exceptions* (listed below), you should not be impacted by any rate limits when pulling images from Docker Hub through CircleCI.

However, these rate limits may go into effect for CircleCI users in the future. That’s why we’re encouraging you and your team to add Docker Hub authentication to your CircleCI configuration and consider upgrading your Docker Hub plan, as appropriate, to prevent any impact from rate limits in the future.

We have additional information and up-to-date details on the "Authenticate with Docker to avoid impact of Nov. 1st rate limits" Discuss post.

*Exceptions: Remote Docker and Machine Executors will be impacted by the rate limiting unless pulling CircleCI-published images.

Below we will describe frequently asked questions on how this affects your CircleCI jobs and how to handle each situation. This change does affect both Cloud and Enterprise CircleCI customers.

How do I authenticate with Docker Hub?

You will need to pass an auth key with your Docker image and utilize a username and password. Specific steps can be found here: https://circleci.com/docs/2.0/private-images/

How can I confirm my authentication was successful?

If you have passed the auth key on your Docker pulls and the pulls continue to be successful, such as on the "Spin up environment" step, your credentials are being successfully used. At this time we don't have a way to see in the UI the authenticated credentials.

However, if you are passing the key and the wrong credentials are set, you will receive an error and the build will fail.

Note: CircleCi shows the warning message in the spin up environment step if you have not added auth key yet.

>Warning: No authentication provided, this pull may be subject to rate limits.

My builds use the Docker executor, does this affect me?

No, as mentioned previously CircleCI has partnered with Docker to ensure that our users can continue to access Docker Hub without rate limits.

As mentioned in the exceptions section though, the use of Remote Docker with a non-CircleCI convenience image is affected by these rate limits and you will want to ensure you authenticate.

My builds use the Machine executor, does this affect me?

Maybe, if you utilize Docker (i.e. pull images) within your Machine executor jobs this could affect you. As mentioned in the above exceptions, if you are using a non-CircleCI image you will be impacted.

You will need to use the docker login -u username -p password command with your credentials to log in to Docker Hub, before running the docker commands that potentially pull images; such as, docker pull, docker run, or docker build.

I use CircleCI convenience images, does this affect me?

No, as mentioned previously CircleCI has partnered with Docker to ensure that our users can continue to access Docker Hub without rate limits.

I use custom Docker images, does this affect me?

If the custom Docker images are being used in a Remote Docker or Machine executor, yes, they are affected.

However, custom docker images used with the Docker executor won't be affected because of the previously mentioned partnership.

Although, these rate limits may go into effect for CircleCI users in the future and we are still encouraging users to authenticate.

I use CircleCI Enterprise (Server), does this affect me?

Yes -- this change affects both Cloud and Enterprise CircleCI customers.

The pulls of circleci and cimg images are exempt from the rate limit even for customers outside of the CircleCI environment. However, it still makes sense for Server customers to authenticate with their Docker Hub accounts (through proxy or otherwise) as they probably also pull other, non-CircleCI images from Docker Hub.

Server customers can set up a Docker Hub pull-through registry mirror pre-configured with Docker Hub account credentials. Know more!

What actions can I take ahead of November 1st, 2020?

Depending on your CircleCI setup, you may not need to take action. Please reference the other FAQ sections to see if rate limiting will be applied to your builds.

If you do need to take action, it would be to set up Docker authentication in your projects, we have more information on those actions in the "My build fails to pull Docker image with "Too Many Requests" - Rate Limiting by Docker Hub" article.

Can I update all of my projects that require Docker authentication en masse?

At the moment each project will need to be updated individually.

If I am blocked after the change happens, what actions can I take?

If you haven't set up the authentication described above, that would be the first step.

If you have set up authentication and hit the limits, you will either want to change how often you are building or explore Docker Hub's plans that allow for unlimited pulls.

How can I authenticate on OSS projects?

If the OSS projects utilize machine and remote docker you can add some conditional logic that will authenticate based on the user's access levels:

- run: 
    name: Docker Hub login if available
    command: |
      if [ -n "$DOCKERHUB_USERNAME" ]; then
        docker login --username $DOCKERHUB_USERNAME --password $DOCKERHUB_PASSWORD
      else
        echo "Docker Hub credentials unavailable, pulling anonymously"
      fi

With the above, you would store the username and password either in a context or as project-level environment variables. For users who have access to the main repository as a collaborator, their builds will authenticate with the credentials stored. For someone who submits a forked PR to the project, and if the project has "pass secrets to forked pull requests" turned off, the pull/push would be done anonymously.