Troubleshooting caching in the Docker Executor "Spin Up Environment" Step?

Note: This article discusses caching in the Docker Executor relating to the "Spin Up Environment" step for the main container in the job. It does not apply to Docker Layer Caching which is a feature of the Remote Docker environment.

The time it takes to spin up a docker container to run a job can vary based on several different factors, such as the size of the image and if some, or all, of the layers, are already cached on the underlying Docker host machine.

Generally, if you are using a popular image, such as CircleCI Convenience Images, then cache hits are more likely for a larger number of layers. Most of our popular CircleCI images use the same base image so the majority of the base layers will be the same between images and you, therefore, you have a greater chance of having a cache hit.

The environment has to spin up for every new job, regardless of whether it is in the same workflow or if it is a re-run/subsequent run - for security reasons, we never reuse containers. Once the job is finished the container is destroyed. We cannot guarantee that jobs, even in the same workflow, will run on the same docker host machine and therefore the cache status may differ.

Cache hits are not guaranteed, but are a bonus convenience when available. With this in mind, a worst-case scenario of a full image pull should be accounted for in all jobs.

Docker Layer Caching is only beneficial when building an image on CircleCI by using the remote docker environment - it does not affect spin-up times for the actual docker containers used to run the job. More information about DLC can be found in our documentation.

Extending Convenience Images

If there aren't Convenience Images (cimg/) available that meet all of your build's needs, you can build custom images based on a Convenience Image and add necessary additions, such as dependencies. By doing this, you can increase the chances of hitting a cache and spinning up your jobs faster than if you were to build an entirely new image. To do this, you can add the following to the top of your Dockerfile:

FROM cimg/<image>

In summary, the availability of caching is not something that can be controlled via settings or configuration, but by choosing a popular image, such as CircleCI Convenience Images, you will have more cache of hitting cached layers in the "Spin Up Environment" Step.