How to Debug Docker Layer Cache Misses in a Build?



We have a known issue where Docker Buildx does not work with Docker Layer Cache (DLC).
If you are using Docker Buildx and want to cache image layers, please use the --cache-from and --cache-to options instead.
For a guide on how to do so, please see this Discuss post.

Before We Start

Typically, we are trying to compare two builds from the same project.
Let us call them build A and build B, where the following is true:

  • build A ran before build B.
  • both build A and B are creating an image from the same, unchanged Dockerfile.
  • we are expecting build B to leverage some, if not all, cached layers from build A.

If the Dockerfile has been modified, this can cause the cache to bust, and result in cache misses.
Please refer to Docker documentation on the best practices for writing a Dockerfile.


Step 1: Confirm DLC is Enabled

As a quick check, make sure that you have enabled DLC on your build indeed.
Depending on your job's executor, you should have enabled DLC on either the Remote Docker (for Docker executor) or the Machine image definition itself (for Machine executor).


When DLC is enabled, confirm that a volume (specifically, by volume ID) is assigned for the build.



Inspect the volume ID in the DLC-enabled Remote Docker engine for a Docker executor build.


Inspect the volume ID for a DLC-enabled Machine executor build.


Step 2: Verify if the Same Volume Was Used


Check that the volume assigned to both build A and B are the same (i.e, same volume ID).

If they are not the same, we will likely not see 100% cache hits for all layers.
For example, all layers of this Docker image in build B would need to be built from the start if a fresh volume for this project was assigned.

Because assigned volumes may be different, a full DLC cache hit thus is not a guarantee.

CircleCI assigns a volume to a build based on availability and recency of use.

In other words, currently, there is no way to pin any affinity between a specific volume and a build.

Please refer to our documentation on how volumes are assigned.



Was this article helpful?
0 out of 3 found this helpful



Article is closed for comments.