Docker builds fail after a machine image update (DLC)

Overview

CircleCI Docker Layer Caching (DLC) stores Docker image layers so later builds can reuse them. DLC is shared at the organization level (15 GiB limit across projects that use it).

When CircleCI promotes a newer machine executor image, that image includes a newer Docker engine. Layer data written under an older engine is not always reusable under the new one. Builds can then fail even when DLC restore completes without error.

This is different from:

  • DLC not restoring (restore step fails or times out)
  • Cache misses (slower builds but no digest errors)
  • Registry / tagging issues (for example, tag does not exist at push)—often seen on the same image roll, but fixed by aligning build and push image names

Why this happens

Rolling machine image tags (defaultcurrent)

If your config uses a rolling tag:

machine:
  image: default   # or ubuntu-2204:current
  docker_layer_caching: true

that tag periodically resolves to a new dated image (for example, ubuntu-2204:2026.05.1). Recent Linux rolls have shipped Docker 29 and related CLI/engine changes.

Incompatible cached layers

DLC restore copies previously stored layers into the job. The new engine may not be able to read or export layers that were cached under the previous engine. The failure often appears after restore, during buildx export or layer unpack, with a content digest error.

Because DLC is org-scoped, one project can write cache data that another project later restores. Clearing DLC on only one failing project may help once; the problem can recur when another project in the org rebuilds the cache with mixed or stale layers.

Unnamed buildx builders

If builds use docker buildx with the implicit docker:default builder (no fixed name), builder instances and volumes may not be tracked consistently across runs. CircleCI recommends naming your buildx builder when using DLC.


Verify the cause

Step 1: Check which machine image the job used

  1. Open a failing job in the CircleCI app.
  2. Open the Spin up environment step (or the earliest machine setup step).
  3. Confirm the image tag (for example, ubuntu-2204:2026.05.1).

If you use image: default or current, compare with a job from before failures started. A newer dated tag plus digest errors strongly points to engine/cache incompatibility.

Step 2: Confirm DLC restored before the failure

In the same job:

  1. Find Restoring DLC / Restore DLC success.
  2. Find the docker buildx build (or docker build) step where the error occurs.

If restore succeeded and the error is content digest ... not found during build/export, treat this as stale or incompatible DLC, not a broken DLC service.

Step 3: Optional — rule out DLC only

On one project, run a single pipeline with DLC disabled on the same machine image:

machine:
  image: ubuntu-2204:2026.05.1
  docker_layer_caching: false

If that build passes consistently while DLC builds fail, the remaining issue is cache-related rather than the Dockerfile or registry.


What to do

Stop using rolling tags until builds are stable. Pin an explicit dated tag in config (or in a shared orb used org-wide):

machine:
  image: ubuntu-2204:2025.09.1
  docker_layer_caching: true
  • ubuntu-2204:2025.09.1 is a common short-term choice before adopting the latest roll.
  • After purging DLC (below), you can pin ubuntu-2204:2026.05.1 if you want to stay on the newest image deliberately.

Available tags: Ubuntu 22.04 machine images.

Background on recent image changes: Ubuntu Q2 2026 edge release.

2. Purge DLC on affected projects

Clear incompatible cache data on each affected project, not only after a failure:

Web app

  1. Open the project in the CircleCI app.
  2. Go to Project Settings → Docker Layer Caching.
  3. Select Delete Cache Contents.

CLI (CircleCI CLI v0.1.24705+)

circleci project dlc purge github <org-name> <project-name>

Repeat for other flaky repositories. With many projects in one org, plan a coordinated purge across affected repos. One purge is sometimes not enough; clear again on projects that fail after pinning.

Details: Docker layer caching — purge DLC.

3. Name your buildx builder

If you use docker buildx build with DLC, create and use a fixed builder name on every run:

docker buildx create --name my-org-dlc-builder --use
docker buildx build ...

Avoid relying on the implicit docker:default builder when DLC is enabled.

See: Buildx builder instances.

4. Roll out config through your orb or templates

If many projects share a private orb or template, apply image pinnamed buildx builder, and DLC purge through that shared config so all pipelines pick up the same fix on the next orb/version bump.


On the same machine image roll, some teams see push failures such as tag does not exist when the build step tagged an image with an implicit docker.io/ prefix (for example, docker.io/my.registry.example/...) but push used my.registry.example/... without that prefix.

To fix, use the same full image reference on build and push (often in a private orb or shared scripts). That is separate from DLC digest errors but may start at the same time as an image promotion.


FAQ

Does this mean DLC is broken on CircleCI?

Usually no. If logs show Restore DLC success and the failure is a content digest error during build, DLC is working; the cached layer data does not match the current Docker engine.

Why does clearing DLC fix one run but not permanently?

The next run rebuilds cache without the bad layers. With org-wide DLC and many projects, another pipeline can write incompatible data again, or a single purge may not clear everything still referenced. Pin the image and purge across affected projects for a durable fix.

We did not change our docker orb. What changed?

The machine executor image (especially when using default or current) can change underneath you. New images ship a newer Docker engine; behavior of buildx, tagging, and cache compatibility can change without an orb version bump.

Should we stay on machine: image: default?

default and current are convenient but move over time. For production Docker builds with DLC, pin a dated image tag until you are ready to test and purge cache on a deliberate upgrade.

Does this affect Remote Docker (setup_remote_docker)?

This article focuses on the machine executor with DLC. Remote Docker has its own image/version options; if you see similar digest errors there, purge DLC for the project and review Docker layer caching overview.

Will failed DLC jobs still use credits?

Yes. Failed jobs may still consume credits or resources for the time they ran. Cancel unneeded reruns if you catch a bad cache early.

 

Was this article helpful?
0 out of 0 found this helpful

Comments

0 comments

Article is closed for comments.