How to debug Docker Push to GCR
Sometimes, when pushing a built docker image to the Google Container Registry (GCR), it can fail with the following error:
denied: Unable to write blob sha256:29416d5f02a649cf688b551c289c535c2177de6263d43c5279db2cba514315bc
Here are some steps to get your docker pushes working correctly.
Step 1 - Check DLC Usage
In some cases, this can be due to a broken DLC cache. You can verify if you are using DLC by viewing the
Setup Remote Docker Engine step or under the
Up Environment step if using Docker with Remote Docker or the Machine Executor respectfully.
You can simply set
docker_layer_cache: false for 3 days on a project, which will clear all layers. If builds have been succeeding without DLC, you can re-enable DLC after the 3 days.
Step 2 - Check IAM Permissions
GCR and a few docker registry services have IAM settings to control actions. For GCR, it requires some permissions to push the docker image to GCR. You can see the example settings in the following GCR doc.
Adding the following IAM permissions can provide a positive experience:
Step 3 - Check Registry Configuration
Registries also have some security features, and it sometimes causes an issue. In this example, the
Google Cloud Storage (GCS) Retention Policy, which prevents objects in the bucket from being deleted or modified for a specified minimum period of time after they are uploaded, caused this issue when uploading a large image.
When pushing a blob object that is slightly larger like 400MB in size, the docker push command uploads the data in smaller pieces. However, the
Retention Policy does not allow modifications to the file in GCS and it declines the second request causing the docker push to fail. In this case, you will need to disable the Retention Policy.