How to debug Docker Push to GCR

Sometimes, when pushing a built docker image to the Google Container Registry (GCR), it can fail with the following error:

denied: Unable to write blob sha256:29416d5f02a649cf688b551c289c535c2177de6263d43c5279db2cba514315bc

Here are some steps to get your docker pushes working correctly. 

Step 1 - Check DLC Usage

In some cases, this can be due to a broken DLC cache. You can verify if you are using DLC by viewing the Setup Remote Docker Engine step or under the Spin Up Environment step if using Docker with Remote Docker or the Machine Executor respectfully. 

You can simply set docker_layer_cache: false for 3 days on a project, which will clear all layers. If builds have been succeeding without DLC, you can re-enable DLC after the 3 days. 

Step 2 - Check IAM Permissions

GCR and a few docker registry services have IAM settings to control actions. For GCR, it requires some permissions to push the docker image to GCR. You can see the example settings in the following GCR doc. 

Adding the following IAM permissions can provide a positive experience:

storage.objects.delete
storage.objects.update

Step 3 - Check Registry Configuration 

Registries also have some security features, and it sometimes causes an issue. In this example, the Google Cloud Storage (GCS) Retention Policy, which prevents objects in the bucket from being deleted or modified for a specified minimum period of time after they are uploaded, caused this issue when uploading a large image.

When pushing a blob object that is slightly larger like 400MB in size, the docker push command uploads the data in smaller pieces. However, the Retention Policy does not allow modifications to the file in GCS and it declines the second request causing the docker push to fail. In this case, you will need to disable the Retention Policy.