[Server] Resolving Image Pull Errors During CircleCI Upgrade On Event Log

Overview

Encountering image pull errors during a CircleCI upgrade can disrupt production workflows. This issue often arises due to misconfigured image pull secrets in the Helm values file, leading to errors such as ErrImagePull ImagePullBackOff FailedToRetrieveImagePullSecret when running kubectl get pod -n circleci-server.

Prerequisites

• Access to your Helm values file (values.yaml).
• Administrative access to your CircleCI environment.

Solution

Run kubectl describe pod <pod_name_of_err_image_pull> -n circleci-server command and look at the event logs and if you are seeing Unable to retrieve some image pull secrets (map[name:regcred]) please proceed to the following steps:

To resolve the image pull errors, follow these steps:

1. Remove Incorrect regcred Reference:
   • Open your values.yaml file.
   • Locate and remove any lines referencing the - name: regcred secret. This misconfiguration is causing the image pull failures. 

   • Example lines to remove: 

     imagePullSecrets:
       - name: regcred

   • Alternatively you may rewrite this section as follows (removing -name): 

     imagePullSecrets:
       - regcred

     This is because the imagePullSecrets value was changed to support a list of credentials after

2. Simplify Your Helm Values Configuration:
   • Ensure your Helm values are clear and concise to prevent future misconfigurations.
   • Below is a suggested structure for your values.yaml: 

   • global:
       domainName: "<full-domain-name-of-your-install>"
       license: '<license>'
       container:
         registry: cciserver.azurecr.io
         org:

   • Replace <full-domain-name-of-your-install>, <license>, and <your-organization> with your actual deployment details.
3. Run Helm upgrade and make sure the failing pod gets deleted:

•  
  •  
    • Apply the updated Helm values using the following command:

    • helm upgrade circleci-server oci://cciserver.azurecr.io/circleci-server -n circleci-server --version <server-version> -f <path-to-values.yaml>

    • If you encounter the error message Error: UPGRADE FAILED: pre-upgrade hooks failed:, modify your command with additional options to bypass this issue. Use the --debug --no-hooks --force flags as shown below:

    • helm upgrade circleci-server oci://cciserver.azurecr.io/circleci-server -n circleci-server --version <server-version> -f <path-to-values.yaml> --debug --no-hooks --force

      • This approach should only be attempted if you encounter the exact error mentioned. After running the upgrade command delete the failing pod. 
      • For other issues, please open a support ticket and provide the support bundle along with the error message.

By removing the incorrect regcred reference and simplifying your Helm values configuration, you can prevent similar issues in future upgrades.

Additional Resources

• CircleCI Documentation
• Kubernetes Image Pull Secrets
• Updating a Server 4.x Environment with a new license