[SERVER] Resolving oidc-service CrashLoopBackOff Error in CircleCI Server 4.8.6

Overview

When updating CircleCI Server from version 4.8.6, you might encounter a CrashLoopBackOff error with the oidc-service pod. This issue arises due to a change in the execution command for the oidc-service container, which results in the container failing to start because it cannot find /bin/sh.
 

kubelet Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: "/bin/sh": stat /bin/sh: no such file or directory: unknown

Prerequisites

• Access to your Kubernetes cluster where CircleCI Server is deployed.
• Basic knowledge of using kubectl to manage Kubernetes resources.

Solution

To resolve the CrashLoopBackOff error, you need to modify the deployment configuration for the oidc-service pod. Follow these steps:

1. Edit the Deployment: Use the following command to edit the oidc-service deployment: kubectl edit deploy oidc-service -n circleci-server
2. Modify the Command and Args: In the deployment configuration, locate the command and args fields. Delete args and change the command to /app 
   Change them as follows:

   • Before: 

     command: ['/bin/sh']
     args: ['-c', './server']

   • After: 

     command: ['/app']

3. Save and Exit: After making the changes, save the file and exit the editor. This will trigger a rolling update of the deployment.
4. Verify the Update: Check the status of the pods to ensure the new pod is running successfully: kubectl get pods -n circleci-server

Additional Resources

• Kubernetes Documentation
• CircleCI Server Documentation