[Server] Applying Temporary Patches to CircleCI Service Using nREPL

When need to apply temporary patches to the CircleCI service to fix issues or update configurations. This guide will show you how to connect to a Kubernetes pod using REPL and make necessary changes. Server 4.8+ no longer ships with the lein binary, therefore must be connected via an external container.

Prerequisites

Access to your Kubernetes cluster.
kubectl installed and configured.
Local docker access with the ability to pull the Clojure image.

Instructions

Step 1: Confirm the Service Pod

First, make sure you can identify the pod in your CircleCI namespace. This time we are specifying app=frontend but this will change depending on the service we want to patch. Use the following command to confirm the pod name:

namespace="circleci-server"
kubectl get po -n $namespace -l app=frontend -o jsonpath='{.items[0].metadata.name}'

Step 2: Port forward to make REPL accessible

Once you have confirmed the pod name, port forward this pod to expose REPL instance.

namespace="circleci-server"
kubectl port-forward $(kubectl get pods -l app=frontend -o jsonpath='{.items[0].metadata.name}' -n $namespace) -n $namespace 16000:6005 &

Step 3: Access pod to make changes

After the port has been forwarded, connect to the exposed REPL using the Clojure docker container:

docker run --rm -it clojure lein repl :connect host.docker.internal:16000

Once you are connected you can make configuration changes via the REPL shell. Type exit to quit and don't forget to shutdown the port forwarding background process once done.

Important Note

This patch is temporary and will be reset if the service (deployment) is re-rolled. For a permanent solution, wait for the official patch release.

Additional Resources