Introduction:
Pods such as contexts-service can get stuck in a CrashLoopBackOff status when failing to get a root token from vault. This can be verified by checking the logs for the affected service.
Prerequisites
Access to your Kubernetes cluster is needed in order to run the kubectl commands below.
Instructions
- Step 1 - Verifying the pod logs
-
First, run a command like the following to get the most recent logs for the affected pod(s).
kubectl -n <namespace> get logs -l app=contexts-service
Logs such as the following indicate an issue connecting to Vault:
Using Vault encryption backend Retrieving vault root token from url "http://vault:8201/client-token" ... ... failed to retrieve root token. Retrying 9 more times ... ... failed to retrieve root token. Retrying 8 more times ... ... failed to retrieve root token. Retrying 7 more times ... ... failed to retrieve root token. Retrying 6 more times ... ... failed to retrieve root token. Retrying 5 more times ... ... failed to retrieve root token. Retrying 4 more times ... ... failed to retrieve root token. Retrying 3 more times ... ... failed to retrieve root token. Retrying 2 more times ... ... failed to retrieve root token. Retrying 1 more times ... ... failed to retrieve root token.
-
- Step 2 - Rolling vault-0 pod
-
You will need to roll the vault-0 pod. This can be done by deleting the pod with the following command, after which the pod will automatically be recreated.
kubectl delete pods -n <namespace> -l app=vault
-
- Step 3 - Rolling contexts-service pods
-
You will need to roll the contexs-service pods as well. This can be done by deleting the pod with the following command, after which the pods will automatically be recreated.
kubectl delete pods -n <namespace> -l app=contexts-service
-
Outcome:
Builds will start running normally, and there will no longer be failed to retrieve root token generated in the pod logs.
Comments
Article is closed for comments.