Kubernetes Troubleshooting

Diagnosing Container Agent Issues

There are a variety of ways that a Kubernetes cluster can be misconfigured. Below are some gathered techniques for diagnosing cluster issues vs Container Agent issues. Our support team has also developed a tool to help diagnose common Container Agent issues that can be found here:

https://github.com/kelvintaywl-cci/circleci-container-runner-help

Check Kubernetes API service IP

kubectl get service kubernetes -n default

This command will return information about the kubernetes service that is automatically created by Kubernetes and points to the API server. The IP reported here can be checked against any error coming from Container Agent on startup. It will output something like:

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 28d

Check Kubernetes API service environment variables

Using exec to start a shell session on a container running in the target namespace you can see the environment variables added to the running container environment. Using set | grep KUBERNETES you can check the variables that refer to the Kubernetes API service:

KUBERNETES_SERVICE_HOST='172.20.0.1'
KUBERNETES_SERVICE_PORT='443'
KUBERNETES_SERVICE_PORT_HTTPS='443'

Again these can be cross-referenced against the service output & any error logging output.

Directly check for API connectivity

Using exec to start a shell session on a container running in the target namespace you can then directly check the API connectivity using curl. Running the following command

curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt https://kubernetes.default.svc -m 30

or alternatively

curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt https://KUBERNETES_SERVICE_HOST:KUBERNETES_SERVICE_PORT -m 30

will make an HTTP request to the API server (using a 30s timeout). This should return a 403 (as there is no bearer token configured) if the API server is reachable.