There are a variety of ways that a Kubernetes cluster can be misconfigured. Below are some gathered techniques for diagnosing cluster issues vs Container Agent issues. In time hopefully these will be codified in a diagnostic tool for customers to run in their clusters.
Check Kubernetes API service IP
kubectl get service kubernetes -n default
This command will return information about the kubernetes
service that is automatically created by Kubernetes that points to the API server. The IP reported here can be checked against any error coming from Container Agent on startup. It will output something like:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 28d
Check Kubernetes API service environment variables
Using exec to start a shell session on a container running in the target namespace you can see the environment variables added to the running container environment. Using set | grep KUBERNETES
you can check the variables that refer to the Kubernetes API service:
KUBERNETES_SERVICE_HOST='172.20.0.1'
KUBERNETES_SERVICE_PORT='443'
KUBERNETES_SERVICE_PORT_HTTPS='443'
Again these can be cross referenced against the service output & any error logging output
Directly check for API connectivity
Using exec
to start a shell session on a container running in the target namespace you can then directly check the API connectivity using curl. Running the following command
curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt https://kubernetes.default.svc -m 30
or alternatively
curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt https://KUBERNETES_SERVICE_HOST:KUBERNETES_SERVICE_PORT -m 30
will make an HTTP request to the API server (using a 30s timeout). This should return a 403
(as there is no bearer token configured) if the API server is reachable
Comments
Article is closed for comments.