Debugging Kubernetes
Recently I’ve been setting up a k8s cluster in EKS and deploying vendor provided helm charts into it. We’ve often had to debug issues and I’ve learnt a number of useful commands to help with this along the way
Helpful Setup
Autocomplete
Consider using the fish shell for the great auto completion experience it provides. Completions are not just for commands,
but for names of items within the cluster too. This helps with easily TAB to complete the randomly generated suffixes of pod names from their
source name (the prefix beginning the deployment or daemonset name etc)
Details on setting up kubectl completions
Details on setting up helm completions
Both tools also support completions on bash, zsh and powershell if you use those
Context
It can become a pain constantly needing to specify -n <your namespace> with every command
Set the context to the namespace you need to interact with most
kubectl config set-context --current --namespace=my-namespace
kubectl config view | grep namespace:
If you need to revert it
kubectl config set-context --current --namespace=""
High level overviews
To get the initial context, check through the common resource to see if anything is flagging up as odd
kubectl get <pods/deployments/ds/rs/services/jobs> # or any other kind of resources
You can comma separate to get multiple at once
kubectl get ds,deployments,rs,statefulsets
Sometimes it’s helpful to output the results as yaml to inspect it or edit and redeploy to test things
kubectl get deployment <your-deployment> -o yaml > deployment-to-play-with.yaml
This can also pair well with the yq CLI for querying specific contents from large yaml docs
kubectl get deployment <your-deployment> -o yaml | yq '.spec.template.spec.containers'
Often useful additional info is found with the wide output:
- Pods: the node they are running on
- Service: the exposed hostname (can be
pendingif there are issues with the cloud load balancer integration)
kubectl get <object> -o wide
Other useful flags
-o json
-o jsonpath='{.path.to.json.object}' # Another way to query
-A # All namespaces
Cluster Resource Utilisation
CPU and memory usage
kubectl top node
Digging Deeper
Describe
If your objects are having trouble getting set up by k8s, you’ll often find useful info in the events section of describe
kubectl describe <Kind>
Logs
If kubectl get pods shows crash looping then it’s likely the application code within a pod/container
that’s erroring, so check the logs
kubectl logs <your pod>
kubectl logs <your pod> -c <your container within the pod>
If you’re not sure of the name of the container in the pod, just kubectl logs <your pod> and it will
show you the names of the other logs in the pod if they exist before it outputs the logs from
the main container
Because k8s will try to restart the pod, it’s often helpful to review the logs from the previous run, before the restart, so that you get the full logs
kubectl logs <your pod> --previous # short flag -p
Other useful flags
-f, --follow # tail the log
--tail <int> # number of previous lines, defaults to all logs (-1)
Port forwarding
To check if an application is working correctly or the networking to it is the issue, you can short cut the cluster networking at many places and directly connect to it from your local machine.
# kubectl port-forward <object>/<object name> <local port>:<remote port>
kubectl port-forward pods/<your pod> 8080:80 # pods/ is implicit so not needed
kubectl port-forward deployment/<your deployment> 8080:80
kubectl port-forward service/<your service> 8080:80
Secrets
Checking the values of secrets requires base64 decoding them
kubectl get secret <your secret> -o jsonpath='{.data.<path.to.secret.value>}' | base64 -d
Accessing a running container
Sometimes you just need to poke around inside a container to see what’s going on
kubectl exec --stdin --tty <your pod> -- /bin/bash # or /bin/sh