Debugging Kubernetes

Posted: 2024-07-23

Recently I’ve been setting up a k8s cluster in EKS and deploying vendor provided helm charts into it. We’ve often had to debug issues and I’ve learnt a number of useful commands to help with this along the way

Helpful Setup

Autocomplete

Consider using the fish shell for the great auto completion experience it provides. Completions are not just for commands, but for names of items within the cluster too. This helps with easily TAB to complete the randomly generated suffixes of pod names from their source name (the prefix beginning the deployment or daemonset name etc)

Details on setting up kubectl completions

Details on setting up helm completions

Both tools also support completions on bash, zsh and powershell if you use those

Context

It can become a pain constantly needing to specify -n <your namespace> with every command

Set the context to the namespace you need to interact with most

kubectl config set-context --current --namespace=my-namespace
kubectl config view | grep namespace:

If you need to revert it

kubectl config set-context --current --namespace=""

High level overviews

To get the initial context, check through the common resource to see if anything is flagging up as odd

kubectl get <pods/deployments/ds/rs/services/jobs> # or any other kind of resources

You can comma separate to get multiple at once

kubectl get ds,deployments,rs,statefulsets

Sometimes it’s helpful to output the results as yaml to inspect it or edit and redeploy to test things

kubectl get deployment <your-deployment> -o yaml > deployment-to-play-with.yaml

This can also pair well with the yq CLI for querying specific contents from large yaml docs

kubectl get deployment <your-deployment> -o yaml | yq '.spec.template.spec.containers'

Often useful additional info is found with the wide output:

• Pods: the node they are running on
• Service: the exposed hostname (can be pending if there are issues with the cloud load balancer integration)

kubectl get <object> -o wide

Other useful flags

-o json
-o jsonpath='{.path.to.json.object}'  # Another way to query  
-A                                    # All namespaces

Cluster Resource Utilisation

CPU and memory usage

kubectl top node

Digging Deeper

Describe

If your objects are having trouble getting set up by k8s, you’ll often find useful info in the events section of describe

kubectl describe <Kind> 

Logs

If kubectl get pods shows crash looping then it’s likely the application code within a pod/container that’s erroring, so check the logs

kubectl logs <your pod>

kubectl logs <your pod> -c <your container within the pod>

If you’re not sure of the name of the container in the pod, just kubectl logs <your pod> and it will show you the names of the other logs in the pod if they exist before it outputs the logs from the main container

Because k8s will try to restart the pod, it’s often helpful to review the logs from the previous run, before the restart, so that you get the full logs

kubectl logs <your pod> --previous  # short flag -p

Other useful flags

-f, --follow  # tail the log
--tail <int>  # number of previous lines, defaults to all logs (-1)

Port forwarding

To check if an application is working correctly or the networking to it is the issue, you can short cut the cluster networking at many places and directly connect to it from your local machine.

# kubectl port-forward <object>/<object name> <local port>:<remote port>

kubectl port-forward pods/<your pod> 8080:80  # pods/ is implicit so not needed
kubectl port-forward deployment/<your deployment> 8080:80
kubectl port-forward service/<your service> 8080:80

Secrets

Checking the values of secrets requires base64 decoding them

kubectl get secret <your secret> -o jsonpath='{.data.<path.to.secret.value>}' | base64 -d

Accessing a running container

Sometimes you just need to poke around inside a container to see what’s going on

kubectl exec --stdin --tty <your pod> -- /bin/bash # or /bin/sh