Preliminary diagnostic data for Kubernetes troubleshooting

This document explains how to gather preliminary diagnostic data for troubleshooting Swimlane Platform deployments that include Kubernetes clusters. To get detailed information about the overall health of your cluster, you can run:

kubectl cluster-info dump | tee cluster-info.log

Nodes:

Check if all the nodes are all registered correctly.

kubectl get nodes -o wide | tee nodes.log

Verify that all of the expected nodes are present and that they are all in the Ready state.

Describe each node that doesn’t have STATUS equal Ready.

kubectl describe node <name> | tee <name>-desc.log

Confirm that both the kubelet and docker services are running:

for SERVICES in containerd kubelet; do echo --- $SERVICES --- ; systemctl is-active $SERVICES ; systemctl is-enabled $SERVICES ; echo ""; done

If either of the services are inactive, capture the last 3 hours of log:

journalctl -u <service-name> --since "3 hour ago" > /tmp/<service-name>-log.txt

Pods: Most importantly, collect information about the health of the pods within the cluster.

Run to capture all recent events.

kubectl get events --field-selector type!=Normal --sort-by='.metadata.creationTimestamp' --all-namespaces > /tmp/k8s-events.log

Note: By default, the amount of time to retain events is 1h.

Run and capture its output.

kubectl get pods --all-namespaces -o wide | tee all-pods.log

For each pod that’s shown in the get pods output to be in any state other than running/completed:

Run capture its output.

kubectl logs -n <namepsace-name> <pod-name> > /tmp/<pod-name>-log.txt

Run capture its output.

kubectl describe pod -n <namespace-name> <pod-name> > /tmp/<pod-name>-desc.txt

Since the output from these several commands can be voluminous, copy each command's output into a text file and attach these files to your Support Portal ticket.

To minimize the amount of copying and pasting when formulating the above commands, try the following:

# Save the name of the Swimlane namespace into an environment variable export ns=replicated-very-long-app-id kubectl get pods -n $ns

Support Bundle

If your Swimlane deployment includes an Admin console/dashboard then follow the steps below to generate and submit a Support Bundle:

Visit https://<load-balancer-ip>:8800

Browse to the Support/Troubleshoot section.

Download the Support Bundle to your workstation.

Attach the Support Bundle to your Support ticket.

Troubleshooting kubectl:

If the first attempt to invoke kubectl results in this error, The connection to the server localhost:8080 was refused…, then run:

# Remember to specify the dash when changing user accounts sudo su - echo $KUBECONFIG

If the output thereof is empty then:

export KUBECONFIG=/etc/kubernetes/admin.conf

If that works then add this export command to your Linux user profile within the Master nodes so that it’s invoked whenever you begin an SSH session.

Thank you for helping us to gather this diagnostic data!