Basic Swimlane Platform Installer (SPI) Troubleshooting Checklist

the following is a list of initial steps when troubleshooting spi issues ensure swimlane is running inside the kubernetes cluster \# swimlane health check \# web service endpoints export ns=\<swimlane namespace> $ kubectl n $ns get ep sw web o jsonpath='{range subsets\[ ] addresses\[ ]}{ ip}{"\n"}{end}' | while read r i; do echo "testing /nginx health on $i " curl ks https //${i} 4443/nginx health echo done \# api service endpoints $ kubectl n $ns get ep sw api o jsonpath='{range subsets\[ ] addresses\[ ]}{ ip}{"\n"}{end}' | while read r i; do echo "======================" echo "sw api pod $i" echo "======================" echo "test /heath endpoint " curl http //${i} 5000/health echo e "\\\n" echo "test /settings/version endpoint " curl http //${i} 5000/settings/version echo e "\\\n" done ensure firewalld and ufw is disabled if it is suppose to be enabled then verify as well systemctl status firewalld systemctl status ufw ensure containerd and kubelet are running containerd run systemctl status docker if the service is in a failed state search the service log for errors journalctl fu containerd kubelet run systemctl status kubelet if the service is in a failed state search the service log for errors journalctl fu kubelet check cluster node health run kubectl get nodes owide all nodes should have show ready in the status column if any nodes are not showing as ready, run kubectl describe node nodename and look for relevant issues/errors under conditions and events check pods status run kubectl get pods a owide | grep v completed investigate further if any pods have status not equal to “running” or status is equal to “running” but not all containers within the pod are running (e g 0/1, 0/2, 1/2) check the pod logs for issues with kubectl logs all containers podname n namespace check the pod describe output for issues with kubectl describe pod podname n namespace check for errors in the events run kubectl get events sort by= metadata creationtimestamp ensure the infrastructure is configured according to the swimlane install guide the following is a list of resources to check and logs/information to retrieve request that a support bundle is provided follow the below steps in order to generate the support bundle see how to generate a support bundle for spi using the admin console or cli via kubelet this bundle includes both swimlane and kubernetes cluster information use the /usr/local/bin/kubectl support bundle binary this bundle includes both swimlane and kubernetes cluster information /usr/local/bin/kubectl support bundle secret/default/kotsadm swimlane platform supportbundle generate a support bundle when kubernetes cluster is completely down this bundle only includes kubernetes cluster information use the /usr/local/bin/kubectl support bundle binary with an alternative url /usr/local/bin/kubectl support bundle interactive=false https //raw\ githubusercontent com/replicatedhq/troubleshoot specs/main/host/cluster down yaml in the ticket, specify which method was used a) admin console b) kubectl cli (secret/default/kotsadm swimlane platform supportbundle) c) binary (secret/default/kotsadm swimlane platform supportbundle) d) binary (https //raw\ githubusercontent com/replicatedhq/troubleshoot specs/main/host/cluster down yaml) in addition to the support bundle, get the status and logs of the docker and kubelet services for services in containerd kubelet; do echo $services ; systemctl is active $services ; systemctl is enabled $services ; echo ""; done for services in containerd kubelet; do echo $services ; systemctl status $services ; systemctl status $services ; echo ""; done journalctl u kubelet since "3 hour ago" > /tmp/kubelet \<node name> log txt journalctl u containerd since "3 hour ago" > /tmp/docker \<node name> log txt if a support bundle is unable to be generated (the issue is occurring before the spi is successfully installed and running), or the support bundle generation fails, then run the following commands and provide the output kubectl get pods a owide | grep v running | grep v completed if any pods are listed include the pod logs with kubectl logs all containers podname n namespace include the pod describe output with kubectl describe pod podname n namespace kubectl get pods a owide kubectl get deploy a kubectl get sts a kubectl get pvc a kubectl get pv a kubectl get svc a kubectl get httpproxy a kubectl get nodes if any nodes have a status other than ready, also include the describe output of the node with kubectl describe node nodename kubectl get events sort by= metadata creationtimestamp get the status and logs of the docker and kubelet services for services in containerd kubelet; do echo $services ; systemctl is active $services ; systemctl is enabled $services ; echo ""; done for services in containerd kubelet; do echo $services ; systemctl status $services ; systemctl status $services ; echo ""; done journalctl u kubelet since "3 hour ago" > /tmp/kubelet \<node name> log txt journalctl u containerd since "3 hour ago" > /tmp/docker \<node name> log txt provide the following os and version infrastructure setup instance provider bare metal, vm, aws instance, azure instance, gcp instance, etc instance sizes memory, cpu, disk/partitions load balancer type and set up