Pre-Upgrade Health Check

node status check that all nodes are in ready state kubectl get nodes owide if any nodes are in notready, schedulingdisabled, etc then it would be recommended to review and fix them before upgrading pod status check that all pods are in running state and the pod values are matching 1/1, 3/3, etc kubectl get pods a if any pods are not running i e, 0/1, 2/3, crashloopbackoff, error, etc then it would be advised to review the issue and fix it prior to upgrading cluster status check that kubernetes is up and running and healthy kubectl cluster info if kubernetes indicates that it is not healthy then review and establish where the issue is prior to upgrading etcd health etcd needs to be in a stable state prior to upgrading any issues for pod in $(kubectl get pods l component=etcd n kube system \\ o jsonpath='{ items\[ ] metadata name}') do echo "### etcd pod ${pod} ###" kubectl n kube system exec ${pod} /bin/sh \\ c "etcdctl api=3 etcdctl \\ \ cert=/etc/kubernetes/pki/etcd/server crt \\ \ key=/etc/kubernetes/pki/etcd/server key \\ \ cacert=/etc/kubernetes/pki/etcd/ca crt \\ endpoint health" done should any other states be listed on the output then it would be advised to investigate the logs of the pod to establish the issue hangfire status check that hangfire is running and there is no large amount of enqueued jobs it would be recommended to grab a screenshot of the hangfire dashboard prior to upgrading to check the amount processed for deleted jobs logging status check the logging section within the ui for error messages it would be recommended to grab a screenshot of the logging section to ensure that any post upgrade issues are correctly identified velero status and backups ensure a backup is taken prior to upgrading should any issues arise during upgrade to be able to perform a restore velero get backups turbine specific review the turbine queues to ensure that the queues are healthy should be low numbers across the board kubectl exec it rabbitmq server 0 rabbitmqctl list queues mongodb records amount establishing the amount of swimlane records the customer has use swimlanedb records find() count() establishing the amount of swimlanehistory records the customer has use swimlanehistorydb records find() count() mongodb collection sizes establishing the collection sizes within the swimlanedb use swimlane function getreadablefilesizestring(filesizeinbytes) { var i = 1; var byteunits = \[' kb', ' mb', ' gb', ' tb', 'pb', 'eb', 'zb', 'yb']; do { filesizeinbytes = filesizeinbytes / 1024; i++; } while (filesizeinbytes > 1024); return math max(filesizeinbytes, 0 1) tofixed(1) + byteunits\[i]; }; var collectionnames = db getcollectionnames(), stats = \[]; collectionnames foreach(function (n) { stats push(db getcollection(n) stats()); }); stats = stats sort(function(a, b) { return b\['size'] a\['size']; }); for (var c in stats) { print(stats\[c]\['ns'] + " " + getreadablefilesizestring(stats\[c]\['size']) + " (" + getreadablefilesizestring(stats\[c]\['storagesize']) + ")"); } establishing the collection sizes within the swimlanehistorydb use swimlanehistory function getreadablefilesizestring(filesizeinbytes) { var i = 1; var byteunits = \[' kb', ' mb', ' gb', ' tb', 'pb', 'eb', 'zb', 'yb']; do { filesizeinbytes = filesizeinbytes / 1024; i++; } while (filesizeinbytes > 1024); return math max(filesizeinbytes, 0 1) tofixed(1) + byteunits\[i]; }; var collectionnames = db getcollectionnames(), stats = \[]; collectionnames foreach(function (n) { stats push(db getcollection(n) stats()); }); stats = stats sort(function(a, b) { return b\['size'] a\['size']; }); for (var c in stats) { print(stats\[c]\['ns'] + " " + getreadablefilesizestring(stats\[c]\['size']) + " (" + getreadablefilesizestring(stats\[c]\['storagesize']) + ")"); } download the configvalues file if kotsadm goes in crashloopbackoff while upgrading, its good to collect below information while have pre upgrade checks call downloaded kots plugin (replace app namespace ) kubectl kots download app slug n app namespace dest /manifests overwrite decrypt password values with the namespace on the cluster where you installed your application app slug with the slug of the application example kubectl kots download swimlane platform n default dest /manifests overwrite decrypt password values