Swimlane 10.26.0 Upgrade Instructions

use these instructions to upgrade your swimlane 10 x instance the instructions are divided into subsections that guide you through the upgrade process based on your installation type, whether it is an embedded installation or an existing cluster red hat 8 is no longer supported beginning with swimlane 10 25 2 as part of the swimlane 10 25 2 embedded cluster upgrades, kubernetes is updated to version 1 33, which does not support red hat 8 (specifically, linux kernel versions 4 18 and earlier) customers with embedded clusters running red hat 8 must upgrade to red hat 9 before upgrading to swimlane 10 25 2 upgrade swimlane on embedded cluster installations use the following instructions to upgrade this version of swimlane on an embedded cluster installation prerequisites swimlane version 10 25 2 or later if you are not on a version that is 10 25 2 or later , contact your swimlane support representative mongodb 8 0 26 important! swimlane recommends that you perform a mongodb backup before starting this update for more information, see backup and restore on an embedded cluster with snapshots if swimlane uses an external mongodb, scale down the cluster by setting the replicas for deployments and stateful sets (sts) to 0 then upgrade your external mongodb servers to version 8 0 29 and follow the upgrade instructions during the startup of the external mongodb process, use the following commands \ tlsallowconnectionswithoutcertificates tlsallowinvalidcertificates kubectl scale deploy \<deployment name> n \<name space> replicas=0 kubectl scale sts \<sts name> n \<name space> replicas=0 upgrade instructions ssh into any node in your deployment and execute the following command to begin the swimlane platform installer component upgrade $ curl ssl https //kurl sh/swimlane platform stable 10 26 0 | sudo bash s ha note if you have configured selinux, add the following to the end of the command above installer spec file=se yaml a kubernetes upgrade may be required depending on the current version in your environment for a multi node (ha) cluster, the script prompts you to drain node one (1) and apply the upgrade once the upgrade on node one (1) is complete, the process continues running it then asks for confirmation to drain node two (2) as well as a command to run in that node for example ✔ kubernetes host packages installed node/ip 10 1 0 147 swimlane io already uncordoned drain node ip 10 1 0 186 swimlane io to prepare for upgrade? (y/n) y pod/swimlane syslog receiver 86f79c8fd6 2rvwt evicted pod/swimlane tasks 5dfd4d5f94 tkrb2 evicted node/ip 10 1 0 186 swimlane io evicted run the upgrade script on remote node to proceed ip 10 1 0 186 swimlane io curl https //kurl sh/swimlane platform stable 10 26 0/upgrade sh | sudo bash s kubernetes version=1 35 7 run the upgrade command on the node that was drained use kubectl get nodes o wide to find the ip of the node that was drained the primary script that ran in node 1 will detect that node 2 has been upgraded and will now ask to drain and upgrade node 3 follow the same steps for node 2 to complete the kubernetes portion of the upgrade once the additional nodes are complete, press enter in the terminal window of the first node on which you ran the install command to complete the installation log in to the swimlane platform installer dashboard (https //\<swimlanedns> 8800) go to the version history tab and locate the newly published version if this version doesn’t appear initially, click check for updates a pop up window appears, click go to updated version click deploy on the new 10 26 0 version with the highest sequence number at the top of the versions list (currently 10 26 0 xxx) once the application tab shows ‘ready’, your upgrade is complete note certificate authority update for mongodb (ensure all mongodb pods are restarted) # certificate authority update for mongodb ## scope manual intervention only applies to ha upgrades single node and new installs are not affected as part of this release, we are introducing cert manager to automate the lifecycle of mongodb certificates previously, mongodb used long lived static certificates generated automatically with this change, mongodb certificates will now be managed by cert manager and issued by a new certificate authority (ca) because the ca is changing, mongodb will not automatically trust the new certificates until all mongodb pods have been restarted this step is required to avoid connectivity issues between mongodb nodes # impact action required potential manual restart of all mongodb pods after applying the upgrade downtime a brief downtime will occur while mongodb pods restart, as all members of the replica set must be cycled to pick up the new ca the cycling of mongodb pods should complete on its own without the need for intervention in most cases however, if the rollout is stuck for more than 15 minutes, it may be required to restart the mongodb pods manually to pick up the new certificates see update the swimlane python driver python package docid\ vbpdzatjjgjirwmidh ix if you want to update the driver at this time upgrade swimlane on existing cluster installations prerequisites swimlane version 10 25 2 or later if you are not on a version that is 10 25 2 or later , contact your swimlane support representative mongodb 8 0 26 kubernetes 1 34 9 kots kubectl plugin 1 130 9 velero 1 18 2 important! swimlane recommends that you perform a mongodb backup prior to starting this update for more information, see backup and restore on an existing cluster with snapshots upgrading to kotsadm v1 130 9 download the latest spi kubectl add on for the os version where you run kubectl commands from $ curl https //kots io/install/v1 130 9 | bash $ kubectl kots version the kots kubectl plugin version should match what is listed in the prerequisites above upgrading to velero 1 18 x verify if velero needs to be updated $ /velero version download velero 1 18 x for your os version https //github com/vmware tanzu/velero/releases/tag/v1 18 2 https //github com/vmware tanzu/velero/releases/tag/v1 16 2 untar the file and then change directory (cd) into the uncompressed directory $tar zxf \<filename> tar gz $cd \<directory> update velero custom resource definitions $ /velero install crds only dry run o yaml | kubectl apply f upgrade container images as instructed in the velero article upgrade to 1 18 confirm that both the client and server versions are on the same 1 18 2 release $ /velero version upgrade the swimlane kots application next, upgrade the swimlane platform installer (spi) $ kubectl kots admin console upgrade n \<namespace> log in to the swimlane platform installer dashboard (https //\<swimlanedns> 8800) go to the version history tab and locate the newly published version if this version doesn’t show up initially, click check for updates a pop up window appears, click go to updated version or visit the version history tab click deploy on the new 10 26 0 version with the highest sequence number at the top of the versions list once the application tab shows ready, your upgrade is complete the cycling of mongodb pods should complete on its own without the need for intervention in most cases however, if the rollout is stuck for more than 15 minutes, it may be required to restart the mongodb pods manually to pick up the new certificates downtime a brief downtime will occur while mongodb pods restart, as all members of the replica set must be cycled to pick up the new ca action required potential manual restart of all mongodb pods after applying the upgrade \# impact because the ca is changing, mongodb will not automatically trust the new certificates until all mongodb pods have been restarted this step is required to avoid connectivity issues between mongodb nodes as part of this release, we are introducing cert manager to automate the lifecycle of mongodb certificates previously, mongodb used long lived static certificates generated automatically with this change, mongodb certificates will now be managed by cert manager and issued by a new certificate authority (ca) \# certificate authority update for mongodb ## scope manual intervention only applies to ha upgrades single node and new installs are not affected note certificate authority update for mongodb (ensure all mongodb pods are restarted) see update the swimlane python driver python package docid\ vbpdzatjjgjirwmidh ix if you want to update the driver at this time upgrade swimlane on embedded airgap cluster installations prerequisites swimlane version 10 25 2 or later if you are not on a supported version that is 10 25 2 or later , contact your swimlane support representative mongodb 8 0 26 links to the offline installer package and offline airgap bundle if you do not have these links, contact your swimlane support representative important! swimlane recommends that you perform a mongodb backup prior to starting this update for more information, see backup and restore on an embedded cluster with snapshots upgrade instructions copy the latest offline installer package to each air gapped server untar the offline installer package on each server $ tar xzvf swimlane platform 10 26 0 xxx tar gz run the swimlane platform installer update on any node in your deployment $ cat install sh | sudo bash s airgap ha once the installation is complete, if you have a multi node (ha) cluster, the script prompts you to run an additional command on every node run this script on node swimlane 2 to load required images before proceeding cat /tasks sh | sudo bash s load images have images been loaded on node swimlane 2? (y/n) run cat /tasks sh | sudo bash s load images on nodes two (2) and node three (3) before proceeding to run the command on node one (1) this command can be run simultaneously on nodes 2 and 3 once the load images script completes on nodes 2 and 3, enter y in the node 1 terminal session twice the install script now runs on node 1 for a few minutes once it completes, follow the prompts to run the upgrade script on nodes 2 and 3, similar to this example (the ip address will depend on your deployment) ✔ kubernetes host packages installed node/ip 10 1 0 147 swimlane io already uncordoned drain node ip 10 1 0 186 swimlane io to prepare for upgrade? (y/n) y pod/swimlane syslog receiver 86f79c8fd6 2rvwt evicted pod/swimlane tasks 5dfd4d5f94 tkrb2 evicted node/ip 10 1 0 186 swimlane io evicted run the upgrade script on remote node to proceed ip 10 1 0 186 swimlane io cat upgrade sh | sudo bash s kubernetes version=1 34 9 docker registry ip=10 96 3 115 when both upgrade scripts are complete, apply this workaround known kotsadm issue a known problem prevents the kotsadm pod from starting with imagepullbackoff errors after embedded, airgap upgrades to resolve the problem, these additional steps must be performed on the node where kotsadm is scheduled to start after the kots upgrade is complete $ cd /var/lib/kurl/addons/kotsadm/1 130 9/images $ sudo bash c 'cat kotsadm migrations tar gz | gunzip | ctr n=k8s io images import ' $ sudo bash c 'cat kotsadm tar gz | gunzip | ctr n=k8s io images import ' $ sudo kubectl delete pod l app=kotsadm for ha clusters, then execute these commands except for the last one (deleting the kotsadm pod) on each of the remaining nodes then press enter on node 1 and upload the airgap bundle to the spi dashboard log in to the swimlane platform installer dashboard (https //\<swimlanedns> 8800) go to the version history tab and click upload a new version , then browse to the swimlane platform 10 26 0 xxx airgap file and click upload the sequence number “ xxx” of the airgap file may be higher than shown if a new sequence has been published once the upload of the airgap completes, refresh the page or switch tabs to see the status of the file extraction go back to the version history tab and click deploy on the new 10 26 0 version at the top of the versions list once the application tab shows ‘ready', your upgrade is complete the cycling of mongodb pods should complete on its own without the need for intervention in most cases however, if the rollout is stuck for more than 15 minutes, it may be required to restart the mongodb pods manually to pick up the new certificates downtime a brief downtime will occur while mongodb pods restart, as all members of the replica set must be cycled to pick up the new ca action required potential manual restart of all mongodb pods after applying the upgrade impact because the ca is changing, mongodb will not automatically trust the new certificates until all mongodb pods have been restarted this step is required to avoid connectivity issues between mongodb nodes as part of this release, we are introducing cert manager to automate the lifecycle of mongodb certificates previously, mongodb used long lived static certificates generated automatically with this change, mongodb certificates will now be managed by cert manager and issued by a new certificate authority (ca) \# certificate authority update for mongodb ## scope manual intervention only applies to ha upgrades single node and new installs are not affected note certificate authority update for mongodb (ensure all mongodb pods are restarted) see update the swimlane python driver python package docid\ vbpdzatjjgjirwmidh ix if you want to update the driver at this time upgrade swimlane on existing airgap cluster installations prerequisites swimlane version 10 25 2 or later if you are not on a supported version , contact your swimlane support representative mongodb 8 0 26 kubernetes 1 34 9 kots kubectl plugin 1 130 9 velero 1 18 x important! swimlane recommends that you perform a mongodb backup prior to starting this update for more information, see backup and restore on an existing cluster with snapshots upgrading to kotsadm v1 130 9 download the kots kubectl add on linux https //github com/replicatedhq/kots/releases/download/v1 130 9/kots linux amd64 tar gz https //github com/replicatedhq/kots/releases/download/v1 129 4/kots linux amd64 tar gz osx https //github com/replicatedhq/kots/releases/download/v1 130 9/kots darwin all tar gz https //github com/replicatedhq/kots/releases/download/v1 130 0/kots darwin all tar gz download the latest spi offline package https //github com/replicatedhq/kots/releases/download/v1 130 9/kotsadm tar gz https //github com/replicatedhq/kots/releases/download/v1 130 0/kotsadm tar gz copy the kots kubectl add on and spi offline packages from your jumpbox to each node $ scp kots \<os version> tar gz user@\<swimlanedns> /tmp/kots \<os version> tar gz $ scp kotsadm tar gz user@\<swimlanedns> /tmp/kotsadm tar gz untar the spi kubectl add on $ tar zxvf kots \<os version> tar gz rename or remove the original kubectl kots file $ rm /usr/local/bin/kubectl kots rename and move the latest spi kubectl add on file to /usr/local/bin/kubectl kots $ mv kots kubectl kots $ mv kubectl kots /usr/local/bin/ verify version $ kubectl kots version upgrading to velero 1 18 x verify if velero needs to be updated $ /velero version download velero 1 18 to the jumpbox for the jumpbox's os version https //github com/vmware tanzu/velero/releases/tag/v1 18 2 untar the file and then change directory into the uncompressed directory $ tar zxf \<filename> tar gz cd \<directory> update velero custom resource definitions $ /velero install crds only dry run o yaml | kubectl apply f download the new velero images and push them to your private registry from your jumpbox for specific steps on how to download and push new velero images, see download and push new velero images upgrade container images as instructed in the velero article confirm both client and server version are on the same 1 18 2 release $ /velero version upgrade the swimlane kots application push the spi (kotsadm) images to your registry $ kubectl kots admin console push images kotsadm tar gz \<your registry endpoint>/\<your registry name> registry username \<username> registry password \<passw ord> next, upgrade the swimlane platform installer (spi) $ kubectl kots admin console upgrade kotsadm registry \<your registry endpoint>/\<your registry name> registry username \<username> registry password \<password> n \<namespace> when the spi upgrade is complete, upload the swimlane airgap bundle to the spi dashboard log in to the swimlane platform installer dashboard (https //\<swimlanedns> 8800) go to the version history tab and click upload a new version , then browse to the swimlane platform 10 26 0 xxx airgap file and click upload the sequence number “ xxx” of the airgap file may be higher than shown if a new sequence has been published once the upload of the airgap completes, refresh the page or switch tabs to see the status of the file extraction go to the version history tab you will see a version for 10 26 0 xxx click deploy on the new 10 26 0 xxx version with the highest sequence number at the top of the versions list once the application tab shows ‘ready’ your upgrade is complete the cycling of mongodb pods should complete on its own without the need for intervention in most cases however, if the rollout is stuck for more than 15 minutes, it may be required to restart the mongodb pods manually to pick up the new certificates downtime a brief downtime will occur while mongodb pods restart, as all members of the replica set must be cycled to pick up the new ca action required potential manual restart of all mongodb pods after applying the upgrade \# impact because the ca is changing, mongodb will not automatically trust the new certificates until all mongodb pods have been restarted this step is required to avoid connectivity issues between mongodb nodes as part of this release, we are introducing cert manager to automate the lifecycle of mongodb certificates previously, mongodb used long lived static certificates generated automatically with this change, mongodb certificates will now be managed by cert manager and issued by a new certificate authority (ca) \# certificate authority update for mongodb ## scope manual intervention only applies to ha upgrades single node and new installs are not affected note certificate authority update for mongodb (ensure all mongodb pods are restarted) see update the swimlane python driver python package docid\ vbpdzatjjgjirwmidh ix if you want to update the driver at this time download and push new velero images use these commands to download and push new velero images $ docker pull velero/velero\ v 1 18 2 $ docker tag velero/velero\ v1 18 2 \<my registry>/\<my namespace>/velero/velero\ v1 18 2 $ docker push \<my registry>/\<my namespace>/velero/velero\ v1 18 2 $ docker pull velero/velero restic restore helper\ v1 18 2 $ docker tag velero/velero restic restore helper\ v1 18 2 \<my registry>/\<my namespace>/velero/velero restic restore helper\ v1 18 2 $ docker push \<my registry>/\<my namespace>/velero/velero restic restore helper\ v1 18 2 $ docker pull velero/velero plugin for aws\ v1 11 1 $ docker tag velero/velero plugin for aws\ v1 11 1 \<my registry>/\<my namespace>/velero/velero plugin for aws\ v1 11 1 $ docker push \<my registry>/\<my namespace>/velero/velero plugin for aws\ v1 10 1 $ docker pull velero/velero plugin for gcp\ v1 11 1 $ docker tag velero/velero plugin for gcp\ v1 11 1 \<my registry>/\<my namespace>/velero/velero plugin for gcp\ v1 11 1 $ docker push \<my registry>/\<my namespace>/velero/velero plugin for gcp\ v1 11 1 $ docker pull velero/velero plugin for microsoft azure\ v1 11 1 $ docker tag velero/velero plugin for microsoft azure\ v1 11 1 \<my registry>/\<my namespace>/velero/velero plugin for microsoft azure\ v1 11 1 $ docker push \<my registry>/\<my namespace>/velero/velero plugin for microsoft azure\ v1 11 1 troubleshooting upgrading may occasionally leave the hangfire notifications in a corrupted state, causing the swimlane api and swimlane tasks pods to crash loop with logs like the following swimlane api hangfire mongo migration mongomigrationexception migration failed in hangfire mongo migration steps version17 addnotificationscollection swimlane tasks core errorhandling contractrequiresexception rabbitmqconnectionstring is not defined in orchestration configuration deleting the hangfire notifications collection in mongodb resolves this issue kubectl exec swimlane tools 0 /bin/bash c 'mongo $swimlane data mongo swimlaneconnectionstring quiet tls tlsallowinvalidcertificates eval \\'db\["hangfire notifications"] drop()\\'' if dashboards or data do not load correctly after the upgrade, this is typically due to cached frontend assets clear browser cache and site data for the swimlane url, then perform a hard refresh if using cloudflare, purge the cloudflare cache