Swimlane 10.12.0 Upgrade Instructions

Use these upgrade instructions to upgrade your instance of Swimlane 10.x. The instructions are organized by subsections of this document that will help you upgrade by specific installation type, whether you have an embedded, or existing cluster installation.

Upgrade Swimlane on Embedded Cluster Installations

Use the following instructions to upgrade this version of Swimlane on an embedded cluster installation.

Prerequisites:

Swimlane version 10.11.3+ If you are not on a version that is > 10.11, contact your Swimlane support representative. The upgrade to 10.12+ includes updating the container runtime from docker to containerd. See the new partition sizes in the System Requirements for an Embedded Cluster Install. The old /var/lib/docker partition can be removed after a successful transition has been completed.

MongoDB 5.0.14

Important! Swimlane recommends that you perform a MongoDB backup prior to starting this update. For more information, see Backup and Restore on an Embedded Cluster with Snapshots.

Upgrade Instructions:

SSH into the any node in your deployment and execute the following commands to begin the Swimlane Platform Installer component upgrade: $ kubectl annotate crd prometheuses.monitoring.coreos.com kubectl.kubernetes.io/last-applied-configuration- $ curl -sSL https://kurl.sh/swimlane-platform-stable-10-12-0 | sudo bash -s ha NOTE: If you have configured SELinux, add this to the end of the command above: installer-spec-file=se.yaml

For a multi-node (HA) cluster, the script prompts you to drain node one (1) and apply the upgrade. Once the upgrade on node one (1) is complete, the process continues running. It then asks for confirmation to drain node two (2) as well as a command to run in that node.

See this example:✔ Kubernetes host packages installed node/ip-10-1-0-147.swimlane.io already uncordoned Drain node ip-10-1-0-186.swimlane.io to prepare for upgrade? (Y/n) y ... pod/swimlane-syslog-receiver-86f79c8fd6-2rvwt evicted pod/swimlane-tasks-5dfd4d5f94-tkrb2 evicted node/ip-10-1-0-186.swimlane.io evicted Run the upgrade script on remote node to proceed: ip-10-1-0-186.swimlane.io curl https://kurl.sh/swimlane-testing-scott-qa-spt-7770-1/upgrade.sh | sudo bash -s kubernetes-version=1.21.14

Run the upgrade command on the node that was drained. Use kubectl get nodes -o wide to find the IP of the node that was drained. The primary script that ran in node 1 will detect that node 2 has been upgraded, and will now ask to drain and upgrade node three (3). Follow the same steps for node 2 to complete the kubernetes portion of the upgrade.

Next, the install will ask for permission to start the CNI migration from weave to flannel

⚙ Addon flannel 0.21.2 The migration from Weave to Flannel will require whole-cluster downtime. Flannel requires UDP port 8472 for communication between nodes. Please make sure this port is open prior to running this migration. Would you like to continue? (Y/n)

For HA clusters, the installation asks for a command to be run on each additional node. Note, there is an error in the command given. The airgap flag should be removed from the command that the script. This bug will likely be resolved in any patch level updates of 10.12.

Moving primary nodes from Weave to Flannel requires removing certain weave files and restarting kubelet. Please run the following command on each of the listed primary nodes: qa-kc-2 qa-kc-3 curl -fsSL https://kurl.sh/version/v2023.02.23-0/swimlane-platform/tasks.sh | sudo bash -s weave-to-flannel-primary airgap cert-key=******

Once the additional nodes are complete, press Enter in the terminal window of the first node in which you ran the install command to complete the installation.

Go to the Version History tab and look for version 10.12.0_503 If this version doesn’t show up initially, click Check For Updates.

A pop-up window appears, click Go to updated version.

Click Deploy on the new 10.12.0_503 version with the highest sequence number at the top of the versions list.

Once the Application tab shows ‘Ready’ your upgrade is complete.

See Update the Swimlane Python Driver Python Package if you want to update the driver at this time.

Upgrade Swimlane on Existing Cluster Installations

Prerequisites:

Swimlane version 10.6 or greater If you are not on a version that is > 10.6, contact your Swimlane support representative.

MongoDB 5.0.14

Kubernetes 1.19-1.24

KOTS kubectl plugin 1.96.2

Velero 1.9.x

Important! Swimlane recommends that you perform a MongoDB backup prior to starting this update. For more information, see Backup and Restore on an Existing Cluster with Snapshots.

Upgrade Instructions:

Download the latest SPI kubectl add-on for the OS version where you run kubectl commands from: Linux: https://get.swimlane.io/existing-cluster/install/linux OSX: https://get.swimlane.io/existing-cluster/install/osx

Untar the SPI kubectl add-on: $ tar -zxvf kots_.tar.gz

Rename or remove the original kubectl-kots file, $ rm /usr/local/bin/kubectl-kots

Rename and move the latest SPI Kubectl Add-on file to /usr/local/bin/kubectl-kots $ mv kots kubectl-kots $ mv kubectl-kots /usr/local/bin/ $ kubectl kots version The KOTS kubectl plugin version should match what is listed in the prerequisites above.

Next, upgrade the Swimlane Platform Installer (SPI): $ kubectl kots admin-console upgrade -n <swimlane-namespace>

Download Velero 1.9.5 for your OS version: https://github.com/vmware-tanzu/velero/releases/tag/v1.9.5 Untar the file and then change directory (cd) into the uncompressed directory: $ tar -zxf .tar.gz $ cd

Update Velero custom resource definitions: $ ./velero install --crds-only --dry-run -o yaml | kubectl apply -f -

Update the images Velero and Restic use to 1.9.5. $ kubectl set image deployment/velero velero=velero/velero:v1.9.5 --namespace velero $ kubectl set image daemonset/restic restic=velero/velero:v1.9.5 --namespace velero Note: This update may take a minute in order to get the new pods rolled out.

Confirm that both the client and server versions are on the same 1.9.5 release: $ ./velero version

Go to the Version History tab and look for version 10.12.0_503

If this version doesn’t show up initially, click **Check For Updates**. If you are asked to configure, scroll to **PostgreSQL**

On PostrgreSQL Settings, input values for the required fields and then click Save config.

PostgreSQL requires a persistent volume, so you must set a PVC storageclass. The default size is 8 GB.

A pop-up window appears, click Go to updated version or visit the Version History tab.

Click Deploy on the new 10.12.0_503 version with the highest sequence number at the top of the versions list.

Once the Application tab shows ‘Ready’ your upgrade is complete.

See Update the Swimlane Python Driver Python Package if you want to update the driver at this time.

Upgrade Swimlane on Embedded Airgap Cluster Installations

Prerequisites

Swimlane version 10.11+ If you are not on a version that is > 10.11, contact your Swimlane support representative.

MongoDB 5.0.14

Links to the Offline Installer Package and Offline Airgap Bundle If you do not have these linked, contact your Swimlane support representative.

Important! Swimlane recommends that you perform a MongoDB backup prior to starting this update. For more information, see Backup and Restore on an Embedded Cluster with Snapshots.

Upgrade Instructions:

Copy the latest offline installer package to each airgapped server. Untar the offline installer package on each server. $ tar -xzvf swimlane-platform-10.12.0_503.tar.gz

Run this command on any node in your deployment before the upgrade can start, it removes an annotation from a Prometheus resource. $ kubectl annotate crd prometheuses.monitoring.coreos.com kubectl.kubernetes.io/last-applied-configuration-

Run the Swimlane Platform Installer update on any node in your deployment: $ cat install.sh | sudo bash -s airgap ha

Once the install is complete, if you have a multi-node (HA) cluster, the script prompts you to run an additional command on every node: Run this script on node swimlane-2 to load required images before proceeding: cat ./tasks.sh | sudo bash -s load-images Have images been loaded on node swimlane-2? (Y/n)

Run cat ./tasks.sh | sudo bash -s load-images on nodes two (2) and node three (3) before proceeding to run the command on node one (1). This command can be run simultaneously on nodes 2 and 3. Once the load-images script completes on nodes 2 and 3, enter Y in the node 1 terminal session twice.

The install will then ask for permission to start the CNI plugin migration from weave to flannel

A command is given to be run on each additional node

Moving primary nodes from Weave to Flannel requires removing certain weave files and restarting kubelet. Please run the following command on each of the listed primary nodes: qa-kc-2 qa-kc-3 cat ./tasks.sh | sudo bash -s weave-to-flannel-primary airgap cert-key=******

The install script now runs on node 1 for a few minutes. Once it completes, follow the prompts to run the upgrade script on nodes 2 and 3, similar to this example (the IP address will depend on your deployment):

Installation Complete ✔ Run this script on all remote nodes to apply changes cat upgrade.sh | sudo bash -s airgap docker-registry-ip=10.96.3.115 Press enter to proceed

When both upgrade scripts are complete, press Enter on node 1 and then upload the airgap bundle to the SPI dashboard.

Go to the Version History tab and look for version 10.12.0_503.

If this version doesn’t show up initially, click Check For Updates

Browse to the swimlane-platform-10.12.0_503 .airgap file and click upload.

Once the upload of the airgap completes, refresh the page or switch tabs to see the status of the file extraction.

Go back to the Version History tab and click Deploy on the new 10.10.0_466 version with the highest sequence number at the top of the versions list.

Once the Application tab shows ‘Ready’ your upgrade is complete.

See Update the Swimlane Python Driver Python Package if you want to update the driver at this time.

Upgrade Swimlane on Existing Airgap Cluster Installations

Prerequisites

Swimlane version 10.6+ If you are not on a version that is > 10.6, contact your Swimlane support representative.

MongoDB 5.0.14

Kubernetes 1.19-1.24

KOTS kubectl plugin 1.96.2

Velero 1.9.x

Important! Swimlane recommends that you perform a MongoDB backup prior to starting this update. For more information, see Backup and Restore on an Existing Cluster with Snapshots.

Upgrade Instructions:

Download the latest SPI Kubectl Add-on Linux: https://get.swimlane.io/existing-cluster/install/linux OSX: https://get.swimlane.io/existing-cluster/install/osx

Download the latest SPI offline package: https://get.swimlane.io/existing-cluster/install/offline-package

Copy the latest SPI Kubectl Add-on and SPI offline packages to your jumpbox: $ scp kots_.tar.gz user@:/tmp/kots_.tar.gz $ scp kotsadm.tar.gz user@:/tmp/kotsadm.tar.gz

Untar the SPI Kubectl Add-on: $ tar -zxvf kots_.tar.gz

Rename or remove the original kubectl-kots file: $ rm /usr/local/bin/kubectl-kots

Rename and move the latest SPI Kubectl Add-on file to /usr/local/bin/kubectl-kots $ mv kots kubectl-kots $ mv kubectl-kots /usr/local/bin/

Download Velero 1.9.5 to the jumpbox for the jumpbox's OS version: https://github.com/vmware-tanzu/velero/releases/tag/v1.9.5

Untar the file and then change directory into the uncompressed directory: $ tar -zxf .tar.gzcd

Update Velero custom resource definitions: $ ./velero install --crds-only --dry-run -o yaml | kubectl apply -f -

Download the new Velero images and push them to your private registry from your jumpbox.

For specific steps on how to download and push new Velero images, see [Download and Push New Velero Images](#).

Update the images Velero and Restic are using to 1.9.5 (this update will take a minute to get new pods rolled out):

`$ kubectl set image deployment/velero velero=/velero/velero:v1.9.5 --namespace velero` `$ kubectl set image daemonset/restic restic=/velero/velero:v1.9.5 --namespace velero` **Note:** This update may take a minute in order to get the new pods rolled out.

Confirm both client and server version are on the same 1.9.5 release:

`$ ./velero version`

Push the SPI (kotsadm) images to your registry:

`$ kubectl kots admin-console push-images kotsadm.tar.gz / --registry-username --registry-password `

Next, upgrade the Swimlane Platform Installer (SPI):

`$ kubectl kots admin-console upgrade --kotsadm-registry / --registry-username --registry-password -n ` When the SPI upgrade is complete, upload the Swimlane airgap bundle to the SPI dashboard.

Go to the Version History tab and click Upload a new version, then browse to the swimlane-platform-10.12.0_503 .airgap file and click Upload.

Once the upload of the airgap completes, refresh the page or switch tabs to see the status of the file extraction.

Go to the Version History tab - you will see a version for 10.12.0_503.

If you are asked to configure, scroll to **PostgreSQL**.

On PostgreSQL Settings, input the values for the required fields and then click Save config.

PostgreSQL requires a persistent volume, so you must set a PVC storageclass. The default size is 8 GB.

A pop-up window appears, click Go to updated version or visit the Version History tab.

Click Deploy on the new 10.12.0_503 version with the highest sequence number at the top of the versions list.

Once the Application tab shows ‘Ready’ your upgrade is complete.

See Update the Swimlane Python Driver Python Package if you want to update the driver at this time.

Download and Push New Velero Images Use these commands to download and push new Velero images.

$ docker pull velero/velero:v1.9.5 $ docker tag velero/velero:v1.9.5 //velero/velero:v1.9.5 $ docker push //velero/velero:v1.9.5 $ docker pull velero/velero-restic-restore-helper:v1.9.5 $ docker tag velero/velero-restic-restore-helper:v1.9.5 //velero/velero-restic-restore-helper:v1.9.5 $ docker push //velero/velero-restic-restore-helper:v1.9.5 $ docker pull velero/velero-plugin-for-aws:v1.5.0 $ docker tag velero/velero-plugin-for-aws:v1.5.0 //velero/velero-plugin-for-aws:v1.5.0 $ docker push //velero/velero-plugin-for-aws:v1.5.0 $ docker pull velero/velero-plugin-for-gcp:v1.5.0 $ docker tag velero/velero-plugin-for-gcp:v1.5.0 //velero/velero-plugin-for-gcp:v1.5.0 $ docker push //velero/velero-plugin-for-gcp:v1.5.0 $ docker pull velero/velero-plugin-for-microsoft-azure:v1.5.0 $ docker tag velero/velero-plugin-for-microsoft-azure:v1.5.0 //velero/velero-plugin-for-microsoft-azure:v1.5.0 $ docker push //velero/velero-plugin-for-microsoft-azure:v1.5.0

Troubleshooting

Upgrading will sometimes leave the Hangfire notifications in a corrupted state. The swimlane-api and swimlane-tasks pods will be crashlooping with logs of this form

swimlane-api: Hangfire.Mongo.Migration.MongoMigrationException: Migration failed in Hangfire.Mongo.Migration.Steps.Version17.AddNotificationsCollection swimlane-tasks: Core.ErrorHandling.ContractRequiresException: RabbitMQConnectionString is not defined in Orchestration configuration.

Dropping the Hangfire notifcations collection in Mongo will remedy this.

kubectl exec swimlane-tools-0 -- /bin/bash -c 'mongo $SWIMLANE_Data__Mongo__SwimlaneConnectionString --quiet --tls --tlsAllowInvalidCertificates --eval \'db["hangfire.notifications"].drop()\''

On rare occasion, pgpool has been reported to be in an unhealthy state after an update. To remediate, restart the pgpool deployment object kubectl rollout restart deployment postgresql-pgpool --namespace