Turbine Upgrade Guide β Online 3-Node HA Embedded Cluster
target release turbine 26 1 4 | cluster type embedded (kurl) kubernetes, 3 control plane nodes, high availability about this guide upgrading a 3 node ha embedded cluster happens in two distinct phases understanding the split up front makes the rest of the guide much easier to follow phase what gets upgraded where you work phase 1 pre upgrade validation β cluster health, storage, networking, connectivity all three nodes phase 2 infrastructure β kubernetes, containerd, etcd, kurl add ons driven from node 1, with scripts run on node 2 and node 3 phase 3 turbine application β mongodb compatibility, then the app itself admin console ui (https // \<swimlanedns> 8800 ) kubernetes is upgraded cumulatively this cluster goes from 1 32 9 β 1 33 11 β 1 34 4 every node completes 1 33 11 before any node starts 1 34 4 this is why the installer walks the cluster twice and why the process takes several hours node naming used in the examples the screenshots in this guide come from a real cluster map them to your own hosts as follows reference in this guide hostname in screenshots ip in screenshots role node 1 (the driver node) jer turb 10 33 102 115 control plane, master node 2 jer turb1 10 33 102 116 control plane, master node 3 jer turb2 10 33 102 117 control plane, master node 1 is the node you ssh into first and where the main installer runs it orchestrates the whole upgrade and hands you scripts to paste into the other two nodes critical rules β read before you start these are the mistakes that most often break an in flight upgrade open an ssh session to all three nodes before you begin you will be switching between them repeatedly, and there is no time to open a session mid prompt never press enter or ctrl + c on node 1 while it is displaying a script for another node doing so terminates the upgrade partway through only select and copy the text take a verified backup first do not rely on an untested snapshot if you use a proxy, firewall, or selinux, you must append your override yaml file to every installer command, including the ones you paste into node 2 and node 3 do not walk away during the final add on phase pods restart during this stage, and a stuck pod will silently stall the upgrade until you intervene red hat 8 is no longer supported beginning with turbine 26 1 4 as part of the turbine 26 1 4 embedded cluster upgrades, kubernetes is updated to version 1 33, which does not support red hat 8 (specifically, linux kernel versions 4 18 and earlier) customers with embedded clusters running red hat 8 must upgrade to red hat 9 before upgrading to turbine 26 1 4 reference documentation topic link turbine 26 1 4 upgrade instructions https //docs swimlane com/turbine upgrade guide/turbine 2614 upgrade instructions https //docs swimlane com/turbine upgrade guide/turbine 2614 upgrade instructions backup and restore with snapshots https //docs swimlane com/turbine installer/backup and restore on an embedded cluster with snapshots https //docs swimlane com/turbine installer/backup and restore on an embedded cluster with snapshots overriding installer settings (proxy / firewall / selinux) https //docs swimlane com/turbine installer/overriding installer settings https //docs swimlane com/turbine installer/overriding installer settings phase 1 β pre upgrade validation complete every check in this phase before touching the installer if any check fails, stop and resolve it β or contact swimlane technical support β rather than pushing forward step 1 β read the documentation and take a verified backup start by reading the official upgrade instructions for the target release https //docs swimlane com/turbine upgrade guide/turbine 2614 upgrade instructions https //docs swimlane com/turbine upgrade guide/turbine 2614 upgrade instructions then make sure you have a valid, verified backup taken before any upgrade activity follow the backup documentation https //docs swimlane com/turbine installer/backup and restore on an embedded cluster with snapshots https //docs swimlane com/turbine installer/backup and restore on an embedded cluster with snapshots step 2 β request the pre upgrade script from swimlane support ask swimlane technical support to provide the pre upgrade script run it on all kubernetes nodes and send the results back to swimlane technical support so they can review cluster health and confirm you are clear to proceed step 3 β confirm all pods are running kubectl get pods a note some pods will show 0/1 these are mostly kubernetes or application jobs β confirm they are marked completed rather than failing example output namespace name ready status restarts age default cert manager 7d87f5f494 xdmzq 1/1 running 342 (34d ago) 137d default cert manager cainjector 85d6797c85 jw5l6 1/1 running 6 (34d ago) 34d default cert manager webhook 7c6bd4dd5d 26htx 1/1 running 4 34d default changestream data migration 29752575 45ccz 0/1 completed 0 11m default changestream data migration 29752580 7m4vv 0/1 completed step 4 β confirm all nodes are online and ready kubectl get nodes a example output root\@jer turb \\ \\# kubectl get nodes \\ a name status roles age version jer turb ready control plane,master 137d v1 32 9 jer turb1 ready control plane,master 137d v1 32 9 jer turb2 ready control plane,master 137d v1 32 9 note the output must show three control plane,master nodes, all in ready status step 5 β confirm kubernetes and containerd versions match across nodes all nodes must run the same kubernetes version and the same containerd version kubectl get nodes owide example output jer turb ready control plane,master 137d v1 32 9 10 33 102 115 \\\<none\\> ubuntu 22 04 5 lts 5 15 0 171 generic containerd //1 7 29 jer turb1 ready control plane,master 137d v1 32 9 10 33 102 116 \\\<none\\> ubuntu 22 04 5 lts 5 15 0 171 generic containerd //1 7 29 jer turb2 ready control plane,master 137d v1 32 9 10 33 102 117 \\\<none\\> ubuntu 22 04 5 lts 5 15 0 181 generic containerd //1 7 29 step 6 β confirm every partition is below 75% used run this on each node df h | head n 30 example output root\@jer ubuntu \\ \\# df \\ h | head \\ n 30 filesystem size used avail use% mounted on tmpfs 6 3g 1 3m 6 3g 1% /run /dev/sda2 49g 13g 35g 27% / tmpfs 32g 0 32g 0% /dev/shm tmpfs 5 0m 0 5 0m 0% /run/lock /dev/sda4 98g 24k 93g 1% /var/lib/containerd /dev/sda5 98g 24k 93g 1% /var/lib/kubelet /dev/sda3 295g 28k 280g 1% /var/openebs tmpfs 6 3g 4 0k 6 3g 1% /run/user/1000 note if any partition is above 75%, report it to swimlane support so they can help review the environment, or reach out to your linux team to increase the space before attempting the upgrade step 7 β verify proxy configuration (if a proxy is in use) if a proxy exists in the environment, confirm that no proxy is exported and populated a missing no proxy will block the upgrade env | grep i proxy example output http\ proxy="http //proxy example com 8080" https\ proxy="http //proxy example com 8080" no\\ proxy="localhost,127 0 0 1, example com" http\ proxy="http //proxy example com 8080" https\ proxy="http //proxy example com 8080" no\\ proxy="localhost,127 0 0 1, \[example com]\(http //example com)" step 8 β check firewall status if the environment does not use a firewall, a disabled or inactive result is fine β just note it and move on sudo systemctl status firewalld # for red hat, rocky linux sudo ufw status # for ubuntu step 9 β verify connectivity to the load balancer run this from every node curl vk https //\<load balancer fqdn or ip address> 6443 the expected result is a successful connection with no http errors if this fails, the kube apiserver will fail during the upgrade step 10 β verify etcd health for pod in $(kubectl get pods l component=etcd n kube system o jsonpath='{ items\[ ] metadata name}') do echo "### etcd pod ${pod} ###" kubectl n kube system exec ${pod} /bin/sh c "etcdctl api=3 etcdctl cert=/etc/kubernetes/pki/etcd/server crt key=/etc/kubernetes/pki/etcd/server key cacert=/etc/kubernetes/pki/etcd/ca crt endpoint health" done note every endpoint must report healthy if not, the upgrade will fail β report this to swimlane technical support before continuing step 11 β verify flannel vxlan connectivity on port 8472 test connectivity between nodes on the flannel overlay port run this from one node toward another nc u l 8472 \<destination node ip address> the result should be successful note nc holds the connection for a few seconds and then disconnects β that is expected behaviour, not a failure step 12 β check selinux status sestatus if selinux is enabled, it must be disabled or set to permissive mode open /etc/selinux/config in an editor and set selinux=permissive step 13 β create an installer override yaml (proxy, firewall, or selinux only) if any proxy or firewall is enabled in your environment, you need an override yaml patch file that declares both the firewall and the proxy, so that installation packages can bypass them and download successfully follow the official documentation to build the file https //docs swimlane com/turbine installer/overriding installer settings https //docs swimlane com/turbine installer/overriding installer settings note after creating the yaml file, place it on each node , in the same location the upgrade script will be run from you will reference it in steps 16, 24, 29, 34, and 36 phase 2 β infrastructure upgrade (kubernetes and kurl) you are now beginning the upgrade itself, following the steps in the upgrade documentation https //docs swimlane com/turbine upgrade guide/turbine 2614 upgrade instructions#red hat 8 is no longer supported beginning with tu https //docs swimlane com/turbine upgrade guide/turbine 2614 upgrade instructions#red hat 8 is no longer supported beginning with tu step 14 β ssh into node 1 ssh into any node in your deployment that node becomes node 1 for the rest of this guide β the node that drives the upgrade make sure your ssh sessions to node 2 and node 3 are also open and ready step 15 β scale down ekco and kotsadm rqlite kubectl n kurl scale deployment ekc operator replicas=0 kubectl n default scale statefulset kotsadm rqlite replicas=0 note ekco and kotsadm rqlite scale back up automatically once the script completes no manual action is required afterwards step 16 β run the turbine platform installer upgrade on node 1 curl ssl https //kurl sh/turbine turbine 26 1 4 | sudo bash s ha note if selinux, a firewall, or a proxy is configured, append your override yaml file (the one you created in step 13) to the command installer spec file=se yaml step 17 β accept the initial prompt the installer downloads kurl bin utils and checks whether a firewall, selinux, or swap is enabled it also reports the version transition β in this example, kubernetes 1 32 9 β 1 34 4 note at this point, all pods in the kurl namespace must be running ; otherwise, the upgrade will halt installer startup β checks firewall/selinux/swap and confirms the kubernetes 1 32 9 to 1 34 4 upgrade path installer startup β checks firewall/selinux/swap and confirms the kubernetes 1 32 9 to 1 34 4 upgrade path accept the prompt with y and continue step 18 β wait for package download the installer now downloads all the new packages no action is needed step 19 β review the preflight check results after the download, the installer runs preflight checks all checks should pass, as in the screenshot below host and in cluster preflight checks all passing host and in cluster preflight checks all pass note the ephemeral disk storage message relates to the /var/lib/kubelet partition it can be safely ignored and will not fail the upgrade step 20 β approve draining node 1 the installer prepares to upgrade the components, including kubernetes to upgrade the components, all pods on the node must be drained it first displays the component version table for the v1 32 β v1 33 11 step, then asks for confirmation accept the drain by entering y component upgrade table for v1 33 11 followed by the βdrain local node and apply upgrade?β prompt component upgrade table for v1 33 11 followed by the βdrain local node and apply upgrade?β prompt step 21 β expect pod disruption budget errors during the drain while draining the node, the installer may repeatedly display cannot evict pod as it would violate the pod's disruption budget repeated pod disruption budget eviction errors for postgresql 1 during node drain repeated pod disruption budget eviction errors for postgresql 1 during node drain this message is normal and will not fail the upgrade by default, some statefulset pods β such as the postgresql pods β have a pod disruption budget configured step 22 β expect βunable to fix broken packagesβ and approve the node 2 drain you may see an unable to fix broken packages error this is an os package installation message and can be ignored β it will not fail the upgrade immediately after node 1 finishes, you will be prompted to drain node 2 enter y to proceed broken packages message, node 1 upgraded to kubernetes 1 33 11, and the drain prompt for jer turb1 broken packages message, node 1 upgraded to kubernetes 1 33 11, and the drain prompt for jer turb1 step 23 β node 2 drains automatically after you enter y , the installer drains node 2 and continues with the installation step 24 β copy the upgrade script from node 1 to node 2 the installer now prints a script for you to run on node 2 critical while copying the upgrade script from node 1 to node 2, do not press enter or ctrl + c β either will terminate the upgrade partway through simply select the script text from the output and paste it into the node it is asking for run the upgrade script on the remote node to proceed curl fssl https //kurl sh/version/v2026 05 05 0/turbine turbine 26 1 4/upgrade sh | sudo bash s kubernetes version=1 33 11 docker registry ip=10 96 3 220 primary host=10 33 102 115 primary host=10 33 102 116 primary host=10 33 102 117 note if you are upgrading with a firewall, selinux, or a proxy, append your installer override yaml file to the end of the script before running it on node 2 example curl fssl https //kurl sh/version/v2026 05 05 0/turbine turbine 26 1 4/upgrade sh | sudo bash s kubernetes version=1 33 11 docker registry ip=10 96 3 220 primary host=10 33 102 115 primary host=10 33 102 116 primary host=10 33 102 117 installer spec file=override yaml step 25 β run the script on node 2 paste the copied script into your node 2 session and run it the upgrade script pasted and running in the node 2 (jer turb1) terminal the upgrade script pasted and running in the node 2 (jer turb1) terminal step 26 β wait while node 2 upgrades the installer shuts down all kubernetes pods on node 2 while installing kubernetes and the required components kubelet restarts all components once that upgrade finishes this process can take a while static pod upgrade output on node 2 β kube apiserver, kube controller manager, and kube scheduler upgraded static pod upgrade output on node 2 β kube apiserver, kube controller manager, and kube scheduler upgraded step 27 β confirm node 2 reached kubernetes 1 33 11 wait until installation is complete on node 2 node 2 reports βkubernetes node upgraded to 1 33 11β and βupgrade completeβ node 2 reports βkubernetes node upgraded to 1 33 11β and βupgrade completeβ step 28 β approve the node 3 drain on node 1 back on node 1, the script continues and prompts you to drain node 3, exactly as it did for node 2 node 1 confirms jer turb1 is upgraded and prompts to drain jer turb2 node 1 confirms jer turb1 is upgraded and prompts to drain jer turb2 step 29 β copy the upgrade script from node 1 to node 3 node 1 now prints a script to run on the third node critical do not press enter or ctrl + c on node 1 β just copy the script remember to append your override yaml file if needed, exactly as described in step 24 node 1 displaying the upgrade script to run on the remote node jer turb2 node 1 displaying the upgrade script to run on the remote node jer turb2 step 30 β wait for node 3 to complete installation begins on node 3, downloading the upgrade packages and completing the upgrade node 3 reports βkubernetes node upgraded to 1 33 11β and βupgrade completeβ node 3 reports βkubernetes node upgraded to 1 33 11β and βupgrade completeβ step 31 β node 1 begins the kubernetes 1 34 4 upgrade all three nodes are now on 1 33 11 node 1 continues with the upgrade to kubernetes 1 34 4 and drains itself to begin upgrading kubernetes and the required components component upgrade table for v1 34 4 followed by the βdrain local node and apply upgrade?β prompt component upgrade table for v1 34 4 followed by the βdrain local node and apply upgrade?β prompt step 32 β approve the node 1 drain enter y to drain node 1 and start the upgrade why the upgrade walks the cluster twice version 25 5 1 shipped kubernetes 1 32 9, and version 26 1 4 ships kubernetes 1 34 4 kubernetes only supports cumulative upgrades, so the cluster must reach 1 33 before it can move to 1 34 4 the upgrade script therefore upgrades kubernetes on all three nodes to 1 33 11 first, then repeats the walk for 1 34 4 step 33 β approve the node 2 drain for 1 34 4 once kubernetes and the required components are upgraded on node 1 to 1 34 4, you are prompted to drain node 2 node 1 upgraded to kubernetes 1 34 4, with the drain prompt for jer turb1 node 1 upgraded to kubernetes 1 34 4, with the drain prompt for jer turb1 step 34 β run the script on node 2 for 1 34 4 you are prompted to copy the upgrade script from node 1 and run it on node 2, exactly as described in step 24 do not press enter on node 1, and remember to add your yaml file if needed node 2 then completes its kubernetes upgrade to 1 34 4 node 2 reports βkubernetes node upgraded to 1 34 4β and βupgrade completeβ node 2 reports βkubernetes node upgraded to 1 34 4β and βupgrade completeβ step 35 β approve the node 3 drain for 1 34 4 the upgrade continues on node 1 and you are prompted to drain node 3 press enter to drain node 3 and obtain the script that upgrades its kubernetes to 1 34 4 jer turb1 confirmed at 1 34 4, with the drain prompt for jer turb2 jer turb1 confirmed at 1 34 4, with the drain prompt for jer turb2 step 36 β run the script on node 3 for 1 34 4 copy the script from node 1 and run it on node 3 when prompted, following the same instructions as step 24 once node 3βs kubernetes upgrade completes, node 1 continues to finalise the upgrade node 3 reports βkubernetes node upgraded to 1 34 4β and βupgrade completeβ node 3 reports βkubernetes node upgraded to 1 34 4β and βupgrade completeβ kubernetes is now upgraded on all three nodes other component (add on) upgrades still remain node 1 completes its add ons first, then provides a single script to run on both node 2 and node 3 at the same time step 37 β monitor node 1 for stuck pods during the add on phase important kubernetes pods are restarted during this final upgrade phase, so keep monitoring the progress on node 1 if the upgrade stays on one component for a long time with no movement, it means a pod is failing to restart on the node to resolve it, ssh to the node and use kubectl to find the affected pod, then forcibly terminate it for example kubectl get pods n kurl if the ekc pod is stuck in terminating, delete it kubectl delete pod ekc xxx force n kurl the upgrade will then continue step 38 β run the final script on node 2 and node 3 the upgrade is now complete on node 1, and you are prompted to run a script on the other two nodes node 1 displaying the final βrun this script on all remote nodes to apply changesβ command node 1 displaying the final βrun this script on all remote nodes to apply changesβ command add on unpacking completing on the remote nodes with βupgrade completeβ add on unpacking completing on the remote nodes with βupgrade completeβ step 39 β finish the infrastructure upgrade wait until the upgrade is complete on both node 2 and node 3 then return to node 1 and press enter to proceed the kubernetes and kurl component upgrade is now complete final installer output showing the kotsadm url, kubectl access instructions, and the node join commands final installer output showing the kotsadm url, kubectl access instructions, and the node join commands phase 3 β turbine application upgrade prerequisite β set the mongodb feature compatibility version step 1 β connect to mongodb as an admin replace default with your actual namespace if it differs kubectl n default exec it mongo 0 mongosh u admin p $(kubectl n default get secret mongo admin o jsonpath="{ data password}" | base64 decode) authenticationdatabase admin tls tlsallowinvalidcertificates admin note the command above normally logs you into the mongodb shell without prompting if you are prompted for the mongodb admin password, enter it you will land in the mongodb shell as shown below mongosh connected to the replica set, showing the rs0 \[direct primary] admin> prompt mongosh connected to the replica set, showing the rs0 \[direct primary] admin> prompt step 2 β check the current mongodb version use admin db system version find() expected output rs0 \[direct primary] admin> use admin db version() already on db admin rs0 \[direct primary] admin> db system version find() \[ { id 'featurecompatibilityversion', version '6 0' }, { id 'authschema', currentversion 5 } ] rs0 \[direct primary] admin> step 3 β verify the feature compatibility version read the featurecompatibilityversion value from the output above in this example it is 6 0, which is below the required 7 0 β so it must be updated step 4 β update the feature compatibility version if it is below 7 0 db admincommand({ setfeaturecompatibilityversion '7 0' }) expected output after the change rs0 \[direct primary] admin> use admin db version() already on db admin rs0 \[direct primary] admin> db system version find() \[ { id 'featurecompatibilityversion', version '7 0' }, { id 'authschema', currentversion 5 } ] rs0 \[direct primary] admin> re run db system version find() to confirm the value now reads 7 0 before continuing complete the turbine application upgrade from the admin console step 1 β log in and note the current version log in to the turbine platform installer dashboard https // \<swimlanedns> 8800 take note of the currently deployed version shown on the dashboard β in this example, 25 5 1 971 admin console dashboard showing the currently deployed version 25 5 1 971 admin console dashboard showing the currently deployed version 25 5 1 971 step 2 β check for updates and run the preflight checks click the version history tab, then click check for update version history tab with the check for update link highlighted version history tab with the check for update link highlighted in the pop up window, click go to the updated version , then click the preflight check icon on the version you intend to deploy version history list with the preflight check icon highlighted on the 26 1 4 1098 row version history list with the preflight check icon highlighted on the 26 1 4 1098 row the expected preflight check results are shown below preflight check results β all required checks passing, with an ephemeral storage warning preflight check results β all required checks passing, with an ephemeral storage warning note the ephemeral storage caution can be ignored if your backup size is less than 600 gb this check refers to the size of /var/lib/kubelet step 3 β deploy the target application version click deploy on your required version with the highest sequence number in this example, we are deploying version 26 1 4 1098 (sequence 14) deploy button highlighted on the 26 1 4 1098 row in version history deploy button highlighted on the 26 1 4 1098 row in version history step 4 β monitor the deployment click the dashboard tab to monitor the progress of the deployment dashboard tab highlighted in the admin console navigation dashboard tab highlighted in the admin console navigation you will see the newly deployed version shown in green dashboard showing 26 1 4 1098 as the currently deployed version dashboard showing 26 1 4 1098 as the currently deployed version a green version badge does not mean the upgrade is finished to see which component is currently being upgraded, click the details link resource status panel showing unavailable, degraded, and updating workloads mid upgrade resource status panel showing unavailable, degraded, and updating workloads mid upgrade you can also watch progress from the command line kubectl get pods w kubectl get pods w | grep v running the upgrade is complete once the application status shows ready admin console dashboard showing the application status as ready on version 26 1 4 1098 admin console dashboard showing the application status as ready on version 26 1 4 1098 you can confirm the same from the command line kubectl get pods a step 5 β verify the version in the turbine ui log in to the turbine application ui and check the version you should now see the upgraded version about turbine dialog showing version 26 1 4 about turbine dialog showing version 26 1 4 appendix a β version reference component before (turbine 25 5 1) intermediate after (turbine 26 1 4) turbine 25 5 1 971 β 26 1 4 1098 kubernetes 1 32 9 1 33 11 1 34 4 etcd 3 5 16 0 3 5 24 0 3 6 5 0 coredns 1 11 3 1 12 0 1 12 1 mongodb fcv 6 0 β 7 0 (minimum) kurl v2025 11 15 0 β v2026 05 05 0 appendix b β expected messages you can safely ignore message when it appears why it is safe cannot evict pod as it would violate the pod's disruption budget during node drains (step 21) statefulsets such as postgresql have a pdb configured by default the installer retries until eviction succeeds unable to fix broken packages manual intervention is required during os package installation (step 22) an os level package message that does not affect the kubernetes upgrade ephemeral disk usage /var/lib/kubelet warning installer preflight (step 19) and admin console preflight (step 2 2) advisory only ignore if your backup size is under 600 gb configmaps "kurl config" is forbidden during node upgrades transient rbac message while the control plane restarts failed to get unit file state for firewalld service installer startup (step 17) simply means firewalld is not installed on the host appendix c β troubleshooting the upgrade appears frozen on a single component for a long time a pod is failing to restart ssh to the node, identify the pod, and force delete it kubectl get pods n kurl kubectl delete pod ekc xxx force n kurl the upgrade terminated unexpectedly while i was copying a script you most likely pressed enter or ctrl + c on node 1 while it was waiting contact swimlane technical support before re running the installer packages fail to download behind a proxy or firewall confirm no proxy is exported (step 7) and that your installer override yaml is appended to every installer command, including the node 2 and node 3 scripts the installer halts immediately after startup verify that all pods in the kurl namespace are running (step 17 requirement), then re run preflight fails on the kubernetes version check in the admin console phase 2 has not completed successfully confirm all three nodes report v1 34 4 with kubectl get nodes owide before attempting the application upgrade