--- title: Runbook for Airgapped HA Turbine with HAProxy slug: runbook-for-airgapped-ha-turbine-with-haproxy docTags: createdAt: 2026-09-16T05:42:10.619Z --- Execute these carefully, as they involve enabling SELinux and Firewalld on the nodes and the HAProxy Load Balancer. ## I. Preparation ## 1. Prerequisites Install Rocky Linux 9.4 OS: Three nodes for Turbine and one node for the HAProxy load balancer. Artifacts & secrets to prepare: - Turbine YAML license for the KOTS Admin Console (.yaml file) - Turbine offline installer package (e.g. turbine-26.1.4\_1098.tar.gz) - Turbine Airgap bundle - Turbine application license for first UI login (.lic file) - Planned Turbine hostname (FQDN / DNS name) - Strong Mongo encryption key and Mongo password — store in a secure vault (effectively immutable after install) - HA load balancer (or VIP) address for the Kubernetes API / control plane - Odd-number capacity for replica sets (minimum 3 nodes for MongoDB and PostgreSQL) For Rocky Linux OS, the following utilities must be installed before installing Turbine: - nfs-utils - Conntrack-tools - Socat - Git - fio For the current system requirements refer to: System Requirements for an Embedded Cluster Install For the current installation guide refer to: Offline Embedded Cluster Installation ## 2. Verify Prerequisites: HA Small Model Components (refer to the Installation Guide for other sized models) | **Component** | **Value** | **How to Confirm?** | | ------------------- | ---------------------------------------------------------------- | -------------------------------------------------------------------- | | CPU | 16 CPU cores | $ lscpu \| grep 'CPU(s)' (should return 16) | | CPU Instruction Set | AVX required | $ lscpu \| grep -E "avx\|Flags"
(should return “avx” or “avx2”) | | Memory | 64 GB RAM | $ free -h | | Storage | 600 GB SSD / 3000 IOPS per node | $ df -h | | Pods | API: 3
Tasks: 3
Web: 3
MongoDB: 3
Reports: 3 | $ kubectl get pods | Other Requirements | **Requirement** | **How to Confirm?** | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | OS version | $ cat /etc/os-release
(returns OS name and version) | | Static IP address | $ ip addr \| grep -i dynamic
(should return nothing) | | Static hostname | $ hostnamectl –static | | No existing installation of kubernetes, docker, containerd (see exception below) | $ which kubectl kubeadm kubelet docker containerd && ps auxww \| egrep -i 'kubectl\|kubeadm\|kubelet\|docker\|containerd' \| grep -v grep (should return nothing) | | (Ubuntu 24.04) containerd v1.7 installed | $ containerd –version | | IPv4 forwarding enabled | $ sysctl net.ipv4.ip\_forward (should return net.ipv4.ip\_forward = 1) | | Utilities for Rocky Linux or RHEL
nfs-utils
conntrack-tools
socat
git
fio | $ dnf list installed nfs-utils conntrack-tools socat git fio | | Sudo/root access | $ sudo id (should return uid=0(root)) | | NUMA disabled | $ numactl –hardware or
$ lspcu \| grep -i numa | | Accurate system time | $ date -u | | selinux disabled (for the installation) | $ getenforce
(should return “Permissive”)
Allso will be prompted to turn it off during install | | All nodes must be in the same cloud provider region or physical data center network. | Customer confirmation | | Partition sizes
/: 50 GB
/var/lib/containerd: 100 GB
/var/lib/kubelet: 100 GB
/var/openebs: 300 GB
/var/log: 5 GB | $ df -h | | DNS record created for LB | Customer confirmation | | ACL Exceptions:
get.swimlane.io
k8s.kurl.sh
kurl.sh
kurl-sh.s3.amazonaws.com
registry.replicated.com
proxy.replicated.com
ghcr.io
registry.k8s.io
k8s.gcr.io
storage.googleapis.com
quay.io
cdn.quay.io
cdn01.quay.io
cdn02.quay.io
cdn03.quay.io
cdn04.quay.io
cdn05.quay.io cdn06.quay.io
replicated.app
auth.docker.io
registry-1.docker.io
production.cloudflare.docker.com
files.pythonhosted.org
``:6443 | $ curl -IL https\://`` | | Open TCP Ports
443
80
8800
22 | Ubuntu: $ sudo ufw status verbose
Rocky Linux: $ sudo firewall-cmd --list-all | | Open TCP Ports for HA
2379 - 2381
6443
8472 (TCP & UDP)
10248 - 10252
10257
10259 | Ubuntu: $ sudo ufw status verbose
Rocky Linux: $ sudo firewall-cmd --list-all | For the HAProxy Load Balancer, the following specifications are recommended: - **CPU:** 8 vCPUs - **Storage:** 80 GB SSD - **Memory:** 16–32 GB RAM (depending on the resources available in the customer's environment) ### 3. Configure the HAProxy Load Balancer server **A. Enable Firewalld to start** sudo systemctl enable firewalld **B. Ensure Firewalld is running** sudo firewall-cmd --state **C. Enable SELinux** - Change the status of the service in the /etc/selinux/config file to Enforcing. - Run the following command to make sure it is enabled and enforcing: sestatus **D. Configure firewalld** Run the below commands on the haproxy server: ```shell sudo firewall-cmd --remove-service cockpit sudo firewall-cmd --remove-service dhcpv6-client sudo firewall-cmd --remove-service ssh sudo firewall-cmd --zone public --add-port 22/tcp sudo firewall-cmd --zone public --add-port 443/tcp sudo firewall-cmd --zone public --add-port 6443/tcp sudo firewall-cmd --zone public --add-port 8800/tcp sudo firewall-cmd --runtime-to-permanent sudo firewall-cmd --reload ``` These commands modify the firewalld configuration on your Linux server. They remove some default allowed services and explicitly allow only the ports required for SSH, HTTPS, the Kubernetes API, and the Replicated Admin Console. ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/OmVV_fATuqjhlaoIgkBDe_image20.jpg" size="160" isUploading="false" initialPath="images/image20.jpg" githubPath="images/image20.jpg" width="800" height="395" darkWidth="800" darkHeight="395" showCaption="false"} **E. Configure SElinux** ```shell sudo semanage port --add --type http_port_t --proto tcp 6443 sudo semanage port --add --type http_port_t --proto tcp 8800 ``` ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/TRGNEKhZQ55huDg8g2wgw_image12.png" size="160" isUploading="false" initialPath="images/image12.png" githubPath="images/image12.png" width="800" height="52" darkWidth="800" darkHeight="52" showCaption="false"} **F. Download and install the additional required OS utilities ** If not already installed, on the online server download the RPMs for the additional required OS utilities: mkdir rpms cd rpms sudo dnf download --resolve nfs-utils conntrack-tools socat git fio scp -r \~/haproxy-rpms user@``:/tmp/ Copy those RPMs to each of the three nodes and install them: mkdir rpms cd rpms scp -r ``@``/rpms/\* . sudo dnf install ./\*.rpm **G. Install Haproxy Load Balancer on the node** On the online server, download the haproxy RPM file(s): cd /tmp dnf download --resolve haproxy On the haproxy server, copy over the haproxy RPM file(s) and install: cd /tmp scp -r ``@``:/tmp/\*.rpm . sudo dnf install ./\*.rpm Verify the installation: rpm -q haproxy nfs-utils **H. Test connectivity between nodes** for ip in `` `` ``; do echo "Testing $ip" nc -vz "$ip" 6443 nc -vz "$ip" 8800 nc -vz "$ip" 443 done ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/IZ88TSiwfeNhOg7Sh01An_image11.jpg" size="160" isUploading="false" initialPath="images/image11.jpg" githubPath="images/image11.jpg" width="800" height="515" darkWidth="800" darkHeight="515" showCaption="false"} **I. On each air-gapped node, check** ```shell firewall-cmd --state firewall-cmd --list-all ``` **J. Allow the required ports** ```shell firewall-cmd --permanent --add-port=443/tcp firewall-cmd --permanent --add-port=6443/tcp firewall-cmd --permanent --add-port=8800/tcp firewall-cmd --reload ``` ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/idEYRrtVNsdKwRsB8zWI1_image13.jpg" size="160" isUploading="false" initialPath="images/image13.jpg" githubPath="images/image13.jpg" width="800" height="608" darkWidth="800" darkHeight="608" showCaption="false"} **K. Retest from the HAProxy server** for ip in `` `` ``; do echo "Testing $ip" nc -vz "$ip" 6443 nc -vz "$ip" 8800 nc -vz "$ip" 443 done After opening the ports, these are the possible results: - Connection succeeded — service is running and reachable. - Connection refused — firewall is no longer blocking, but no service is listening yet. - No route to host — firewall or network rejection still exists. If Kubernetes/Turbine is not installed yet, Connection refused is expected until those services start. Now add your HAProxy config file: Validate it: haproxy -c -f /etc/haproxy/haproxy.cfg Expected: Configuration file is valid Then restart HAProxy: ```shell systemctl restart haproxy systemctl status haproxy ``` **L. **Configure haproxy per this document Example configuration for a Layer 4 HAProxy server: ```haproxy ################## # GLOBAL OPTIONS # ################## global defaults timeout client 30s timeout server 30s timeout connect 30s listen stats bind *:8080 mode http stats enable stats uri / stats hide-version # Turbine Frontend / Backend frontend turbine-frontend mode tcp bind *:80 # Optional https redirection bind *:443 http-request redirect scheme https unless { ssl_fc } # Optional https redirection default_backend turbine-backend backend turbine-backend mode tcp balance roundrobin option tcp-check server tpi-node-1 tpi-node-1.swimlane.io:443 check server tpi-node-2 tpi-node-2.swimlane.io:443 check server tpi-node-3 tpi-node-3.swimlane.io:443 check # Turbine Platform Installer UI Frontend / Backend frontend replicated-frontend mode tcp bind *:8800 default_backend replicated-backend backend replicated-backend mode tcp balance roundrobin option tcp-check server tpi-node-1 tpi-node-1.swimlane.io:8800 check server tpi-node-2 tpi-node-2.swimlane.io:8800 check server tpi-node-3 tpi-node-3.swimlane.io:8800 check # Kubernetes API Frontend / Backend frontend kube-api-frontend mode tcp bind *:6443 default_backend kube-api-backend backend kube-api-backend mode tcp balance roundrobin option tcp-check server tpi-node-1 tpi-node-1.swimlane.io:6443 check server tpi-node-2 tpi-node-2.swimlane.io:6443 check server tpi-node-3 tpi-node-3.swimlane.io:6443 check ``` Reload and restart the haproxy service: ```shell haproxy -c -f /etc/haproxy/haproxy.cfg systemctl enable haproxy systemctl start haproxy systemctl status haproxy ``` ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/4b9_EX0pHQGqS4dyuc3y__image15.jpg" size="160" isUploading="false" initialPath="images/image15.jpg" githubPath="images/image15.jpg" width="800" height="350" darkWidth="800" darkHeight="350" showCaption="false"} For a Kubernetes/Turbine cluster, net.ipv4.ip\_forward should be 1 To check: sysctl net.ipv4.ip\_forward **M. Configure IPv4 forwarding to persistent across reboots** ```shell cat <` 6443 ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/I5SgbBqqUlUNVq18XkhMM_image17.png" size="160" isUploading="false" initialPath="images/image17.png" githubPath="images/image17.png" width="800" height="136" darkWidth="800" darkHeight="136" showCaption="false"} curl -k-v ``:6443 **P. Create a patch YAML file to include firewall and SELinux (Optional):** ```yaml apiVersion: "cluster.kurl.sh/v1beta1" kind: "Installer" metadata: name: "patch" spec: kubernetes: HACluster: true loadBalancerAddress: "haproxy-iq.ts.swimlane.us:6443" firewalldConfig: firewalld: "enabled" bypassFirewalldWarning: true disableFirewalld: false hardFailOnFirewalld: false preserveConfig: false selinuxConfig: selinux: "enforcing" type: "targeted" preserveConfig: false disableSelinux: false ``` ## II. Install To perform an offline installation of Turbine with the Turbine Platform Installer, the offline installer package and an airgap bundle are required. The airgap bundle works with the Installer to allow an offline installation. Contact Swimlane Support for the Offline packages. Airgap installs require a jumpbox that has access to: - the internet - the airgapped network - a browser, from which you can access the Turbine Platform Installer Before you begin, copy the offline installer package to the jumpbox and then to each node in the cluster. **A. Copy the offline installer package to the instance that will become the first cluster primary node and untar the package with the following command:** tar zxvf ``.[tar.gz](http://tar.gz) ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/hJnwlLQapQvxVpiF9CqQn_image16.jpg" size="160" isUploading="false" initialPath="images/image16.jpg" githubPath="images/image16.jpg" width="800" height="116" darkWidth="800" darkHeight="116" showCaption="false"} **B. After untaring the above file, you will get:** ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/nmKVp1K7bOZA4eoP3Mqim_image19.jpg" size="160" isUploading="false" initialPath="images/image19.jpg" githubPath="images/image19.jpg" width="800" height="216" darkWidth="800" darkHeight="216" showCaption="false"} **C. Next, run this install command, with the “ installer-spec-file=patch.yaml” flag if needed:** cat install.sh | bash -s airgap ha ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/TFjzWdl43ceNevwwcIgPb_image18.jpg" size="160" isUploading="false" initialPath="images/image18.jpg" githubPath="images/image18.jpg" width="800" height="426" darkWidth="800" darkHeight="426" showCaption="false"} You get this prompt, so type Y and press Enter. ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/cQmala4zPQT90Y-NtjKhM_image1.jpg" size="160" isUploading="false" initialPath="images/image1.jpg" githubPath="images/image1.jpg" width="800" height="487" darkWidth="800" darkHeight="487" showCaption="false"} The script will complete on node #1 and generate the join commands. Ensure you use only the control plane (master) join command on the remaining two nodes so they are added as control plane nodes. **D. Once the installation completes, save the URL and password that the installation generates. In addition, copy the add primary node command to use to add additional masters.** **E. Untar the offline installer package in the second VM.** **F. Run the primary node join command that was output in step #4.** **G. Repeat steps #5 and #6 for the third and final VM. Add the installer-spec-file=patch.yaml if needed while running on the other nodes and follow the prompts.** ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/y4xDwscASnkDL7KT2hBsI_image3.jpg" size="160" isUploading="false" initialPath="images/image3.jpg" githubPath="images/image3.jpg" width="800" height="203" darkWidth="800" darkHeight="203" showCaption="false"} Once the install completes, save the URL and password that the install generates. ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/YZfP9O7lguOTvsdAEzG3D_image2.jpg" size="160" isUploading="false" initialPath="images/image2.jpg" githubPath="images/image2.jpg" width="800" height="192" darkWidth="800" darkHeight="192" showCaption="false"} **H. Run this command to make sure each node has successfully joined the Kubernetes cluster: ** kubectl get nodes You'll also receive the KOTS Admin Console (KOTSADM) login credentials, including the initial password. Please make sure to save these credentials, as the password is displayed only once during installation. You can change it later if needed. Also, run the following on node #1 to start using the kubectl commands: ```shell mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config ``` Kubectl get pods -A ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/-O5UnHiz6qPqmWCbpSSyo_image5.jpg" size="160" isUploading="false" initialPath="images/image5.jpg" githubPath="images/image5.jpg" width="800" height="328" darkWidth="800" darkHeight="328" showCaption="false"} **I. Log in to the Turbine Admin console (you saved it earlier: **[http://XXX.XXX.XXX.XXX:8800](http://XXX.XXX.XXX.XXX:8800)**).** **J. Click on Continue:** ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/u0EpGxeZphkTQWQZlkQKO_image4.png" size="160" isUploading="false" initialPath="images/image4.png" githubPath="images/image4.png" width="800" height="486" darkWidth="800" darkHeight="486" showCaption="false"} **K. Click on Advanced:** ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/_tVV6KPNqlnLDrwGvZFed_image7.png" size="160" isUploading="false" initialPath="images/image7.png" githubPath="images/image7.png" width="800" height="473" darkWidth="800" darkHeight="473" showCaption="false"} L. Continue with the Self-Signed Cert. You can also add your load balancer FQDN. ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/6XU9g_lAUtb6vtpXidkQd_image6.png" size="160" isUploading="false" initialPath="images/image6.png" githubPath="images/image6.png" width="800" height="460" darkWidth="800" darkHeight="460" showCaption="false"} **M. Upload the License file (.yaml) that you received from the Swimlane Support for any version** Note: The license file should be pinned to Stable-Multitenant on the Replicated Portal. ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/riREXkpLpHZ8Deaggb9oZ_image9.png" size="160" isUploading="false" initialPath="images/image9.png" githubPath="images/image9.png" width="800" height="581" darkWidth="800" darkHeight="581" showCaption="false"} **N. Upload the Airgap Bundle. It may require a couple of hours to upload depending on the size and network connectivity.** ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/Gv-Lw5trl-uzBgZOC_LoP_image8.png" size="160" isUploading="false" initialPath="images/image8.png" githubPath="images/image8.png" width="800" height="582" darkWidth="800" darkHeight="582" showCaption="false"} ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/pMEmGxHrpGcxmyr1JovZ7_image10.png" size="160" isUploading="false" initialPath="images/image10.png" githubPath="images/image10.png" width="800" height="496" darkWidth="800" darkHeight="496" showCaption="false"} **O. Enter your hostname:** ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/TzDtTzaUz4_OfXs17VZJJ_image22.jpg" size="160" isUploading="false" initialPath="images/image22.jpg" githubPath="images/image22.jpg" width="800" height="499" darkWidth="800" darkHeight="499" showCaption="false"} **P. Enter your Database Encryption Key and Password.** Note: The Database Encryption Key and Password can’t be changed after initially set. ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/I7PmIjfSyfh28GCZcqGvK_image21.jpg" size="160" isUploading="false" initialPath="images/image21.jpg" githubPath="images/image21.jpg" width="800" height="430" darkWidth="800" darkHeight="430" showCaption="false"} **Q. Enable HA and enter default values:** ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/n1I7GHPN2yGnddxhHQa3F_image24.png" size="119" isUploading="false" initialPath="images/image24.png" githubPath="images/image24.png" width="800" height="1080" darkWidth="800" darkHeight="1080" showCaption="false"} **R. Save the config and deploy** ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/tsprnCXfMpFFf_n3No_Ov_image23.png" size="160" isUploading="false" initialPath="images/image23.png" githubPath="images/image23.png" width="800" height="454" darkWidth="800" darkHeight="454" showCaption="false"} **S. Turbine deploying in progress:** ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/bMagRxa2-2X4mCpMSIXmu_image26.png" size="160" isUploading="false" initialPath="images/image26.png" githubPath="images/image26.png" width="800" height="466" darkWidth="800" darkHeight="466" showCaption="false"} **T. Last window of deployment status from the Admin Console:** ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/CTkj2XFqX1joTvfVKrwHN_image25.jpg" size="160" isUploading="false" initialPath="images/image25.jpg" githubPath="images/image25.jpg" width="800" height="507" darkWidth="800" darkHeight="507" showCaption="false"} **U. Verify all pods are healthy:** kubectl get pods -A -owide **V. Log in to the Turbine UI via your Load Balancer and Upload the Turbine UI license file (.lic)** ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/29E4oZ3aFj1SewabZcCjz_image28.jpg" size="156" isUploading="false" initialPath="images/image28.jpg" githubPath="images/image28.jpg" width="800" height="820" darkWidth="800" darkHeight="820" showCaption="false"} **W. Continue and create your admin credentials and log in:** ::Image[]{src="https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/9jqQbsE4bz1qbg9OJcrV6_image27.jpg" size="160" isUploading="false" initialPath="images/image27.jpg" githubPath="images/image27.jpg" width="800" height="418" darkWidth="800" darkHeight="418" showCaption="false"} Installation Complete!