Runbook for Airgapped HA Turbine with HAProxy

Execute these carefully, as they involve enabling SELinux and Firewalld on the nodes and the HAProxy Load Balancer.

I. Preparation

1. Prerequisites

Install Rocky Linux 9.4 OS: Three nodes for Turbine and one node for the HAProxy load balancer.

Artifacts & secrets to prepare:

Turbine YAML license for the KOTS Admin Console (.yaml file)

Turbine offline installer package (e.g. turbine-26.1.4_1098.tar.gz)

Turbine Airgap bundle

Turbine application license for first UI login (.lic file)

Planned Turbine hostname (FQDN / DNS name)

Strong Mongo encryption key and Mongo password — store in a secure vault (effectively immutable after install)

HA load balancer (or VIP) address for the Kubernetes API / control plane

Odd-number capacity for replica sets (minimum 3 nodes for MongoDB and PostgreSQL)

For Rocky Linux OS, the following utilities must be installed before installing Turbine:

nfs-utils

Conntrack-tools

Socat

For the current system requirements refer to: System Requirements for an Embedded Cluster Install

For the current installation guide refer to: Offline Embedded Cluster Installation

2. Verify Prerequisites:

HA Small Model Components (refer to the Installation Guide for other sized models)

Component	Value	How to Confirm?
CPU	16 CPU cores	$ lscpu \| grep 'CPU(s)' (should return 16)
CPU Instruction Set	AVX required	$ lscpu \| grep -E "avx\|Flags" (should return “avx” or “avx2”)
Memory	64 GB RAM	$ free -h
Storage	600 GB SSD / 3000 IOPS per node	$ df -h
Pods	API: 3 Tasks: 3 Web: 3 MongoDB: 3 Reports: 3	$ kubectl get pods

Other Requirements

Requirement	How to Confirm?
OS version	$ cat /etc/os-release (returns OS name and version)
Static IP address	$ ip addr \| grep -i dynamic (should return nothing)
Static hostname	$ hostnamectl –static
No existing installation of kubernetes, docker, containerd (see exception below)	$ which kubectl kubeadm kubelet docker containerd && ps auxww \| egrep -i 'kubectl\|kubeadm\|kubelet\|docker\|containerd' \| grep -v grep (should return nothing)
(Ubuntu 24.04) containerd v1.7 installed	$ containerd –version
IPv4 forwarding enabled	$ sysctl net.ipv4.ip_forward (should return net.ipv4.ip_forward = 1)
Utilities for Rocky Linux or RHEL nfs-utils conntrack-tools socat git fio	$ dnf list installed nfs-utils conntrack-tools socat git fio
Sudo/root access	$ sudo id (should return uid=0(root))
NUMA disabled	$ numactl –hardware or $ lspcu \| grep -i numa
Accurate system time	$ date -u
selinux disabled (for the installation)	$ getenforce (should return “Permissive”) Allso will be prompted to turn it off during install
All nodes must be in the same cloud provider region or physical data center network.	Customer confirmation
Partition sizes /: 50 GB /var/lib/containerd: 100 GB /var/lib/kubelet: 100 GB /var/openebs: 300 GB /var/log: 5 GB	$ df -h
DNS record created for LB	Customer confirmation
ACL Exceptions: get.swimlane.io k8s.kurl.sh kurl.sh kurl-sh.s3.amazonaws.com registry.replicated.com proxy.replicated.com ghcr.io registry.k8s.io k8s.gcr.io storage.googleapis.com quay.io cdn.quay.io cdn01.quay.io cdn02.quay.io cdn03.quay.io cdn04.quay.io cdn05.quay.io cdn06.quay.io replicated.app auth.docker.io registry-1.docker.io production.cloudflare.docker.com files.pythonhosted.org <LoadBalancerIP>:6443	$ curl -IL https://<URL>
Open TCP Ports 443 80 8800 22	Ubuntu: $ sudo ufw status verbose Rocky Linux: $ sudo firewall-cmd --list-all
Open TCP Ports for HA 2379 - 2381 6443 8472 (TCP & UDP) 10248 - 10252 10257 10259	Ubuntu: $ sudo ufw status verbose Rocky Linux: $ sudo firewall-cmd --list-all

For the HAProxy Load Balancer, the following specifications are recommended:

CPU: 8 vCPUs

Storage: 80 GB SSD

Memory: 16–32 GB RAM (depending on the resources available in the customer's environment)

3. Configure the HAProxy Load Balancer server

A. Enable Firewalld to start

sudo systemctl enable firewalld

B. Ensure Firewalld is running

sudo firewall-cmd --state

C. Enable SELinux

Change the status of the service in the /etc/selinux/config file to Enforcing.

Run the following command to make sure it is enabled and enforcing:

sestatus

D. Configure firewalld

Run the below commands on the haproxy server:

sudo firewall-cmd --remove-service cockpit
sudo firewall-cmd --remove-service dhcpv6-client
sudo firewall-cmd --remove-service ssh
sudo firewall-cmd --zone public --add-port 22/tcp
sudo firewall-cmd --zone public --add-port 443/tcp
sudo firewall-cmd --zone public --add-port 6443/tcp
sudo firewall-cmd --zone public --add-port 8800/tcp
sudo firewall-cmd --runtime-to-permanent
sudo firewall-cmd --reload

These commands modify the firewalld configuration on your Linux server. They remove some default allowed services and explicitly allow only the ports required for SSH, HTTPS, the Kubernetes API, and the Replicated Admin Console.

E. Configure SElinux

sudo semanage port --add --type http_port_t --proto tcp 6443
sudo semanage port --add --type http_port_t --proto tcp 8800

F. Download and install the additional required OS utilities

If not already installed, on the online server download the RPMs for the additional required OS utilities:

mkdir rpms

cd rpms

sudo dnf download --resolve nfs-utils conntrack-tools socat git fio

scp -r ~/haproxy-rpms user@<airgapped-server-IP>:/tmp/

Copy those RPMs to each of the three nodes and install them:

mkdir rpms

cd rpms

scp -r <user>@<online_server>/rpms/* .

sudo dnf install ./*.rpm

G. Install Haproxy Load Balancer on the node

On the online server, download the haproxy RPM file(s):

cd /tmp

dnf download --resolve haproxy

On the haproxy server, copy over the haproxy RPM file(s) and install:

cd /tmp

scp -r <user>@<online_server>:/tmp/*.rpm .

sudo dnf install ./*.rpm

Verify the installation:

rpm -q haproxy nfs-utils

H. Test connectivity between nodes

for ip in <node1-IP> <node2-IP> <node3-IP>; do echo "Testing $ip" nc -vz "$ip" 6443 nc -vz "$ip" 8800 nc -vz "$ip" 443 done

I. On each air-gapped node, check

firewall-cmd --state
firewall-cmd --list-all

J. Allow the required ports

firewall-cmd --permanent --add-port=443/tcp
firewall-cmd --permanent --add-port=6443/tcp
firewall-cmd --permanent --add-port=8800/tcp
firewall-cmd --reload

K. Retest from the HAProxy server

for ip in <node1-IP> <node2-IP> <node3-IP>; do echo "Testing $ip" nc -vz "$ip" 6443 nc -vz "$ip" 8800 nc -vz "$ip" 443 done

After opening the ports, these are the possible results:

Connection succeeded — service is running and reachable.

Connection refused — firewall is no longer blocking, but no service is listening yet.

No route to host — firewall or network rejection still exists.

If Kubernetes/Turbine is not installed yet, Connection refused is expected until those services start.

Now add your HAProxy config file:

Validate it:

haproxy -c -f /etc/haproxy/haproxy.cfg

Expected:

Configuration file is valid

Then restart HAProxy:

systemctl restart haproxy
systemctl status haproxy

L. Configure haproxy per this document

Example configuration for a Layer 4 HAProxy server:

##################
# GLOBAL OPTIONS #
##################

global
defaults
    timeout client          30s
    timeout server          30s
    timeout connect         30s
listen stats
    bind *:8080
    mode http
    stats enable
    stats uri /
    stats hide-version

# Turbine Frontend / Backend

frontend turbine-frontend
    mode tcp
    bind *:80 # Optional https redirection
    bind *:443
    http-request redirect scheme https unless { ssl_fc } # Optional https redirection
    default_backend turbine-backend

backend turbine-backend
    mode tcp
    balance roundrobin
    option tcp-check
    server tpi-node-1 tpi-node-1.swimlane.io:443 check
    server tpi-node-2 tpi-node-2.swimlane.io:443 check
    server tpi-node-3 tpi-node-3.swimlane.io:443 check

# Turbine Platform Installer UI Frontend / Backend

frontend replicated-frontend
    mode tcp
    bind    *:8800
    default_backend replicated-backend

backend replicated-backend
    mode tcp
    balance roundrobin
    option tcp-check
    server tpi-node-1 tpi-node-1.swimlane.io:8800 check
    server tpi-node-2 tpi-node-2.swimlane.io:8800 check
    server tpi-node-3 tpi-node-3.swimlane.io:8800 check

# Kubernetes API Frontend / Backend

frontend kube-api-frontend
    mode tcp
    bind    *:6443
    default_backend kube-api-backend

backend kube-api-backend
    mode tcp
    balance roundrobin
    option tcp-check
    server tpi-node-1 tpi-node-1.swimlane.io:6443 check
    server tpi-node-2 tpi-node-2.swimlane.io:6443 check
    server tpi-node-3 tpi-node-3.swimlane.io:6443 check

Reload and restart the haproxy service:

haproxy -c -f /etc/haproxy/haproxy.cfg
systemctl enable haproxy
systemctl start haproxy
systemctl status haproxy

For a Kubernetes/Turbine cluster, net.ipv4.ip_forward should be 1

To check:

sysctl net.ipv4.ip_forward

M. Configure IPv4 forwarding to persistent across reboots

cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes.conf
net.ipv4.ip_forward = 1
EOF

Apply it immediately:

sudo sysctl --system

N. Add required ports to selinux on all 3 nodes

semanage port --add --type http_port_t --proto tcp 2379;
semanage port --add --type http_port_t --proto tcp 2380;
semanage port --add --type http_port_t --proto tcp 6443;
semanage port --add --type http_port_t --proto udp 8472;
semanage port --add --type http_port_t --proto tcp 8800;
semanage port --add --type http_port_t --proto tcp 10250;

Configure the firewalld external zone on all 3 nodes:

firewall-cmd --zone external --add-masquerade;

Configure the firewalld public zone on all 3 nodes:

firewall-cmd --zone public --remove-service cockpit;
firewall-cmd --zone public --remove-service dhcpv6-client;
firewall-cmd --zone public --remove-service ssh;
firewall-cmd --zone public --add-port 22/tcp;
firewall-cmd --zone public --add-port 443/tcp;
firewall-cmd --zone public --add-port 2379/tcp;
firewall-cmd --zone public --add-port 2380/tcp;
firewall-cmd --zone public --add-port 6443/tcp;
firewall-cmd --zone public --add-port 8472/udp;
firewall-cmd --zone public --add-port 8800/tcp;
firewall-cmd --zone public --add-port 10250/tcp;

Configure the firewalld trusted zone on all 3 nodes:

firewall-cmd --zone trusted --add-interface cni0;
firewall-cmd --zone trusted --add-interface flannel.1;
firewall-cmd --zone trusted --add-interface kube-ipvs0;
firewall-cmd --zone trusted --add-port 2379/tcp;
firewall-cmd --zone trusted --add-port 2380/tcp;
firewall-cmd --zone trusted --add-port 4789/udp;
firewall-cmd --zone trusted --add-port 5000/tcp;
firewall-cmd --zone trusted --add-port 6783/tcp;
firewall-cmd --zone trusted --add-port 6783/udp;
firewall-cmd --zone trusted --add-port 6784/udp;
firewall-cmd --zone trusted --add-port 8080/tcp;
firewall-cmd --zone trusted --add-port 8472/udp;
firewall-cmd --zone trusted --add-port 10250/tcp;
firewall-cmd --zone trusted --add-port 10257/tcp;
firewall-cmd --zone trusted --add-port 10259/tcp;

Save and reload configured firewalld rules on all 3 nodes:

firewall-cmd --runtime-to-permanent;
firewall-cmd --reload;

O. Test connectivity between the three nodes and the haproxy on port 6443

Test TCP connectivity from each node to HAProxy:

Run this on each air-gapped node:

nc -vz <haproxy-IP> 6443

curl -k-v <haproxy url>:6443

P. Create a patch YAML file to include firewall and SELinux (Optional):

apiVersion: "cluster.kurl.sh/v1beta1"
kind: "Installer"
metadata:
  name: "patch"
spec:
  kubernetes:
    HACluster: true
    loadBalancerAddress: "haproxy-iq.ts.swimlane.us:6443"
  firewalldConfig:
    firewalld: "enabled"
    bypassFirewalldWarning: true
    disableFirewalld: false
    hardFailOnFirewalld: false
    preserveConfig: false
  selinuxConfig:
    selinux: "enforcing"
    type: "targeted"
    preserveConfig: false
    disableSelinux: false

II. Install

To perform an offline installation of Turbine with the Turbine Platform Installer, the offline installer package and an airgap bundle are required. The airgap bundle works with the Installer to allow an offline installation. Contact Swimlane Support for the Offline packages.

Airgap installs require a jumpbox that has access to:

the internet

the airgapped network

a browser, from which you can access the Turbine Platform Installer

Before you begin, copy the offline installer package to the jumpbox and then to each node in the cluster.

A. Copy the offline installer package to the instance that will become the first cluster primary node and untar the package with the following command:

tar zxvf <your file name>.tar.gz

B. After untaring the above file, you will get:

C. Next, run this install command, with the “ installer-spec-file=patch.yaml” flag if needed:

cat install.sh | bash -s airgap ha

You get this prompt, so type Y and press Enter.

The script will complete on node #1 and generate the join commands. Ensure you use only the control plane (master) join command on the remaining two nodes so they are added as control plane nodes.

D. Once the installation completes, save the URL and password that the installation generates. In addition, copy the add primary node command to use to add additional masters.

E. Untar the offline installer package in the second VM.

F. Run the primary node join command that was output in step #4.

G. Repeat steps #5 and #6 for the third and final VM. Add the installer-spec-file=patch.yaml if needed while running on the other nodes and follow the prompts.

Once the install completes, save the URL and password that the install generates.

H. Run this command to make sure each node has successfully joined the Kubernetes cluster:

kubectl get nodes

You'll also receive the KOTS Admin Console (KOTSADM) login credentials, including the initial password. Please make sure to save these credentials, as the password is displayed only once during installation. You can change it later if needed.

Also, run the following on node #1 to start using the kubectl commands:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Kubectl get pods -A

I. Log in to the Turbine Admin console (you saved it earlier: http://XXX.XXX.XXX.XXX:8800).

J. Click on Continue:

K. Click on Advanced:

L. Continue with the Self-Signed Cert. You can also add your load balancer FQDN.

M. Upload the License file (.yaml) that you received from the Swimlane Support for any version

Note: The license file should be pinned to Stable-Multitenant on the Replicated Portal.

N. Upload the Airgap Bundle. It may require a couple of hours to upload depending on the size and network connectivity.

O. Enter your hostname:

P. Enter your Database Encryption Key and Password.

Note: The Database Encryption Key and Password can’t be changed after initially set.

Q. Enable HA and enter default values:

R. Save the config and deploy

S. Turbine deploying in progress:

T. Last window of deployment status from the Admin Console:

U. Verify all pods are healthy:

kubectl get pods -A -owide

V. Log in to the Turbine UI via your Load Balancer and Upload the Turbine UI license file (.lic)

W. Continue and create your admin credentials and log in:

Installation Complete!