Runbook for Airgapped HA Turbine with HAProxy
execute these carefully, as they involve enabling selinux and firewalld on the nodes and the haproxy load balancer i preparation 1\ prerequisites install rocky linux 9 4 os three nodes for turbine and one node for the haproxy load balancer artifacts & secrets to prepare turbine yaml license for the kots admin console ( yaml file) turbine offline installer package (e g turbine 26 1 4 1098 tar gz) turbine airgap bundle turbine application license for first ui login ( lic file) planned turbine hostname (fqdn / dns name) strong mongo encryption key and mongo password β store in a secure vault (effectively immutable after install) ha load balancer (or vip) address for the kubernetes api / control plane odd number capacity for replica sets (minimum 3 nodes for mongodb and postgresql) for rocky linux os, the following utilities must be installed before installing turbine nfs utils conntrack tools socat git fio for the current system requirements refer to system requirements for an embedded cluster install for the current installation guide refer to offline embedded cluster installation 2\ verify prerequisites ha small model components (refer to the installation guide for other sized models) component value how to confirm? cpu 16 cpu cores $ lscpu | grep 'cpu(s)' (should return 16) cpu instruction set avx required $ lscpu | grep e "avx|flags" (should return βavxβ or βavx2β) memory 64 gb ram $ free h storage 600 gb ssd / 3000 iops per node $ df h pods api 3 tasks 3 web 3 mongodb 3 reports 3 $ kubectl get pods other requirements requirement how to confirm? os version $ cat /etc/os release (returns os name and version) static ip address $ ip addr | grep i dynamic (should return nothing) static hostname $ hostnamectl βstatic no existing installation of kubernetes, docker, containerd (see exception below) $ which kubectl kubeadm kubelet docker containerd && ps auxww | egrep i 'kubectl|kubeadm|kubelet|docker|containerd' | grep v grep (should return nothing) (ubuntu 24 04) containerd v1 7 installed $ containerd βversion ipv4 forwarding enabled $ sysctl net ipv4 ip forward (should return net ipv4 ip forward = 1) utilities for rocky linux or rhel nfs utils conntrack tools socat git fio $ dnf list installed nfs utils conntrack tools socat git fio sudo/root access $ sudo id (should return uid=0(root)) numa disabled $ numactl βhardware or $ lspcu | grep i numa accurate system time $ date u selinux disabled (for the installation) $ getenforce (should return βpermissiveβ) allso will be prompted to turn it off during install all nodes must be in the same cloud provider region or physical data center network customer confirmation partition sizes / 50 gb /var/lib/containerd 100 gb /var/lib/kubelet 100 gb /var/openebs 300 gb /var/log 5 gb $ df h dns record created for lb customer confirmation acl exceptions get swimlane io k8s kurl sh kurl sh kurl sh s3 amazonaws com registry replicated com proxy replicated com ghcr io registry k8s io k8s gcr io storage googleapis com quay io cdn quay io cdn01 quay io cdn02 quay io cdn03 quay io cdn04 quay io cdn05 quay io cdn06 quay io replicated app auth docker io registry 1 docker io production cloudflare docker com files pythonhosted org \<loadbalancerip> 6443 $ curl il https // \<url> open tcp ports 443 80 8800 22 ubuntu $ sudo ufw status verbose rocky linux $ sudo firewall cmd list all open tcp ports for ha 2379 2381 6443 8472 (tcp & udp) 10248 10252 10257 10259 ubuntu $ sudo ufw status verbose rocky linux $ sudo firewall cmd list all for the haproxy load balancer, the following specifications are recommended cpu 8 vcpus storage 80 gb ssd memory 16β32 gb ram (depending on the resources available in the customer's environment) 3\ configure the haproxy load balancer server a enable firewalld to start sudo systemctl enable firewalld b ensure firewalld is running sudo firewall cmd state c enable selinux change the status of the service in the /etc/selinux/config file to enforcing run the following command to make sure it is enabled and enforcing sestatus d configure firewalld run the below commands on the haproxy server sudo firewall cmd remove service cockpit sudo firewall cmd remove service dhcpv6 client sudo firewall cmd remove service ssh sudo firewall cmd zone public add port 22/tcp sudo firewall cmd zone public add port 443/tcp sudo firewall cmd zone public add port 6443/tcp sudo firewall cmd zone public add port 8800/tcp sudo firewall cmd runtime to permanent sudo firewall cmd reload these commands modify the firewalld configuration on your linux server they remove some default allowed services and explicitly allow only the ports required for ssh, https, the kubernetes api, and the replicated admin console e configure selinux sudo semanage port add type http port t proto tcp 6443 sudo semanage port add type http port t proto tcp 8800 f download and install the additional required os utilities if not already installed, on the online server download the rpms for the additional required os utilities mkdir rpms cd rpms sudo dnf download resolve nfs utils conntrack tools socat git fio scp r /haproxy rpms user@ \<airgapped server ip> /tmp/ copy those rpms to each of the three nodes and install them mkdir rpms cd rpms scp r \<user> @ \<online server> /rpms/ sudo dnf install / rpm g install haproxy load balancer on the node on the online server, download the haproxy rpm file(s) cd /tmp dnf download resolve haproxy on the haproxy server, copy over the haproxy rpm file(s) and install cd /tmp scp r \<user> @ \<online server> /tmp/ rpm sudo dnf install / rpm verify the installation rpm q haproxy nfs utils h test connectivity between nodes for ip in \<node1 ip> \<node2 ip> \<node3 ip> ; do echo "testing $ip" nc vz "$ip" 6443 nc vz "$ip" 8800 nc vz "$ip" 443 done i on each air gapped node, check firewall cmd state firewall cmd list all j allow the required ports firewall cmd permanent add port=443/tcp firewall cmd permanent add port=6443/tcp firewall cmd permanent add port=8800/tcp firewall cmd reload k retest from the haproxy server for ip in \<node1 ip> \<node2 ip> \<node3 ip> ; do echo "testing $ip" nc vz "$ip" 6443 nc vz "$ip" 8800 nc vz "$ip" 443 done after opening the ports, these are the possible results connection succeeded β service is running and reachable connection refused β firewall is no longer blocking, but no service is listening yet no route to host β firewall or network rejection still exists if kubernetes/turbine is not installed yet, connection refused is expected until those services start now add your haproxy config file validate it haproxy c f /etc/haproxy/haproxy cfg expected configuration file is valid then restart haproxy systemctl restart haproxy systemctl status haproxy l configure haproxy per this document example configuration for a layer 4 haproxy server \################## \# global options # \################## global defaults timeout client 30s timeout server 30s timeout connect 30s listen stats bind 8080 mode http stats enable stats uri / stats hide version \# turbine frontend / backend frontend turbine frontend mode tcp bind 80 # optional https redirection bind 443 http request redirect scheme https unless { ssl fc } # optional https redirection default backend turbine backend backend turbine backend mode tcp balance roundrobin option tcp check server tpi node 1 tpi node 1 swimlane io 443 check server tpi node 2 tpi node 2 swimlane io 443 check server tpi node 3 tpi node 3 swimlane io 443 check \# turbine platform installer ui frontend / backend frontend replicated frontend mode tcp bind 8800 default backend replicated backend backend replicated backend mode tcp balance roundrobin option tcp check server tpi node 1 tpi node 1 swimlane io 8800 check server tpi node 2 tpi node 2 swimlane io 8800 check server tpi node 3 tpi node 3 swimlane io 8800 check \# kubernetes api frontend / backend frontend kube api frontend mode tcp bind 6443 default backend kube api backend backend kube api backend mode tcp balance roundrobin option tcp check server tpi node 1 tpi node 1 swimlane io 6443 check server tpi node 2 tpi node 2 swimlane io 6443 check server tpi node 3 tpi node 3 swimlane io 6443 check reload and restart the haproxy service haproxy c f /etc/haproxy/haproxy cfg systemctl enable haproxy systemctl start haproxy systemctl status haproxy for a kubernetes/turbine cluster, net ipv4 ip forward should be 1 to check sysctl net ipv4 ip forward m configure ipv4 forwarding to persistent across reboots cat <\<eof | sudo tee /etc/sysctl d/99 kubernetes conf net ipv4 ip forward = 1 eof apply it immediately sudo sysctl system n add required ports to selinux on all 3 nodes semanage port add type http port t proto tcp 2379; semanage port add type http port t proto tcp 2380; semanage port add type http port t proto tcp 6443; semanage port add type http port t proto udp 8472; semanage port add type http port t proto tcp 8800; semanage port add type http port t proto tcp 10250; configure the firewalld external zone on all 3 nodes firewall cmd zone external add masquerade; configure the firewalld public zone on all 3 nodes firewall cmd zone public remove service cockpit; firewall cmd zone public remove service dhcpv6 client; firewall cmd zone public remove service ssh; firewall cmd zone public add port 22/tcp; firewall cmd zone public add port 443/tcp; firewall cmd zone public add port 2379/tcp; firewall cmd zone public add port 2380/tcp; firewall cmd zone public add port 6443/tcp; firewall cmd zone public add port 8472/udp; firewall cmd zone public add port 8800/tcp; firewall cmd zone public add port 10250/tcp; configure the firewalld trusted zone on all 3 nodes firewall cmd zone trusted add interface cni0; firewall cmd zone trusted add interface flannel 1; firewall cmd zone trusted add interface kube ipvs0; firewall cmd zone trusted add port 2379/tcp; firewall cmd zone trusted add port 2380/tcp; firewall cmd zone trusted add port 4789/udp; firewall cmd zone trusted add port 5000/tcp; firewall cmd zone trusted add port 6783/tcp; firewall cmd zone trusted add port 6783/udp; firewall cmd zone trusted add port 6784/udp; firewall cmd zone trusted add port 8080/tcp; firewall cmd zone trusted add port 8472/udp; firewall cmd zone trusted add port 10250/tcp; firewall cmd zone trusted add port 10257/tcp; firewall cmd zone trusted add port 10259/tcp; save and reload configured firewalld rules on all 3 nodes firewall cmd runtime to permanent; firewall cmd reload; o test connectivity between the three nodes and the haproxy on port 6443 test tcp connectivity from each node to haproxy run this on each air gapped node nc vz \<haproxy ip> 6443 curl k v \<haproxy url> 6443 p create a patch yaml file to include firewall and selinux (optional) apiversion "cluster kurl sh/v1beta1" kind "installer" metadata name "patch" spec kubernetes hacluster true loadbalanceraddress "haproxy iq ts swimlane us 6443" firewalldconfig firewalld "enabled" bypassfirewalldwarning true disablefirewalld false hardfailonfirewalld false preserveconfig false selinuxconfig selinux "enforcing" type "targeted" preserveconfig false disableselinux false ii install to perform an offline installation of turbine with the turbine platform installer, the offline installer package and an airgap bundle are required the airgap bundle works with the installer to allow an offline installation contact swimlane support for the offline packages airgap installs require a jumpbox that has access to the internet the airgapped network a browser, from which you can access the turbine platform installer before you begin, copy the offline installer package to the jumpbox and then to each node in the cluster a copy the offline installer package to the instance that will become the first cluster primary node and untar the package with the following command tar zxvf \<your file name> tar gz http //tar gz b after untaring the above file, you will get c next, run this install command, with the β installer spec file=patch yamlβ flag if needed cat install sh | bash s airgap ha you get this prompt, so type y and press enter the script will complete on node #1 and generate the join commands ensure you use only the control plane (master) join command on the remaining two nodes so they are added as control plane nodes d once the installation completes, save the url and password that the installation generates in addition, copy the add primary node command to use to add additional masters e untar the offline installer package in the second vm f run the primary node join command that was output in step #4 g repeat steps #5 and #6 for the third and final vm add the installer spec file=patch yaml if needed while running on the other nodes and follow the prompts once the install completes, save the url and password that the install generates h run this command to make sure each node has successfully joined the kubernetes cluster kubectl get nodes you'll also receive the kots admin console (kotsadm) login credentials, including the initial password please make sure to save these credentials, as the password is displayed only once during installation you can change it later if needed also, run the following on node #1 to start using the kubectl commands mkdir p $home/ kube sudo cp i /etc/kubernetes/admin conf $home/ kube/config sudo chown $(id u) $(id g) $home/ kube/config kubectl get pods a i log in to the turbine admin console (you saved it earlier http //xxx xxx xxx xxx 8800 http //xxx xxx xxx xxx 8800 ) j click on continue k click on advanced l continue with the self signed cert you can also add your load balancer fqdn m upload the license file ( yaml) that you received from the swimlane support for any version note the license file should be pinned to stable multitenant on the replicated portal n upload the airgap bundle it may require a couple of hours to upload depending on the size and network connectivity o enter your hostname p enter your database encryption key and password note the database encryption key and password canβt be changed after initially set q enable ha and enter default values r save the config and deploy s turbine deploying in progress t last window of deployment status from the admin console u verify all pods are healthy kubectl get pods a owide v log in to the turbine ui via your load balancer and upload the turbine ui license file ( lic) w continue and create your admin credentials and log in installation complete!