---
title: Q&A's Consolidated Knowledge Base
slug: troubleshooting/qandas-consolidated-knowledge-base
createdAt: 2026-11-18T12:11:05.307Z
---

**How to use this page**: Expand only the topic you want to review. The content below preserves the current draft wording in a single page for quick scanning and comment collection.

:::Heading{depth="1" indent="1"}
Installation and networking
:::

### 1.1 Hostname for an embedded HA installation

**Overview**

On a three-node embedded cluster, the Hostname value in the installer is the name people type to open Turbine. When a load balancer sits in front of the nodes, that name is the load balancer FQDN.

**Applies to**

- Turbine on-prem
- Embedded Kubernetes cluster
- Three-node HA behind a load balancer

**Prerequisites**

- The FQDN that will be published for the Turbine UI

**What to enter**

On the installer screen, access the Turbine Platform Installer Admin Console configuration page and set the Hostname to the full FQDN of the URL that will be used to open the Turbine UI.

With a load balancer, that FQDN is the load balancer name. It is not the hostname of node 1.

The same name has to be present on the server certificate. See Names required on the server certificate.

### 1.2 Add a load balancer after a single-node installation

**Overview**

A cluster can be installed as one node with no load balancer, then expanded to three nodes. The load balancer address is supplied by running the installer again, which issues a new join command.

**Applies to**

- Turbine on-prem
- Embedded Kubernetes cluster

**Prerequisites**

- A completed single-node install
- The load balancer address that will front the cluster

**Procedure**

### Step 1: Install the first node without a load balancer

Run the single-node installer and do not pass a load balancer address. If the script prompts for one, leave it unset and continue.

### Step 2: Re-run the installer with the load balancer address

When you are ready to add the other nodes, run the installer again with the load balancer flag:

```javascript
--load-balancer-address=<address>
```

That run prints a new join command. Use that command to join the additional nodes. The join command from the original single-node install is not the one to use after the flag is added.

### 1.3 Names required on the server certificate

**Overview**

The certificate presented to browsers must contain every name clients use to open Turbine. On an HA cluster, this includes the load balancer FQDN. A certificate that lists only the three node names leaves the UI unreachable.

**Applies to**

- Turbine on-prem
- Embedded cluster with a load balancer
- Admin Console on port 8800 and the Turbine web UI

**Prerequisites**

- A certificate you can re-issue if the load balancer name is missing

**What the certificate must contain**

Add the load balancer FQDN as a Subject Alternative Name. The common name can carry the same FQDN. Node names alone are not enough, because the browser opens the load balancer name.

Set that hostname in the installer Config screen and import the certificate there. If the certificate does not match the name in the browser, the UI stops loading after this change.

**Company certificates**

Installation creates a self-signed certificate for the platform. Upload the company certificate through the Admin Console. Until that upload, browsers report the site as not secure.

A private CA has no extra CSR restriction beyond a correct common name. A wildcard certificate with only a CN is acceptable on that basis. Production and staging can use different certificates. One environment's certificate does not affect the other's internal traffic.

An expired certificate cannot be uploaded again. Download a certificate while it is still valid if you need a copy. After a custom certificate is applied, it replaces the generated self-signed certificate. There is no reset back to the install-time certificate. If a bad certificate makes the UI unreachable, the recovery is to upload a valid certificate.

### 1.4 Hostnames cannot be changed after installation

**Overview**

The node hostname chosen at install is permanent. Changing it later in the Admin Panel breaks node identity, TLS, and cluster networking.

**Applies to**

- Turbine on-prem
- Embedded Kubernetes cluster
- Single-node and HA

**Why the name is fixed**

Kubernetes registers the node, builds certificates, and routes traffic with that hostname. The installer limitation that node hostnames must be static applies to the Admin Panel, Config, as well as to the operating system hostname.

The sequence install, operate, then change the hostname in Config and deploy, is not supported on a standalone node or on an HA node.

Choose the final FQDN before the first deploy. If the published name must change, that is a reinstall planned with support, not an edit in Config.

### 1.5 Admin Console sign-in after a load balancer outage

**Overview**

After the load balancer in front of an HA cluster has been down for about an hour, the Playbook UI can still accept logins while the Admin Console login button does nothing. The Admin Console is a separate service from the Turbine application.

**Applies to**

- Turbine on-prem 25.3.x and later embedded HA
- Admin Console on port 8800

**Prerequisites**

- Access to the nodes to inspect pods

**What is failing**

The Admin Console is served by kotsadm and its database, kotsadm-rqlite. Those pods are not the Turbine application pods. The login page is static. When the Admin Console API is down, the button submits, and the page stays where it is. No error is shown.

On an embedded HA cluster, the Kubernetes API uses the load balancer address for traffic between nodes. About an hour without that address is enough for leader election and health checks to stall. kotsadm-rqlite is a Raft database and is one of the components that ends up unhealthy after that kind of outage.

**What to check**

Confirm the load balancer is passing traffic again, then check that the kotsadm and kotsadm-rqlite pods are Ready. A healthy Playbook UI does not tell you the Admin Console is healthy.

### 1.6 Traffic between nodes on an embedded HA cluster

**Overview**

Nodes in an embedded HA cluster talk to each other for Kubernetes itself and for every data service Turbine runs. The port list is the embedded-cluster system requirements. There is no second, private port matrix.

**Applies to**

- Turbine on-prem
- Embedded Kubernetes HA

**Services that cross the nodes**

Expect traffic for Velero, MinIO, Goldpinger, CoreDNS, kube-flannel, kube-apiserver, kube-scheduler, the image registry, the Swimlane services, MongoDB, Postgres, and RabbitMQ.

Control plane nodes also talk to each other on etcd directly. That path does not go through the load balancer.

Use the port tables in System requirements for an embedded cluster install. When a firewall change is blocked, capture on every node for the whole installer run and compare the capture with that table before the production window. Image registries also redirect to CDN hosts, and those CDN hosts have to be allowed with the registry name.

### 1.7 Where an in-cluster DNS lookup goes

**Overview**

A name such as ha-minio-3.ha-minio.minio.svc.cluster is resolved by CoreDNS. The packet you see for the lookup is from the calling pod to CoreDNS, not from the calling pod to the MinIO pod.

**Applies to**

- Turbine on-prem
- Embedded Kubernetes cluster

**How the lookup works**

The source is the pod that logged the lookup, often another MinIO pod in the same StatefulSet. The destination is the ClusterIP of the CoreDNS service in kube-system, UDP port 53. Large answers retry on TCP port 53.

After CoreDNS answers, the same pod opens a connection to the resolved pod IP. For MinIO that is usually port 9000 or 9001, depending on the service.

A firewall rule that allows the MinIO ports and blocks DNS to CoreDNS will show a name-resolution failure even though the MinIO port is open.

### 1.8 Outbound access for an embedded cluster

**Overview**

An embedded cluster needs the outbound allow list in the installer system requirements. Swimlane does not publish a second list of subdomains or debug URLs, because those names change.

**Applies to**

- Turbine on-prem
- Embedded cluster, online install and upgrade

**What to allow**

Allow the hosts in System requirements for an embedded cluster install. Allow the whole host, not a snapshot of individual URLs taken on one day.

Addresses used only while someone is debugging, such as a Kubernetes release-download host, are not part of the product allow list. Add a name only after a validation capture shows the installer requesting it.

There is no supported upgrade path that avoids the external names the online installer uses. For a network that must be requested in advance, run the upgrade once in a lab and capture every node from the moment before the script starts until the script exits. Submit that set with the documented list.

### 1.9 Capacity of a temporary single-node installation

**Overview**

A single node is a short-term installation. It is not a way to store more records than a medium three-node cluster, and it is not the production size.

**Applies to**

- Turbine on-prem
- Embedded cluster, single node

**Planning numbers**

Plan about 600 GB of SSD at 3000 IOPS. That size is aimed at about 5 million records, which is several months of retention at a moderate ingest rate.

Storage grows with record volume and with how long records are kept. When either one moves past that range, move the environment to a three-node cluster. A single node does not hold one third more than a medium three-node cluster.

:::Heading{depth="1" indent="1"}
Data and upgrade
:::

### 2.1 How MongoDB stores data on a three-node cluster

**Overview**

On a three-node embedded cluster, MongoDB is a replica set. Writes go to the primary. The secondaries replicate from the primary. Data is not split into three independent pieces.

**Applies to**

- Turbine on-prem
- Embedded HA with bundled MongoDB

**Failure of one node**

If the node that holds the primary fails, the replica set elects another primary. When the failed node comes back, new MongoDB pods on that node replicate from the current primary. Rebuild the node and let the replica set catch up. Do not copy files out of a stopped MongoDB pod and push them into another node by hand.

### 2.2 What an admin console snapshot contains

**Overview**

A snapshot taken in the Admin Console is a full copy of the application data and the KOTS data. It is one object. It is not an incremental backup, and the difference since the last snapshot cannot be copied on its own.

**Applies to**

- Turbine on-prem
- Embedded cluster snapshots

**Prerequisites**

- Enough storage for a full copy each time a snapshot is taken

**How to use it**

Take the snapshot before a change, and keep that full copy somewhere that survives the loss of the cluster. Restoring brings back the whole snapshot. There is no operation that restores only the records created after the snapshot.

A snapshot that remains on the cluster disks is not a disaster-recovery copy. The default storage location is local to the cluster. For production, the copy has to leave the cluster.

### 2.3 Restore a HostPath snapshot onto another server

**Overview**

A HostPath snapshot can be taken on one server and restored on another. HostPath storage is a lab and migration tool. It is not a production backup.

**Applies to**

- Turbine on-prem
- Embedded cluster using HostPath snapshot storage

**Prerequisites**

- A snapshot taken before the change
- A destination server installed with the same Turbine version

**Caution:** HostPath snapshots stay on a disk the cluster can see. They are not stored outside the environment. If the cluster is lost, a HostPath snapshot on that cluster is lost with it.

**Supported restores**

Single node to another single node is supported. Restoring onto a different server from the one that holds the files is supported, as long as the files are copied to the destination first.

Single node to HA is a two-step restore. Restore single node to single node and confirm the restore. Only then expand that restored node to HA. Do not point an HA install directly at a single-node HostPath snapshot as the first action.

### 2.4 Return an on-prem environment to an older Turbine version

**Overview**

Turbine on-prem has no in-place downgrade. An upgrade changes the database, the Kubernetes components, and the application. Those changes are not reversed by selecting an older version.

**Applies to**

- Turbine on-prem

**Prerequisites**

- A full backup taken before the upgrade

**What is supported**

Going back from 26.1.1 to 25.3.4, while keeping the upgraded platform, is not a supported path. The same rule applies between other versions.

Some application data can be rolled back from an Admin Console snapshot. The Kubernetes layer and the platform under that snapshot cannot.

The recovery from a failed upgrade is the full backup taken before the upgrade started. Take that backup, and store it off the cluster before you deploy.

### 2.5 Which version can receive a snapshot restore

**Overview**

A Velero or Admin Console snapshot restores onto the same Turbine version it was taken from. It is a disaster-recovery copy. It is not an upgrade.

**Applies to**

- Turbine on-prem
- Embedded cluster

**Prerequisites**

- The installer for the same version as the snapshot
- The same operating system and install method
- The database encryption key from the source environment

**Rules**

Build the new cluster with the same version installer as the snapshot. A snapshot from 25.3.2 restored into a fresh 26.2.3 installation is not supported. The new containers expect data that has already passed through each upgrade in between.

Match the operating system, the install method, and the database encryption key. Skipping intermediate versions by restoring old data into a new install leaves the schema and the platform out of step.

To move from 25.3.2 to 26.2.3, run the staged upgrade on the existing environment.

### 2.6 CPU features required by bundled MongoDB

**Overview**

Turbine ships MongoDB 5 and later. That MongoDB requires the AVX instruction set. A virtual CPU profile that stops at x86-64-v2 does not provide AVX, and the MongoDB pod never becomes ready.

**Applies to**

- Turbine on-prem with bundled MongoDB
- Turbine 26.x on a hypervisor such as Proxmox

**Prerequisites**

- A VM CPU profile that includes AVX, such as x86-64-v3

**What you see when AVX is missing**

The install or upgrade stops with the MongoDB pod in Init. The MongoDB process exits with an illegal instruction. Changing the VM CPU type from x86-64-v2 to x86-64-v3 clears it, because v3 includes AVX and v2 does not.

This matches the documented requirement that the CPU must be compatible with the bundled MongoDB, unless an external MongoDB is used instead.

### 2.7 MongoDB feature compatibility before Turbine 26.2.3

**Overview**

Turbine 26.2.3 upgrades the MongoDB version from 7.0.28 to 8.0.24 during the application deployment. Before that deploy, set the feature compatibility version to 8.0 while the server is still 7.0.28.

**Applies to**

- Turbine on-prem embedded HA
- Upgrade to Turbine 26.2.3 from a release that is still on MongoDB 7.0

**Prerequisites**

- A successful snapshot of the environment
- The upgrade tested on a non-production cluster first

**Caution:** Take a snapshot before you change the feature compatibility version or start the deployment. Run the upgrade in a test environment before production.

**Procedure**

### Step 1: Set the feature compatibility version

On the MongoDB admin database, while the server is still 7.0.28, run:

```javascript
db.adminCommand({ setFeatureCompatibilityVersion: "8.0", confirm: true })
```

Setting 8.0 before the binary upgrade is the expected order. The application deploys, then upgrades the MongoDB server.

### Step 2: Upgrade Turbine

Follow the Turbine 26.2.3 upgrade instructions. The guide is the supported procedure and the order of operations. The feature compatibility command above is the extra step those instructions require for this MongoDB move.

### 2.8 License to use while testing an upgrade

**Overview**

An upgrade test needs its own license. Pinning the production license to an older or newer build changes the version recorded for production.

**Applies to**

- Turbine on-prem
- Any lab used to rehearse a production upgrade

**Prerequisites**

- A license issued for the test environment

**Why the licenses stay separate**

The license channel is pinned to a version. If the production license is pinned to the version under test, production is pinned to that version as well.

Installing the same license on two environments makes the license appear as two instances. Version tracking for both environments becomes unreliable. Request a test license, start it on the version you need to install, and move only that license as the lab is upgraded.

:::Heading{depth="1" indent="1"}
Operations
:::

### 3.1 Memory use on a production node

**Overview**

Sustained memory use of 80 to 88 percent is an operational risk even when the UI still feels normal. MongoDB is expected to be the largest consumer. Headroom under 20 percent is what a spike will use.

**Applies to**

- Turbine on-prem
- Embedded HA

**How to read the numbers**

On a node with about 71 GB of RAM, MongoDB at roughly a third of that RAM is normal cache behavior for this platform. It is not, by itself, a leak.

At 80 to 88 percent used, a query spike, a playbook surge, or pod scheduling can push the node into swap or an out-of-memory kill. Latency, slow playbooks, and connector timeouts show up before the node dies.

A node that shows 0 bytes of swap and no out-of-memory events in the kernel log is not currently thrashing. It is still short of headroom. Gotenberg and Chromium on one node can be large together, because Gotenberg uses Chromium to render documents.

**What to collect**

On each node, record free memory, the top processes, and whether the kernel log contains an out-of-memory event. Alert on sustained use and on any swap or out-of-memory event. Add RAM or reduce retention before the node lives at this level.

### 3.2 Where files attached to records are stored

**Overview**

Files attached to records are stored in MongoDB with GridFS. They are not written as individual files in a directory you can watch from the operating system.

**Applies to**

- Turbine on-prem
- Embedded cluster with bundled MongoDB

**Path you will see**

The only path involved is the MongoDB data directory, /data/db, inside the MongoDB pod. The files in that directory are WiredTiger data files. An attachment cannot be picked out of that directory by name, and an endpoint product cannot block uploads by watching a folder.

Control of uploads belongs in the application, not in a filesystem monitor on the node.

### 3.3 Read container logs older than a support bundle

**Overview**

A support bundle contains the current container log. On a busy pod, this can be about a day. Older lines, if they still exist, are in the rotated files on the node that runs the pod.

**Applies to**

- Turbine on-prem
- Embedded kURL cluster

**Prerequisites**

- SSH to the cluster nodes
- sudo on those nodes

These commands only read files. They do not change workloads. Kubelet rotates logs by size, not by a fixed number of days. If the oldest line is already later than the time you need, that time is gone from the node. Ship logs off the cluster if you need a longer history. For sign-in and audited user actions, use the Audit Log API instead of the web pod log.

**Procedure**

### Step 1: Find the pod and the node

The label app.kubernetes.io/name=web does not match the web workload. Use one of these:

```javascript
kubectl get pods -A -o wide | grep swimlane-web
kubectl get pods -n default -l service=swimlane-web -o wide
```

Read the NODE column. SSH to that node. In HA, swimlane-web has a replica on more than one node. Repeat the next steps on each node. Access logs are per replica.

### Step 2: Confirm the files are on this node

```javascript
sudo find /var/log/pods/ -path '*swimlane-web*' -type f
```

An empty result means this is the wrong node. /var/log/pods/ exists only on the node where the pod is scheduled. sudo is required. Those directories are restricted to root and the container runtime.

### Step 3: See how far back the files go

Include rotated names, not only files that end in .log.

```javascript
sudo find /var/log/pods/ -path '*swimlane-web*' -type f -name '*.log*' -exec head -n 1 {} \;
sudo find /var/log/pods/ -path '*swimlane-web*' -type f -name '*.log*' -exec tail -n 1 {} \;
```

### Step 4: Search active and compressed logs

```javascript
sudo find /var/log/pods/ -path '*swimlane-web*' -type f -name '*.log*' -exec grep "SEARCH_TERM" {} +
sudo find /var/log/pods/ -path '*swimlane-web*' -type f -name '*.gz' -exec zgrep "SEARCH_TERM" {} +
```

The same steps work for other pods. Replace swimlane-web with the pod name prefix, such as swimlane-api or swimlane-tasks, and SSH to the node listed for that pod. After a restart, kubectl logs --previous still has the previous container. It does not have the rotated files from earlier days.

### 3.4 Health checks for an embedded cluster

**Overview**

Operational monitoring watches node health, disk, and pod readiness. It is a different list from the paths you exclude in an endpoint-security product.

**Applies to**

- Turbine on-prem
- Embedded cluster

**Prerequisites**

- A monitor that runs outside the cluster and can still alert when the cluster is unwell

**What to alert on**

Alert when a node is unhealthy, when any partition is short of free space, when CPU or memory stays high, when disk latency rises, when a pod is not Ready, when a Deployment or StatefulSet has fewer ready replicas than it should, and when a load balancer health check fails. Thresholds for those signals are yours to set, except for the eviction floors below.

**Floors that evict pods**

If a node crosses any of these, pods are removed from that node until the pressure is gone.

- Available memory under 100 MiB.
- Less than 15 percent free on the /var/lib/containerd partition.
- Less than 10 percent free on the /var/lib/kubelet partition.
- Less than 5 percent free inodes on the /var/lib/kubelet partition.

Kubernetes also starts evicting as a filesystem approaches about 85 percent used. Watch that line on a node that keeps everything on one partition.

**What not to install on the node**

Swimlane does not publish a list of Linux process names to monitor. Process names shared for an endpoint product are exclusions, so the product does not block containerd and kubelet. They are not a health checklist.

.NET process monitors such as Dynatrace are known to drive Turbine pods to high CPU and memory and to restarts. Remove that agent if pods begin to crash. Any other agent on the node is an unvalidated configuration. Support will still help with Turbine and will ask you to remove the agent when the fault traces back to it.

Built-in Kibana searches the audit stream logs-audit-swimlane. It does not index container logs or Kubernetes events, and it is not the alerting stack for pod failures. Audit records stay in Turbine for 7 days.

### 3.5 Send Turbine audit logs to Splunk

**Overview**

Turbine does not push audit logs to a SIEM. A job you run pulls them with the Audit Logs API and sends the JSON to Splunk. Container logs and Kubernetes events are a separate pipeline, collected from the nodes.

**Applies to**

- Turbine on-prem
- Splunk or another SIEM

**Prerequisites**

- An account admin personal access token
- A scheduler outside the cluster

**Audit logs**

Call the public audit API on a schedule. Page through the results and store them in Splunk. Seven days is the retention inside Turbine, so the job has to run often enough that a gap in the job does not drop a day.

```javascript
GET `https://<turbine-host>/api/public/audit/account/<account-id>/auditlogs?fromdate=<start>&todate=<end>&pageNumber=1&pageSize=100`
Private-Token: <personal-access-token>
```

Useful fields in the audit documents include ActionType, LogSource, LogType, Category, User, and Description.

**Container logs and pod health**

Those stay on the node and in Kubernetes. Collect them with your own log agent from /var/log/pods and from Kubernetes events. Swimlane does not publish a Splunk collector build for that data. Kibana on the cluster is for the audit stream only.

### 3.6 Certificates and licenses that expire inside Turbine

**Overview**

Several certificates inside an embedded cluster are created for 365 days from install or from the last renewal. They do not follow a calendar year. A cluster installed or renewed in 2025 reaches that date in 2026.

**Applies to**

- Turbine on-prem
- Embedded cluster, Turbine Platform 25.x

**Prerequisites**

- kubectl access to the cluster
- A shell on a control plane node for kubeadm

**What expires**

Envoy pod TLS in the projectcontour namespace, the files ca.crt and tls.crt, used for internal proxy routing. Validity is 365 days.

Kubernetes control plane certificates for the API server, kubelet, etcd, and controller manager, managed by kubeadm and kURL. Validity is 365 days.

The KOTS Admin Console TLS secret kotsadm-tls, for HTTPS on the installer UI. Validity is 365 days.

The platform license file and the application license file, on the dates in the contract.

Company certificates you uploaded for the web UI and for the Admin Console on port 8800. Those are on your calendar. The platform does not renew a certificate you supplied.

**What renews itself**

On embedded 25.x, the EKCO operator at 0.16.0 or newer renews Envoy pod TLS 30 days before it expires. Renewal is manual only when that operator is scaled down, or the cluster is not reconciled.

**Procedure**

### Step 1: Read the Envoy certificate dates

```javascript
for pod in $(kubectl get pods -n projectcontour -l app=envoy -o=name); do
  echo "$pod"
  kubectl exec "$pod" -n projectcontour -c envoy -- openssl x509 -enddate -noout -in /certs/ca.crt
  kubectl exec "$pod" -n projectcontour -c envoy -- openssl x509 -enddate -noout -in /certs/tls.crt
done
```

### Step 2: Confirm the renewer is running

```javascript
kubectl -n kurl get deployment ekc-operator
```

### Step 3: Read control plane certificate dates

On a control plane node:

```javascript
kubeadm certs check-expiration
```

Manual Envoy renewal, when EKCO is not doing it, is the procedure in How to renew Envoy pod TLS certificates: delete the envoycert secret, re-apply the contour-certgen manifest from that page, and roll the Envoy daemonset and the Contour deployment.

### 3.7 Recurring maintenance on a three-node embedded cluster

**Overview**

The work that stays on your calendar is the company TLS certificates, backups, database growth, and disk. Embedded platform certificates are rotated by the platform when its operator is running.

**Applies to**

- Turbine on-prem
- Three-node embedded HA

**Certificates**

Renew the Turbine web TLS certificate before it expires. Renew the Admin Console certificate on port 8800 when that certificate is one you uploaded. Check the internal certificates with the steps in Certificates and licenses that expire inside Turbine.

**Data and disk**

Take backups on a schedule and store them off the cluster. Purge old playbook run data when the database size starts to matter. Watch /var/openebs, which is where persistent volume data, including MongoDB, is kept on this install.

**Platform**

There is no separate Kubernetes patch cycle beyond staying on a supported Turbine version. Falling more than the support window behind is the failure mode to avoid.

### 3.8 Where persistent data lives on an embedded cluster

**Overview**

On-prem embedded clusters use the OpenEBS local storage class. Persistent volume claims, including MongoDB, RabbitMQ, and the Admin Console database, are directories under /var/openebs/local.

**Applies to**

- Turbine on-prem
- Embedded cluster

**Prerequisites**

- kubectl access

**Layout**

The storage class is openebs. PVC data is on the node at /var/openebs/local. Kubelet bind-mounts pod volumes under /var/lib/kubelet/pods. A single large partition is supported. Separate partitions for /var/openebs, /var/lib/containerd, and /var/lib/kubelet are the sizing recommendation, because one full filesystem then affects every component.

**What healthy looks like**

kubectl get pv and kubectl get pvc -A show STATUS Bound. For production data the persistent volume reclaim policy should be Retain. Directory names under /var/openebs/local match the PV names, the directories are not empty, and they are owned by root.

kubectl get pods -A shows Running pods as 1/1. Pods in Completed 0/1 are jobs that run and exit. That state is normal for those jobs. CrashLoopBackOff, Error, Pending, or a rising restart count on a database pod is the signal to investigate.

Watch filesystem use before it reaches about 85 percent. Eviction starts at that Kubernetes threshold, and earlier on the containerd and kubelet floors described in Health checks for an embedded cluster.

:::Heading{depth="1" indent="1"}
Remote Agent
:::

### 4.1 Install a Remote Agent on RHEL 9 with Podman

**Overview**

RHEL 9.6 with Podman 4.4 or newer is a supported Remote Agent host when the agent is installed with Quadlets. The same Quadlet install is supported on RHEL 9.2, 9.4, 9.5, and 9.6. The install script from the UI does not add the Quadlet flag. You append it.

**Applies to**

- Turbine Cloud and Turbine on-prem that register Remote Agents
- RHEL 9 with Podman
- A new agent host, or a host whose old non-Quadlet units have been removed

**Prerequisites**

- Podman 4.4 or newer and Python 3. Install podman-plugins when the Podman packaging uses that package.
- An Account Admin that can register an agent. Turn MFA off for that user, or exempt the user from forced SSO, for the duration of the install. The script cannot answer an MFA prompt. Turn MFA back on after the agent is online. The admin password is used only to register. The agent then uses the machine identity created at install.
- Outbound TCP 443 to the Turbine tenant URL and to quay.io, including quay.io CDN names. Connector images are pulled from the tenant. No inbound port is required.
- Root on the RHEL host.

**Prepare the host**

### Step 1: Confirm Podman, Python, and cgroup v2

```javascript
podman --version
python3 --version
stat -fc %T /sys/fs/cgroup
```

The cgroup command must print cgroup2fs. RHEL 9 defaults to cgroup v2. Hybrid mode and cgroup v1 make the installer stop. Do not install Docker on this host. If both are present, the installer selects Docker.

### Step 2: Allow user namespaces across reboot

```javascript
sudo sysctl user.max_user_namespaces=65536
echo "user.max_user_namespaces=65536" | sudo tee /etc/sysctl.d/99-user-namespaces.conf
sudo sysctl --system
```

### Step 3: Size the host

A standard agent needs at least 2 x64 vCPUs, 2 GB of RAM, and 20 GB on top of the operating system disk. Add CPU, memory, and disk when the playbook volume is heavy.

**Install**

### Step 4: Copy the script from Turbine

In Turbine open Settings, Tenant-Specific, Agents, View Agents, Deploy Agent. On Install an agent set AGENT\_NAME. Use letters, numbers, underscores, or hyphens, with no spaces. The product prefixes the name with agent-, so rhel96-prod-01 is registered as agent-rhel96-prod-01.

Leave every other field blank unless this host needs a proxy or a private CA. Copy the install script before you click away. The window closes when it loses focus, and the script is gone.

### Step 5: Add the Quadlet flag and run as root

On the RHEL host, append uppercase -Q to the copied command and run it with sudo. -Q is not in the command the UI copies. No second flag selects Podman. Quadlet exists only in Podman.

-Q writes the Quadlet unit files under /etc/containers/systemd/. systemd then starts the agent at boot. When the script asks, enter the Account Admin username and password.

### Step 6: Proxy, when the host has no direct internet

For an HTTP proxy, including a proxy that tunnels HTTPS, set both values to an http\:// URL. Use https\:// only when the proxy port itself is TLS. An https\:// URL against a plain HTTP proxy port fails, because the client starts TLS at the proxy.

```javascript
export HTTP_PROXY=http://proxy.example:8080
export HTTPS_PROXY=http://proxy.example:8080
export NO_PROXY=localhost,127.0.0.1,.internal.example
export http_proxy="$HTTP_PROXY"
export https_proxy="$HTTPS_PROXY"
export no_proxy="$NO_PROXY"
```

Export those variables in the same shell that runs the install script. The UI values are passed into the agent and into its update job. The shell still needs them to download the script and pull images. Put the same URLs in the HTTP\_PROXY and HTTPS\_PROXY fields in the install dialog.

With authentication, the form is `http://user:password@proxy.example`:8080. Percent-encode @, :, and / inside the password.

NO\_PROXY is a comma-separated list of hosts that skip the proxy. Leave the Turbine hostname and quay.io off that list when they must go through the proxy. Putting quay.io in NO\_PROXY sends image pulls direct, and the install fails on a proxy-only network. No extra proxy file under systemd is required.

### Step 7: Private CA, when the tenant certificate is private

Place the PEM files in a directory, mode 755 on the directory and 644 on the files. Set CA\_CERT\_PATH to that directory. Leave Ignore Insecure Certificates and Allow Insecure Certificates for Gateway Communication off. Those options disable TLS checks.

Leave SELinux enforcing. If the container cannot read the CA directory, label it:

```javascript
sudo chcon -R -t container_file_t /etc/pki/swimlane-ca
```

**Fields on the install dialog**

AGENT\_NAME is required. AGENT\_INITIAL\_POOLS is optional. Blank joins the $remote pool only. A new playbook action uses $default, so include $default when this agent should run actions without an edit to each action. AGENT\_CONTAINER\_NETWORK\_SUBNET stays blank unless that bridge range overlaps a route on the host. CRON\_SCHEDULE stays blank unless you need a specific update window, in which case it is a five-field cron expression.

**Confirm**

podman ps shows two containers for the agent, one name ending in -turbine-agent and one ending in -podman, and neither is restarting. Quadlet files exist under /etc/containers/systemd/. systemctl list-units --type=service shows the matching podman services active. In View Agents, the agent is listed, and the indicator is green. Reboot the host. Both containers and the green indicator must come back without a login. That is proof that Quadlet started the agent with the system.

**A host that already has a Podman agent**

This procedure is for a new agent. Quadlet and non-Quadlet agents on the same host are not supported. To convert an existing agent, stop and disable the old systemd units, remove the old unit files from /etc/systemd/system/, run systemctl daemon-reload, and add lowercase -q to the update command in crontab. Uppercase -Q is only for a new install.

### 4.2 Redundancy for Remote Agents

**Overview**

Remote Agent has no cluster of its own. Redundancy is two or more agents registered in the same pool. When one agent is down, another agent in that pool takes the work.

**Applies to**

- Turbine Cloud and Turbine on-prem with Remote Agents

**Prerequisites**

- At least two agent hosts

**How the pool behaves**

Register each agent with the same pool name. The usual pool for remote actions is $remote. The product does not prefer one agent in the pool. A remote action is placed on the pool queue, and the first connected agent with free capacity runs it. Adding agents raises how many actions can run at once.

An agent that shows offline stays in the pool. Traffic moves to the agents that are still connected. Bring the failed host back, or add a replacement into the same pool, and it starts taking work without a change to the playbooks.

A new playbook action targets $default unless you set another pool. Put an agent in $default as well as $remote when new actions should land on it without an edit.

:::Heading{depth="1" indent="1"}
Sign-in and access
:::

### 5.1 SAML service provider entity ID

**Overview**

The service provider entity ID uses the word tenant as a fixed path segment. You do not replace that word with the tenant ID.

**Applies to**

- Turbine Cloud and Turbine on-prem with SAML

**Prerequisites**

- The hostname clients use to open Turbine

**Value to give the identity provider**

Set the service provider entity ID and the reply URL to:

```javascript
`https://<turbine-hostname>/tenant/api/saml/consume`
```

The hostname is the real Turbine host. The word tenant stays as written. A tenant id in that position will not match the metadata Turbine publishes.

### 5.2 SAML sign-in with Microsoft Entra ID

**Overview**

A failed SAML login surfaces in Turbine as invalid credentials. The usual cause is a mismatch between the Entra application and the SAML settings stored in Turbine.

**Applies to**

- Turbine with SAML against Microsoft Entra ID

**Prerequisites**

- The Entra enterprise application SAML page
- The Turbine SAML settings page

**Checks**

The Identifier (entity ID) and the Reply URL in Entra are both `https://<turbine-hostname>/tenant/api/saml/consume`. Compare the hostname character by character with the service provider entity ID shown in Turbine. A different suffix, such as .spp against .app, fails the login.

In Entra, the Name ID is the user's email. In Turbine, the Name ID format is Email Address. If Entra sends a username and Turbine expects an email, the same invalid-credentials error is shown.

In Turbine, the identity provider entity ID matches the Entra identifier, and the SSO URL matches the Entra login URL.

If those four values match and login still fails, collect a HAR file of the attempt and the email address of the user who tried.

### 5.3 Turn off Hero AI and who can open Features

**Overview**

Hero AI is turned off for a tenant by a Super Admin. Account Admin, including a user with the Administrator role and access to all elements, does not see the Features page. Turbine has no control that promotes an Account Admin to Super Admin.

**Applies to**

- Turbine Cloud and Turbine on-prem

**Prerequisites**

- The Super Admin account created when Turbine was installed

**Turn Hero AI off**

### Step 1: Sign in as Super Admin

Super Admin is the account created at installation. If that account is still known, sign in with it.

### Step 2: Clear the feature

Open the Admin Panel, go to Settings, Tenant, Features, and turn Hero AI off. Sign out and back in, or hard-refresh, and confirm the AI menus are gone.

**When the install account is gone**

A customer cannot grant Super Admin from the product UI. Swimlane Support can grant it to a named user after the account owner confirms the user, the email address, and that the change is authorized.

Account Admin remains the role for tenant administration. It is a different flag from Super Admin, which is why Features can be visible in one environment and missing in another that looks fully privileged.

### 5.4 When failed passwords lock an account

**Overview**

The lockout count is the number of failed attempts that lock the account. The account locks on that attempt, not on the next one. A locked account stays locked until an administrator sets a new password.

**Applies to**

- Turbine Cloud and Turbine on-prem
- Local passwords. SSO users follow the identity provider.

**Prerequisites**

- An administrator who can open the user record

**The setting**

Admin, Account Settings, Sessions and Security, Password, Number of Allowed Failed Logins. The value is from 3 to 10. A value of 5 locks the account on the fifth wrong password. Attempts before that return invalid credentials.

**Unlock**

The account does not unlock after a wait. Forgot Password does not unlock it. An administrator opens the user, opens the Password tab, chooses Autogenerate new password, and clicks Send Message. The user signs in with that password and sets a new one.

:::Heading{depth="1" indent="1"}
Playbooks and connectors
:::

### 6.1 Size limits between playbook actions

**Overview**

A playbook can pass up to 20 MiB from one action to the next. A single file captured by a playbook can be up to 500 MiB. Canvas playbooks use the same two limits as classic playbooks.

**Applies to**

- Turbine Cloud and Turbine on-prem
- Classic and Canvas playbooks

**What to design for**

Stay under 20 MiB on the data handed from action to action. That includes large JSON bodies. Files use the separate 500 MiB capture limit. Splitting a large result, or writing it to a record or external store and passing a reference, is how a playbook stays inside the action limit.

### 6.2 Create a component from actions on the canvas

**Overview**

The canvas editor does not show a Create Component button. You group the selected actions, and that group is the component.

**Applies to**

- Turbine Canvas playbooks

**Prerequisites**

- A playbook open in the canvas editor

**Procedure**

### Step 1: Select the actions

Hold Shift and select each action that should be inside the component.

### Step 2: Group them

Right-click the selection and choose Group. The current canvas behavior is documented under Organising a playbook in the canvas. Older screenshots that show a Create Component button are from the previous editor.

### 6.3 Email someone when a playbook action fails

**Overview**

Turbine can send mail when an action fails. There is no tenant switch that sends a custom email for every failed action. You attach an On Failure path to each action that should notify. A separate orchestration alert can mail administrators when a run fails.

**Applies to**

- Turbine Cloud and Turbine on-prem
- Canvas and classic playbooks

**Prerequisites**

- A mail action that already works, such as a Microsoft Graph send

**Option A: Mail from the playbook**

### Step 1: Open the On Failure path

Open the playbook. On each action that should notify, hover over the action and choose the On Failure flow. That is the middle, red action-flow handle.

### Step 2: Add the mail action

On that path, add the Microsoft Mail or Graph action you already use, or the HTTP action you already use. Put the failure details in the subject and body. For an HTTP action, a script action, or a connector, the error fields are available as $actions.\<action>.result.error.errorMessage and $actions.\<action>.result.error.statusCode. Include the action name, the playbook name, and a timestamp.

### Step 3: Prove it

Save, force a failure on a non-production path, and confirm the message arrives. Use the same mail action and the same body on every On Failure path so a new action can reuse the pattern. Test in a staging tenant before you copy it into production playbooks.

**Option B: Alert when the run fails**

Admin Panel, Settings, Tenant-specific, Orchestration Alerts. Enable Send via Email and add the recipients. When a run fails, Turbine sends its own alert with a link to the run. Failures inside a 30-minute window can be folded into one message. This mail is the platform alert. It is not a Microsoft Graph message you compose.

Use both. Option A carries the error text you want. Option B tells an administrator that a run failed even when a playbook author forgot an On Failure path. Orchestration Health, under Runs, Events, and Alerts, is the screen for that operational view.

### 6.4 Export a package while playbooks are running

**Overview**

Exporting an application package does not stop playbooks and does not make a running playbook fail. The export reads the application definition. It does not take the orchestration engine down.

**Applies to**

- Turbine Cloud and Turbine on-prem

**If the export itself errors**

Treat that as an export failure, not as a playbook outage. A package that will not export often points to a referenced asset or trigger that no longer exists. Open the action that names the missing asset and select the current asset, or remove a trigger that points to a task that has been deleted. Raise the ticket at the severity the export outage deserves. The playbooks keep running while you do that.

### 6.5 Microsoft Graph call used by Get Emails from a Folder

**Overview**

The Get Emails from a Folder action calls the Microsoft Graph list-messages API for one mailbox folder.

**Applies to**

- Microsoft Graph Mail connector

**Prerequisites**

- A Graph credential that can read the mailbox

**Request**

```javascript
GET `https://graph.microsoft.com/v1.0/users/<mailbox>/mailFolders/<folder-id>/messages`
```

The folder ID can be the Graph ID of the folder, or a well-known name such as inbox, drafts, or sentitems.

### 6.6 How large an LDAP search result can be

**Overview**

The LDAP connector runs inside a connector container with a 1.5 GiB memory limit in production. A search that builds a result larger than that fails the action.

**Applies to**

- LDAP connector on Turbine

**What to change in the playbook**

Page the search. Request a bounded set of entries, process them, and request the next set. Raising the Remote Agent host memory does not raise this container limit. Stay under 1.5 GiB for a single action result.

### 6.7 Private CA certificates for SOAP and for HTTP actions

**Overview**

The SOAP connector can trust a private CA. The HTTP request action cannot borrow that capability. They are different actions.

**Applies to**

- SOAP connector
- HTTP request action

**SOAP**

Point the connector at the HTTPS WSDL and supply the CA certificate as base64. Turbine uses that CA to validate the SOAP service.

**HTTP request action**

A custom CA on the HTTP request action is not available. The SOAP connector cannot be placed in front of an HTTP action to provide it. Until that action accepts a CA, use a certificate signed by a public CA, or disable TLS validation on that action.

Copying a CA file into an agent pod is lost when the pod is recreated. It is not a durable configuration.

### 6.8 HTTP method the Splunk connector uses for search

**Overview**

The One-Shot Search action uses GET or POST based on the Splunk Enterprise Version field on the connector asset. Splunk 10 rejects GET on the export endpoint, so an empty version field fails with HTTP 405.

**Applies to**

- Splunk connector 1.3.1 and later, including 1.7.2-ca-cert.1

**Prerequisites**

- The Splunk major version of the search head

**Which method is chosen**

If Splunk Enterprise Version is empty or set below 9.3, the action calls GET /services/search/jobs/export.

If Splunk Enterprise Version is 9.3 or higher, including Splunk 10, the action calls POST /services/search/v2/jobs/export.

**What to set**

Set Splunk Enterprise Version on the asset to the real major version. For Splunk 10 that value must be 9.3 or higher so the connector uses POST. A 405 from Splunk, with GET disabled, means the asset is still on the GET path.

### 6.9 Output size of the Parse Attachment action

**Overview**

The Parse Attachment action returns at most 20 MiB. A larger parsed body fails the action even when the source file itself is within the playbook file capture limit.

**Applies to**

- Parse Attachment connector action

**What to do with a larger file**

The 500 MiB capture limit is for bringing a file into the playbook. The 20 MiB limit is the size of this action's output. Parse a smaller part of the file, or store the file and pass a reference, when the parsed result would exceed 20 MiB.

:::Heading{depth="1" indent="1"}
Product behavior
:::

### 7.1 Month control hidden on a date condition

**Overview**

On some Turbine Cloud builds, the month control of a date condition opens behind the condition builder. The condition is unchanged. The control is drawn underneath the dialog.

**Applies to**

- Turbine Cloud 26.1.4, 26.2.1, and 26.2.2

**Workaround**

Set the browser zoom to about 75 to 90 percent. The month control becomes visible and can be used. This is a stacking defect, and a fix was scheduled for Turbine Cloud 26.4.0.

The defect fixes in the 26.4.0 release do not require you to rebuild reports, dashboards, or playbooks.

### 7.2 Give a dashboard card its own time window

**Overview**

A card can ignore the dashboard date picker and use a window of its own. That is how you show, for example, records in one status that were created in the last 30 days and are at least 60 minutes old.

**Applies to**

- Turbine dashboards

**Prerequisites**

- A report that already filters on the status you care about

**Procedure**

### Step 1: Save the report

Filter the report to the status, save it, and add it to the dashboard as a card.

### Step 2: Detach the card from the dashboard date range

In the card, turn off Always use the dashboard's date range selection. Set the date field to First Created.

### Step 3: Set the window

Use a relative range. Start now-30d and end now-60m shows records created in the last 30 days that are at least 60 minutes old. Change the start when you want a longer history. Save the card.

### 7.3 See how many playbook actions have run

**Overview**

Action execution counts are on the Usage page in the Admin Panel. The total follows the date range you select.

**Applies to**

- Turbine Cloud and Turbine on-prem
- An administrator account

**Where to look**

Open the Admin Panel, Usage, Actions. Total Action Runs is the count for the selected range: today, 7 days, 30 days, 90 days, or a custom range. The same page breaks the count down by tenant and by playbook.

### 7.4 Source IP restriction on Turbine Cloud

**Overview**

Turbine Cloud cannot limit a tenant to a list of customer public IP addresses. There is no setting for that on the UI or the API.

**Applies to**

- Turbine Cloud

**What controls access**

Use SSO or SAML, two-factor authentication, and roles. Those settings are under Admin Panel, Settings, Sessions and Security. Inbound HTTPS is behind Swimlane's WAF. Customers do not add their own source addresses to that WAF.

### 7.5 PHP vulnerabilities and Turbine

**Overview**

Turbine on-prem and Turbine Cloud do not include a PHP runtime or PHP components. PHP advisories do not apply to either product.

**Applies to**

- Turbine on-prem, including 25.3.2 through 25.5.1
- Turbine Cloud, including 26.4.3 and other released versions

**What to tell a security review**

CVE-2026-109618 and CVE-2026-109619, and the PHP security releases from the same dates, do not affect Turbine. There is no PHP version to patch inside the product, and no Turbine setting to change.

### 7.6 Brotli vulnerability on a relay server

**Overview**

Turbine does not use Brotli compression. A Brotli package on the operating system of a Remote Agent host is outside the product. The product fix for CVE-2026-111497 is an upgrade to Turbine 26.5.1.

**Applies to**

- Turbine Cloud and Turbine on-prem
- Remote Agent hosts that also have an OS brotli package

**Two separate changes**

Upgrade Turbine to 26.5.1 for the product fix. Updating the operating system package brotli is a host change. Turbine does not link against that library, so the package update is not expected to change agent behavior. Apply it on a non-production agent first, the same way you would any other OS update.

Turbine has no configuration to disable Brotli. As an extra control at the edge, drop requests that carry the header Content-Encoding: br.

### 7.7 How long an on-prem Turbine release is supported

**Overview**

Support covers up to twelve calendar months of releases.

**Applies to**

- Turbine on-prem

See the current supported versions list: [https://docs.swimlane.com/turbine-release-notes/swimlane-turbine-end-of-life-product-versions](https://docs.swimlane.com/turbine-release-notes/swimlane-turbine-end-of-life-product-versions)
