--- title: Backup and Restore on an Embedded Cluster with Snapshots slug: install-10x/backup-and-restore-on-an-embedded-cluster-with-snapshots docTags: createdAt: 2025-07-16T13:29:40.584Z --- The Swimlane Platform Installer (SPI) uses snapshots to backup and restore your Swimlane platform and data. :::hint{type="danger"} Note the following items that list limitations with snapshots: By default, snapshots are stored locally on the cluster. For production environments, snapshots should be changed to be stored externally so that a total cluster outage does not result in the loss of snapshots. Taking a snapshot requires enough free disk space for a compressed archive of the Swimlane database to be saved in ephemeral storage before it is uploaded to the snapshot destination. Free disk space on the cluster at `/var/lib/kubelet` should be greater than or equal to the size of the uncompressed database to ensure there is no disk pressure during the snapshot process. Instance snapshots are considered complete and are usable in disaster recovery scenarios. Application snapshots are the legacy snapshot method and are not usable for disaster recovery. Instance snapshots can be used in place of the legacy application snapshots. Snapshots cannot be restored to a different Operating System than when the snapshot was taken. For example, cluster snapshots based on CentOS cannot be restored to clusters based on Ubuntu. Snapshots cannot be restored to different installation methods. For example, online cluster snapshots cannot be restored to offline clusters. AWS S3 buckets with a bucket policy that requires the server-side encryption header are not supported. If you need to require server-side encryption for objects, Swimlane recommends that you enable default encryption on the bucket itself instead. Cleanup and removal of snapshots can only be done through the SPI admin console snapshots tab. Removing data from the snapshot storage itself will result in data corruption and loss of snapshots. ::: ## Configure Snapshot Storage From the top-level SPI UI, click the **Snapshots** section and then click **Settings** (on the lower right side of the screen). ::Image[]{src="https://api.archbee.com/api/optimize/511c8QBH-VHiwWnyzIwUb/7qBzxSAgFsV8mUAhgmbB3_snapshotsettingsdefault.png" size="48" width="3448" height="434" position="center" showCaption="false"} ## Storing Snapshots on Amazon S3 ::Image[]{src="https://api.archbee.com/api/optimize/511c8QBH-VHiwWnyzIwUb/ZFoYorv7-xjqB9qfMjTxQ_snapshotsettingss3.png" size="42" width="866" height="992" position="center" showCaption="false"} ### Requirements Storing snapshots on Amazon S3 requires: - An IAM user or IAM role to authenticate. - The bucket cannot have a bucket policy that requires the server-side encryption header. The recommended method to require server side encryption for objects is to enable default encryption on the bucket itself instead. - The following sample policy can be used after replacing `${BUCKET}` with the AWS ARN of your bucket: ### Instructions 1. Change the **Destination** drop down to **Amazon S3**. 2. Set **Bucket** to the name of the Amazon S3 bucket to store snapshots in. 3. Set **Region** to the name of the AWS region that the S3 bucket is in. 4. Set **Path** to the path in the S3 bucket that the snapshots should be stored under. 5. If your cluster nodes are AWS EC2 instances and you want the AWS permissions to access the S3 bucket managed by an IAM instance role, check the **Use IAM Instance Role** checkbox and leave the **Access Key ID** and **Access Key Secret** fields blank. 6. If you need to use IAM credentials to access the S3 bucket then set **Access Key ID** and **Access Key Secret** to the IAM user's API credentials. ## Storing Snapshots on Azure Blob Storage ::Image[]{src="https://api.archbee.com/api/optimize/511c8QBH-VHiwWnyzIwUb/vIqVtd2ldhAHHWSOwMjBE_snapshotsettingsazure.png" size="36" width="876" height="1524" position="center" showCaption="false"} ### Requirements Storing snapshots on Azure Blog Storage requires: - An Azure service principal and client secret to authenticate. - The storage account and service principal must be in the same subscription, tenant, and resource group. - Required service principal permissions: - The service principal must have the `Storage Account Key Operator Service Role` role on the storage account. - The service principal must have the `Storage Blob Data Contributor` role on the storage container. ### Instructions 1. Change the **Destination** drop down to **Azure Blob Storage**. 2. Configure your Azure settings: | **Field** | **Details** | | ------------------- | -------------------------------------------------------------------------------------- | | Bucket | Set to the name of the Azure storage container where you will store snapshots. | | Path | Set to the path in the Azure storage container where the snapshots should be stored. | | Subscription ID | Set to the Azure subscription ID where your resources are. | | Tenant ID | Set to the Azure tenant ID where your resources are. | | Client ID | Set to the client ID of the Azure application that the service principal is a part of. | | Client Secret | Set to the value of the client secret generated under the Azure service principal. | | Cloud Name | Set to the Azure cloud where your resources are. | | Resource Group Name | Set to the Azure resource group name where your resources are. | | Storage Account ID | Set to the name of the Azure storage account where the storage container is. | ## Storing Snapshots on Google Cloud Storage ::Image[]{src="https://api.archbee.com/api/optimize/511c8QBH-VHiwWnyzIwUb/bWNPCOPnwGR6JkC7uZmWK_snapshotsettingsgoogle.png" size="38" width="885" height="1800" position="center" showCaption="false"} ### Requirements Storing snapshots on Google Cloud Storage requires: - Requires a Google Cloud service account to authenticate. - The service account should have the `storage.objectAdmin` role on the bucket. ### Instructions 1. Change the **Destination** drop down to **Google Cloud Storage**. 2. Set **Bucket** to the name of the Google storage bucket to store snapshots in. 3. Set **Path** to the path in the bucket that the snapshots should be stored under. 4. If your cluster nodes are Google Cloud VMs and you want the AWS permissions to access the Google Cloud Storage bucket managed by an IAM instance role, check the **Use IAM Instance Role** checkbox and leave the **JSON File** field blank. 5. If you need to use IAM credentials to access the Google Cloud Storage bucket then set **JSON File** to the JSON key for the service account. ## Storing Snapshots on a Host Path ::Image[]{src="https://api.archbee.com/api/optimize/511c8QBH-VHiwWnyzIwUb/wq3WA4oDjTJcBTx91yxCS_snapshotsettingshostpath.png" size="42" width="880" height="578" position="center" showCaption="false"} ### Requirements Storing snapshots on a Host Path requires: - The host path storage destination should not be used for production environments. They provide a security risk and the snapshots are not stored externally. Restoration will not be possible in the event of a total cluster loss. - The host path must be a dedicated directory. Do not use a partition used by a service like Docker or Kubernetes for ephemeral storage. - The host path directory must exist on each SPI/TPI node. - The host path directory must be mounted from remote network storage in HA clusters as the Host Path configuration does not replicate snapshot data between nodes. Swimlane highly recommends using the NFS snapshot destination in cases where NFS network storage is available instead of Host Path. - The host path directory must be read-writable by the user\:group 1001:1001 - The host path cannot be used if your cluster requires pods to have resources, service account, affinity, node selectors, or tolerations defined. - This option creates a Minio deployment in the namespace that Swimlane is installed under to handle passing the snapshot data to the host path. Swimlane does not support changing any of those settings for this deployment. ### Instructions 1. Change the **Destination** drop down to **Host Path**. 2. Set **Host Path** to the directory on the cluster nodes that the snapshots should be stored under. ## Storing Snapshots on NFS ::Image[]{src="https://api.archbee.com/api/optimize/511c8QBH-VHiwWnyzIwUb/iaP1mUih2edA8Wlmq7Cfv_snapshotsettingsnfs.png" size="42" width="872" height="574" position="center" showCaption="false"} ### Requirements Storing snapshots on NFS requires: - Supports NFSv3 and NFSv4. - Host/IP authentication must be used as username and password authentication is not supported. - The NFS server must be configured to allow access from all the nodes in the cluster. - The NFS directory must be owned by the user\:group 1001:1001. - The target directory needs to be read-writable by the user\:group 1001:1001 - All the nodes in the cluster must have the necessary NFS client packages installed to be able to communicate with the NFS server. For example, the nfs-common package is a common package used on Ubuntu. - Any firewalls must allow traffic between the NFS server and clients - NFS cannot be used if your cluster requires pods to have resources, service account, affinity, node selectors, or tolerations defined. - This option creates a Minio deployment in the namespace that Swimlane is installed under to handle passing the snapshot data to the host path and it is not currently supported to change any of those settings for this deployment. ### Instructions 1. Change the **Destination** drop down to **Network File System (NFS)**. 2. Set **Server** to the hostname or IP of the NFS server. 3. Set **Path** to the path on the NFS server that the snapshots should be stored under. ## Storing Snapshots on Other S3-compatible Provider ::Image[]{src="https://api.archbee.com/api/optimize/511c8QBH-VHiwWnyzIwUb/ERk8t9WK5ukGab5sy_LZO_snapshotsettingss3-compatible.png" size="40" width="878" height="1124" position="center" showCaption="false"} ### Requirements Storing snapshots on an S3-Compatible Provider requires: - An S3-compatible provider like [min.io](https://min.io/). - The S3-compatible provider should be installed separately from the cluster nodes that Swimlane is installed on to ensure that snapshots are stored externally from the cluster so they can be retrieved in the event of a total cluster loss. ### Instructions 1. Change the **Destination** drop down to **Other S3-Compatible Storage**. 2. Set **Bucket** to the name of the S3-compatible bucket to store snapshots in. 3. Set **Path** to the path in the S3-compatible that the snapshots should be stored under. 4. Set **Access Key ID** and **Access Key Secret** to the credentials required to access the storage provider. 5. Set **Endpoint** to the required value for your storage provider. 6. **Important!** If the snapshot endpoint is using HTTPS, it must present a certificate signed by a commonly-trusted public certificate authority. Self-signed and privately signed certificates are not supported at this time. 7. Set **Region** to the required value for your storage provider. ## Configure Snapshots Schedule and Retention 1. From the Platform Installer UI, click the Snapshots tab. ::Image[]{src="https://api.archbee.com/api/optimize/511c8QBH-VHiwWnyzIwUb/smY0ch7wuNXpIUwSa9tys_nosnapshots.png" size="38" width="688" height="236" position="center" showCaption="false"} The first time you access the Snapshots tab you are prompted to either schedule a snapshot or to begin creating one. After you have taken your first snapshot the schedule settings are available from the **Schedule** link on the snapshots tab. 1. Click **Schedule Snapshots**. If you want to schedule automatic snapshots, click **Enable automatic scheduled snapshots**. ::Image[]{src="https://api.archbee.com/api/optimize/511c8QBH-VHiwWnyzIwUb/QjW2LleVJR6SGl6fXo2sI_enableautomaticsnapshots.png" size="50" width="583" height="523" position="center" showCaption="false" indent="2"} 2. You can schedule automatic snapshots hourly, daily, weekly, monthly, or you can setup a custom snapshot schedule using a cron expression. 3. **Important!** The SPI uses server time to schedule automatic snapshots. 4. Under **Retention policy** you can specify to have the SPI to automatically delete older snapshots. Specify by a number of days, weeks, months, or years, and then click **Update schedule**. ::Image[]{src="https://api.archbee.com/api/optimize/511c8QBH-VHiwWnyzIwUb/dbQmOCRNNJRPtlpYV19J0_schedulesnapshots.png" size="38" width="563" height="276" position="center" showCaption="false" indent="2"} ## Restoring Swimlane Snapshot In-Place Copy the name of the backup you would like to restore from the SPI Dashboard on the “Snapshots” tag. From one of the cluster nodes, run the following commands: :::BlockQuote \# Delete the existing swimlane-tools resource kubectl delete sts swimlane-tools \# Wait for swimlane-tools to finish scaling down sleep 30 \# Proceed with the restore velero restore create --from-backup \ --selector 'app=swimlane-tools' --restore-volumes \# Monitor the logs of the swimlane-tools-0 pod to confirm the restore has completed \# Press CTRL+C to stop following the logs kubectl logs swimlane-tools-0 -f \# Note: \# You may see the following error in the logs: \# snapshot\_restore\_exception: \[velero\_backup\:velero\_snapshot] snapshot does not exist \# \# This error can be safely ignored and does not impact the restore process. \# If you see any other errors or are unsure whether the restore completed successfully, \# please open a ticket with Swimlane Support for further investigation. \# Once the restore is complete, restart all Swimlane pods to reconnect to the datastores kubectl delete pod -l app.kubernetes.io/name=swimlane ::: ## Restoring Swimlane Snapshot in a Disaster Recovery Scenario :::hint{type="info"} - Application deployment to the DR site is a manually controlled process. - The primary site is confirmed to be offline. - The standby/DR site can be kept infrastructure-ready but application-cold. The DR recovery procedure requires newly deployed servers/VMs without the TP application deployed. It is possible to have the embedded kURL cluster deployed and kept updated. - The same system requirements apply for the DR site as for the primary site, the same number of nodes and same amount of resources. The host OS needs to also be the same. - Snapshots should be stored in an external location, not the default setting which is to store them on the primary cluster locally. - An active-active DR strategy is not possible, by design the HA deployment should be deployed on servers managed in such a way (borrowed from cloud nomenclature) so they are in different availability zones but in the same region for the sake of DB latency issues and application stability. - In an active-passive DR design it is not possible to have a synchronised or continuously deployed DR site alongside the primary site. Introducing a warm/hot standby that runs the same Turbine application in both sites introduces a high risk of split brain scenarios. Even if database and queueing services are disabled, snapshot restores or upgrade actions in the Turbine Platform KOTS application implicitly starts all components which can begin processing and modifying data immediately. This might cause data divergence that can not be reconciled. ::: - Install and deploy a new cluster using the same version installer that you want to recover. - In the SPI dashboard, re-configure the new cluster with similar sizing and specifications. Note: The Database Encryption Key must remain the same as the restore data. - In the SPI dashboard, re-configure Snapshots to point to the same snapshot settings that were used during the backup. After a few minutes you should see the available backups. - Once the new deployment is ready and you are able to reach the Swimlane Initialization UI when visiting the application, follow the same process as [Restoring Swimlane Snapshot In-Place](docId\:vy5YByIIifn9D2kZA8LRa).