Backup and Restore on an Embedded Cluster with Snapshots

the swimlane platform installer (spi) uses snapshots to backup and restore your swimlane platform and data note the following items that list limitations with snapshots by default, snapshots are stored locally on the cluster for production environments, snapshots should be changed to be stored externally so that a total cluster outage does not result in the loss of snapshots taking a snapshot requires enough free disk space for a compressed archive of the swimlane database to be saved in ephemeral storage before it is uploaded to the snapshot destination free disk space on the cluster at /var/lib/kubelet should be greater than or equal to the size of the uncompressed database to ensure there is no disk pressure during the snapshot process instance snapshots are considered complete and are usable in disaster recovery scenarios application snapshots are the legacy snapshot method and are not usable for disaster recovery instance snapshots can be used in place of the legacy application snapshots snapshots cannot be restored to a different operating system than when the snapshot was taken for example, cluster snapshots based on centos cannot be restored to clusters based on ubuntu snapshots cannot be restored to different installation methods for example, online cluster snapshots cannot be restored to offline clusters aws s3 buckets with a bucket policy that requires the server side encryption header are not supported if you need to require server side encryption for objects, swimlane recommends that you enable default encryption on the bucket itself instead cleanup and removal of snapshots can only be done through the spi admin console snapshots tab removing data from the snapshot storage itself will result in data corruption and loss of snapshots configure snapshot storage from the top level spi ui, click the snapshots section and then click settings (on the lower right side of the screen) storing snapshots on amazon s3 requirements storing snapshots on amazon s3 requires an iam user or iam role to authenticate the bucket cannot have a bucket policy that requires the server side encryption header the recommended method to require server side encryption for objects is to enable default encryption on the bucket itself instead the following sample policy can be used after replacing ${bucket} with the aws arn of your bucket instructions change the destination drop down to amazon s3 set bucket to the name of the amazon s3 bucket to store snapshots in set region to the name of the aws region that the s3 bucket is in set path to the path in the s3 bucket that the snapshots should be stored under if your cluster nodes are aws ec2 instances and you want the aws permissions to access the s3 bucket managed by an iam instance role, check the use iam instance role checkbox and leave the access key id and access key secret fields blank if you need to use iam credentials to access the s3 bucket then set access key id and access key secret to the iam user's api credentials storing snapshots on azure blob storage requirements storing snapshots on azure blog storage requires an azure service principal and client secret to authenticate the storage account and service principal must be in the same subscription, tenant, and resource group required service principal permissions the service principal must have the storage account key operator service role role on the storage account the service principal must have the storage blob data contributor role on the storage container instructions change the destination drop down to azure blob storage configure your azure settings field details bucket set to the name of the azure storage container where you will store snapshots path set to the path in the azure storage container where the snapshots should be stored subscription id set to the azure subscription id where your resources are tenant id set to the azure tenant id where your resources are client id set to the client id of the azure application that the service principal is a part of client secret set to the value of the client secret generated under the azure service principal cloud name set to the azure cloud where your resources are resource group name set to the azure resource group name where your resources are storage account id set to the name of the azure storage account where the storage container is storing snapshots on google cloud storage requirements storing snapshots on google cloud storage requires requires a google cloud service account to authenticate the service account should have the storage objectadmin role on the bucket instructions change the destination drop down to google cloud storage set bucket to the name of the google storage bucket to store snapshots in set path to the path in the bucket that the snapshots should be stored under if your cluster nodes are google cloud vms and you want the aws permissions to access the google cloud storage bucket managed by an iam instance role, check the use iam instance role checkbox and leave the json file field blank if you need to use iam credentials to access the google cloud storage bucket then set json file to the json key for the service account storing snapshots on a host path requirements storing snapshots on a host path requires the host path storage destination should not be used for production environments they provide a security risk and the snapshots are not stored externally restoration will not be possible in the event of a total cluster loss the host path must be a dedicated directory do not use a partition used by a service like docker or kubernetes for ephemeral storage the host path directory must exist on each spi/tpi node the host path directory must be mounted from remote network storage in ha clusters as the host path configuration does not replicate snapshot data between nodes swimlane highly recommends using the nfs snapshot destination in cases where nfs network storage is available instead of host path the host path directory must be read writable by the user\ group 1001 1001 the host path cannot be used if your cluster requires pods to have resources, service account, affinity, node selectors, or tolerations defined this option creates a minio deployment in the namespace that swimlane is installed under to handle passing the snapshot data to the host path swimlane does not support changing any of those settings for this deployment instructions change the destination drop down to host path set host path to the directory on the cluster nodes that the snapshots should be stored under storing snapshots on nfs requirements storing snapshots on nfs requires supports nfsv3 and nfsv4 host/ip authentication must be used as username and password authentication is not supported the nfs server must be configured to allow access from all the nodes in the cluster the nfs directory must be owned by the user\ group 1001 1001 the target directory needs to be read writable by the user\ group 1001 1001 all the nodes in the cluster must have the necessary nfs client packages installed to be able to communicate with the nfs server for example, the nfs common package is a common package used on ubuntu any firewalls must allow traffic between the nfs server and clients nfs cannot be used if your cluster requires pods to have resources, service account, affinity, node selectors, or tolerations defined this option creates a minio deployment in the namespace that swimlane is installed under to handle passing the snapshot data to the host path and it is not currently supported to change any of those settings for this deployment instructions change the destination drop down to network file system (nfs) set server to the hostname or ip of the nfs server set path to the path on the nfs server that the snapshots should be stored under storing snapshots on other s3 compatible provider requirements storing snapshots on an s3 compatible provider requires an s3 compatible provider like min io https //min io/ the s3 compatible provider should be installed separately from the cluster nodes that swimlane is installed on to ensure that snapshots are stored externally from the cluster so they can be retrieved in the event of a total cluster loss instructions change the destination drop down to other s3 compatible storage set bucket to the name of the s3 compatible bucket to store snapshots in set path to the path in the s3 compatible that the snapshots should be stored under set access key id and access key secret to the credentials required to access the storage provider set endpoint to the required value for your storage provider important! if the snapshot endpoint is using https, it must present a certificate signed by a commonly trusted public certificate authority self signed and privately signed certificates are not supported at this time set region to the required value for your storage provider configure snapshots schedule and retention from the platform installer ui, click the snapshots tab the first time you access the snapshots tab you are prompted to either schedule a snapshot or to begin creating one after you have taken your first snapshot the schedule settings are available from the schedule link on the snapshots tab click schedule snapshots if you want to schedule automatic snapshots, click enable automatic scheduled snapshots you can schedule automatic snapshots hourly, daily, weekly, monthly, or you can setup a custom snapshot schedule using a cron expression important! the spi uses server time to schedule automatic snapshots under retention policy you can specify to have the spi to automatically delete older snapshots specify by a number of days, weeks, months, or years, and then click update schedule restore from a partial (application) snapshot on the snapshots page, you can review a list of all of your application snapshots under the "partial snapshots (application)" menu click the circular icon to restore a certain snapshot to your swimlane instance if you want to restore to the version of the snapshot, click restore from snapshot you are then prompted to enter the slug of the snapshot (confirming the slug name) enter swimlane platform important! restoring to the version you've selected will remove any data since the snapshot was made in addition, during restoration, your swimlane instance will not be available and you will not be able to use the swimlane installer ui until the restore completes return to the main ui once your application status displays ready , then you know that both the ui and your swimlane instance are back up and available again restore from a full (instance) snapshot in a non dr scenario instance snapshots can act as both instance level snapshots and as application level snapshots this section covers restoring the swimlane application with an instance snapshot on the snapshots page, you can review a list of all of your instance snapshots under the "full snapshots (instance)" menu click the circular icon and select "partial restore" to restore a certain snapshot to your swimlane instance you are then prompted to enter the slug of the snapshot (confirming the slug name) enter swimlane platform important! restoring to the version you've selected removes any data since the snapshot was made in addition, during restoration, your swimlane instance is not available and you will not be able to use the spi until the restore completes return to the main ui once your application status displays ready , then you know that both the spi ui and your swimlane instance are back up and available again restore from a full (instance) snapshot in a dr scenario after your new destination cluster nodes are installed, run the following command to configure your storage location aws s3 other s3 nfs if you are in an airgapped installation you also must provide the following arguments kotsadm namespace , kotsadm registry , registry password , registry username hostpath ensure that your snapshot volume mount is mounted and accessible on each individual node, then if you are in an airgapped installation you also must provide the following arguments kotsadm namespace , kotsadm registry , registry password , registry username a process takes place after configuring the snapshot storage location that discovers which snapshots are available for restore after a few minutes, you can run the following to show the backups that are available select the backup you want to restore from the list and restore it via this command