Backup and Restore on an Embedded Cluster with Snapshots

the turbine platform installer (tpi) uses snapshots to backup and restore your turbine platform and data note the following bulleted items that list limitations with snapshots by default, snapshots are stored locally on the cluster for production environments, snapshots should be changed to be stored externally so that a total cluster outage does not result in the loss of snapshot taking a snapshot requires enough free disk space for a compressed archive of the swimlane database to be saved in ephemeral storage before it is uploaded to the snapshot destination free disk space on the cluster at /var/lib/kubelet should be greater than or equal to the size of the uncompressed database to ensure there is no disk pressure during the snapshot process instance snapshots are considered complete and are usable in disaster recovery scenarios application snapshots are the legacy snapshot method and are not usable for disaster recovery instance snapshots can be used in place of the legacy application snapshots snapshots cannot be restored to a different operating system than when the snapshot was taken for example, cluster snapshots based on centos cannot be restored to clusters based on ubuntu snapshots cannot be restored to different installation methods for example, online cluster snapshots cannot be restored to offline clusters aws s3 buckets with a bucket policy that requires the server side encryption header are not supported if you need to require server side encryption for objects, swimlane recommends that you enable default encryption on the bucket itself instead cleanup and removal of snapshots can only be done through the tpi admin console snapshots tab removing data from the snapshot storage itself will result in data corruption and loss of snapshots configure snapshot storage from the top level tpi ui, click the snapshots section and then click settings (on the lower right side of the screen) storing snapshots on amazon s3 requirements storing snapshots on amazon s3 requires an iam user or iam role to authenticate the bucket cannot have a bucket policy that requires the server side encryption header the recommended method to require server side encryption for objects is to enable default encryption on the bucket itself instead the following sample policy can be used after replacing ${bucket} with the aws arn of your bucket instructions change the destination drop down to amazon s3 set bucket to the name of the amazon s3 bucket to store snapshots in set region to the name of the aws region that the s3 bucket is in set path to the path in the s3 bucket that the snapshots should be stored under if your cluster nodes are aws ec2 instances and you want the aws permissions to access the s3 bucket managed by an iam instance role, check the use iam instance role checkbox and leave the access key id and access key secret fields blank if you need to use iam credentials to access the s3 bucket then set access key id and access key secret to the iam user's api credentials storing snapshots on azure blob storage requirements storing snapshots on azure blog storage requires an azure service principal and client secret to authenticate the storage account and service principal must be in the same subscription, tenant, and resource group required service principal permissions the service principal must have the storage account key operator service role role on the storage account the service principal must have the storage blob data contributor role on the storage container instructions change the destination drop down to azure blob storage configure your azure settings field details bucket set to the name of the azure storage container where you will store snapshots path set to the path in the azure storage container where the snapshots should be stored subscription id set to the azure subscription id where your resources are tenant id set to the azure tenant id where your resources are client id set to the client id of the azure application that the service principal is a part of client secret set to the value of the client secret generated under the azure service principal cloud name set to the azure cloud where your resources are resource group name set to the azure resource group name where your resources are storage account id set to the name of the azure storage account where the storage container is storing snapshots on google cloud storage requirements storing snapshots on google cloud storage requires requires a google cloud service account to authenticate the service account should have the storage objectadmin role on the bucket instructions change the destination drop down to google cloud storage set bucket to the name of the google storage bucket to store snapshots in set path to the path in the bucket that the snapshots should be stored under if your cluster nodes are google cloud vms and you want the aws permissions to access the google cloud storage bucket managed by an iam instance role, check the use iam instance role checkbox and leave the json file field blank if you need to use iam credentials to access the google cloud storage bucket then set json file to the json key for the service account storing snapshots on a host path requirements storing snapshots on a host path requires the host path storage destination should not be used for production environments they provide a security risk and the snapshots are not stored externally restoration will not be possible in the event of a total cluster loss the host path must be a dedicated directory do not use a partition used by a service like docker or kubernetes for ephemeral storage the host path directory specified must exist on every node that the tpi pods can be scheduled on to ensure snapshots work even if pod scheduling changes the host path directory must be read writable by the user\ group 1001 1001 host path cannot be used if your cluster requires pods to have resources, service account, affinity, node selectors, or tolerations defined this option creates a minio deployment in the namespace that turbine is installed under to handle passing the snapshot data to the host path turbine does not support changing any of those settings for this deployment instructions change the destination drop down to host path set host path to the directory on the cluster nodes that the snapshots should be stored under storing snapshots on nfs requirements storing snapshots on nfs requires supports nfsv3 and nfsv4 host/ip authentication must be used as username and password authentication is not supported the nfs server must be configured to allow access from all the nodes in the cluster the nfs directory must be owned by the user\ group 1001 1001 the target directory needs to be read writable by the user\ group 1001 1001 all the nodes in the cluster must have the necessary nfs client packages installed to be able to communicate with the nfs server for example, the nfs common package is a common package used on ubuntu any firewalls must allow traffic between the nfs server and clients nfs cannot be used if your cluster requires pods to have resources, service account, affinity, node selectors, or tolerations defined this option creates a minio deployment in the namespace that turbine is installed under to handle passing the snapshot data to the host path and it is not currently supported to change any of those settings for this deployment instructions change the destination drop down to network file system (nfs) set server to the hostname or ip of the nfs server set path to the path on the nfs server that the snapshots should be stored under storing snapshots on other s3 compatible provider requirements storing snapshots on an s3 compatible provider requires an s3 compatible provider like https //min io/ the s3 compatible provider should be installed separately from the cluster nodes that turbine is installed on to ensure that snapshots are stored externally from the cluster so they can be retrieved in the event of a total cluster loss instructions change the destination drop down to other s3 compatible storage set bucket to the name of the s3 compatible bucket to store snapshots in set path to the path in the s3 compatible that the snapshots should be stored under set access key id and access key secret to the credentials required to access the storage provider set endpoint to the required value for your storage provider important! if the snapshot endpoint is using https, it must present a certificate signed by a commonly trusted public certificate authority self signed and privately signed certificates are not supported at this time set region to the required value for your storage provider configure snapshots schedule and retention from the platform installer ui, click the snapshots tab the first time you access the snapshots tab you are prompted to either schedule a snapshot or to begin creating one after you have taken your first snapshot the schedule settings are available from the schedule link on the snapshots tab click schedule snapshots if you want to schedule automatic snapshots, click enable automatic scheduled snapshots you can schedule automatic snapshots hourly, daily, weekly, monthly, or you can setup a custom snapshot schedule using a cron expression important! the tpi uses server time to schedule automatic snapshots under retention policy you can specify to have the tpi to automatically delete older snapshots specify by a number of days, weeks, months, or years, and then click update schedule restoring turbine snapshot in place copy the name of the backup you would like to restore from the tpi dashboard on the “snapshots” tag from one of the cluster nodes, run the following commands \# delete the existing swimlane tools resource kubectl delete sts swimlane tools \# wait for swimlane tools to finish scaling down sleep 30 \# proceed with the restore velero restore create from backup \<backup name> selector 'app=swimlane tools' restore volumes \# monitor the logs of the swimlane tools 0 pod to confirm the restore has completed \# press ctrl+c to stop following the logs kubectl logs swimlane tools 0 f \# note \# you may see the following error in the logs \# snapshot restore exception \[velero backup\ velero snapshot] snapshot does not exist \# \# this error can be safely ignored and does not impact the restore process \# if you see any other errors or are unsure whether the restore completed successfully, \# please open a ticket with swimlane support for further investigation \# once the restore is complete, restart all turbine pods to reconnect to the datastores kubectl delete pod l app kubernetes io/name=turbine restoring turbine snapshot in a disaster recovery scenario application deployment to the dr site is a manually controlled process the primary site is confirmed to be offline the standby/dr site can be kept infrastructure ready but application cold the dr recovery procedure requires newly deployed servers/vms without the tp application deployed it is possible to have the embedded kurl cluster deployed and kept updated the same system requirements apply for the dr site as for the primary site, the same number of nodes and same amount of resources the host os needs to also be the same snapshots should be stored in an external location, not the default setting which is to store them on the primary cluster locally an active active dr strategy is not possible, by design the ha deployment should be deployed on servers managed in such a way (borrowed from cloud nomenclature) so they are in different availability zones but in the same region for the sake of db latency issues and application stability in an active passive dr design it is not possible to have a synchronised or continuously deployed dr site alongside the primary site introducing a warm/hot standby that runs the same turbine application in both sites introduces a high risk of split brain scenarios even if database and queueing services are disabled, snapshot restores or upgrade actions in the turbine platform kots application implicitly starts all components which can begin processing and modifying data immediately this might cause data divergence that can not be reconciled install and deploy a new cluster using the same version installer that you want to recover in the tpi dashboard, re configure the new cluster with similar sizing and specifications note the database encryption key must remain the same as the restore data in the tpi dashboard, re configure snapshots to point to the same snapshot settings that were used during the backup after a few minutes you should see the available backups once the new deployment is ready and you are able to reach the turbine initialization ui when visiting the application, follow the same process as docid\ mfaxpicxqh16treih7fe2 configuring namespace access and memory limit see the https //docs replicated com/enterprise/snapshots velero installing config#increase the memory limit for detailed instructions on how to configure namespace access and the memory limit for velero