Backup and Restore on an Existing Cluster with Snapshots

the turbine installer uses velero and restic as the underlying technology for snapshots and restores note the following items that list limitations with snapshots taking a snapshot requires enough free disk space for a compressed archive of the swimlane database to be saved in ephemeral storage before it is uploaded to the snapshot destination free disk space on the cluster at /var/lib/kubelet should be greater than or equal to the size of the uncompressed database to ensure there is no disk pressure during the snapshot process instance snapshots are considered complete and are usable in disaster recovery scenarios application snapshots are the legacy snapshot method and are not usable for disaster recovery instance snapshots can be used in place of the legacy application snapshots snapshots cannot be restored to a different namespace than when the snapshot was taken snapshots cannot be restored to different installation methods for example, online cluster snapshots cannot be restored to offline clusters aws s3 buckets with a bucket policy that requires the server side encryption header are not supported if you need to require server side encryption for objects, swimlane recommends that you enable default encryption on the bucket itself instead cleanup and removal of snapshots can only be done through the turbine installer admin console snapshots tab removing data from the snapshot storage itself will result in data corruption and loss of snapshots install velero see velero's customize resource requests and limits https //velero io/docs/v1 9/customize installation/#customize resource requests and limits if you need to set the resource requests and limits for the velero and restic pods install velero (online) velero install instructions if your cluster has access to the internet download velero 1 11 0 for the os version from which you run kubectl commands https //github com/vmware tanzu/velero/releases/tag/v1 11 0 untar the file and then change directory into the uncompressed directory tar zxf \<filename> tar gz cd \<directory> install velero with the necessary plugins and placeholder information create an empty credentials file to pass to the install command with touch placeholder credentials the velero binary must be executable by your user install velero into the velero namespace with the following command the bucket, prefix, credentials, and bucket region settings are set as placeholders in this command just to get velero installed you will update the snapshot storage destination later through the turbine installer ui if the kubeconfig environment variable is not set you will need to add kubeconfig /path/to/kube/config to the velero install command above so that it can authenticate to your cluster install velero (offline/airgapped) use these velero install instructions if your cluster has no access to the internet download velero 1 11 0 to the jumpbox for the jumpbox's os version https //github com/vmware tanzu/velero/releases/tag/v1 11 0 untar the file and then change directory into the uncompressed directory tar zxf \<filename> tar gz cd \<directory> download the velero images and push them to your private registry from your jumpbox create the velero restore helper, configmap, to ensure that the velero helper init containers pull from the private registry create the file velero restore helper configmap yaml with the following contents create the config map by running kubectl apply f velero restore helper configmap yaml see velero's customize restore helper container https //velero io/docs/v1 9/restic/#customize restore helper container if you need to set the resource requests and limits for the velero restore helper container install velero with the necessary plugins and placeholder information create an empty credentials file to pass to the install command with touch placeholder credentials the velero binary must be executable by your user install velero into the velero namespace with the following command the bucket, prefix, credentials, and bucket region settings are set as placeholders in this command just to get velero installed you will update the snapshot storage destination later through the turbine installer ui the velero install uses the kubeconfig environment variable to authenticate to the cluster if that environment variable is not set you must add the argument kubeconfig /path/to/kube/config to the install command above install velero (openshift) install velero using the install velero (online) instructions above set the required openshift policy for the velero service account to let the velero pods run oc adm policy add scc to user privileged z velero n velero update the velero restic daemonset to run the pods with the required security context configure snapshot storage from the top level turbine installer ui, click the snapshots section and then the settings & schedule link on the right side when the snapshot settings are updated the velero and restic pods are restarted to apply the new settings this can take several seconds and can temporarily make the snapshot pages show that velero isn't running once the pods are restarted, the snapshot pages correctly display storing snapshots on amazon s3 requirements storing snapshots on amazon s3 requires an iam user or iam role to authenticate the bucket cannot have a bucket policy that requires the server side encryption header the recommended method to require server side encryption for objects is to enable default encryption on the bucket itself instead the following sample policy can be used after replacing ${bucket} with the aws arn of your bucket instructions change the destination drop down to amazon s3 set bucket to the name of the amazon s3 bucket to store snapshots in set region to the name of the aws region that the s3 bucket is in set path to the path in the s3 bucket that the snapshots should be stored under if your cluster nodes are aws ec2 instances and you want the aws permissions to access the s3 bucket managed by an iam instance role, check the use iam instance role checkbox and leave the access key id and access key secret fields blank if you need to use iam credentials to access the s3 bucket then set access key id and access key secret to the iam user's api credentials if you need to upload a third party issued ca certificate used for trusting the authenticity of the snapshot storage endpoint you may upload them using add a ca certificate only one file can be uploaded however, it is possible to concatenate multiple certificates into one file supported formats are pem, cer, crt, ca, and key storing snapshots on azure blob storage requirements storing snapshots on azure blog storage requires an azure service principal and client secret to authenticate the storage account and service principal must be in the same subscription, tenant, and resource group required service principal permissions the service principal must have the storage account key operator service role role on the storage account the service principal must have the storage blob data contributor role on the storage container if you need to upload a third party issued ca certificate used for trusting the authenticity of the snapshot storage endpoint you may upload them using add a ca certificate only one file can be uploaded however, it is possible to concatenate multiple certificates into one file supported formats are pem, cer, crt, ca, and key instructions change the destination drop down to azure blob storage configure your azure settings field details bucket set to the name of the azure storage container where you will store snapshots path set to the path in the azure storage container where the snapshots should be stored subscription id set to the azure subscription id where your resources are tenant id set to the azure tenant id where your resources are client id set to the client id of the azure application that the service principal is a part of client secret set to the value of the client secret generated under the azure service principal cloud name set to the azure cloud where your resources are resource group name set to the azure resource group name where your resources are storage account id set to the name of the azure storage account where the storage container is storing snapshots on google cloud storage requirements storing snapshots on google cloud storage requires requires a google cloud service account to authenticate the service account should have the storage objectadmin role on the bucket if you need to upload a third party issued ca certificate used for trusting the authenticity of the snapshot storage endpoint you may upload them using add a ca certificate only one file can be uploaded however, it is possible to concatenate multiple certificates into one file supported formats are pem, cer, crt, ca, and key instructions change the destination drop down to google cloud storage set bucket to the name of the google storage bucket to store snapshots in set path to the path in the bucket that the snapshots should be stored under if your cluster nodes are google cloud vms and you want the aws permissions to access the google cloud storage bucket managed by an iam instance role, check the use iam instance role checkbox and leave the json file field blank if you need to use iam credentials to access the google cloud storage bucket then set json file to the json key for the service account storing snapshots on a host path requirements storing snapshots on a host path requires the host path storage destination should not be used for production environments they provide a security risk and the snapshots are not stored externally restoration will not be possible in the event of a total cluster loss the host path must be a dedicated directory do not use a partition used by a service like docker or kubernetes for ephemeral storage the host path directory specified must exist on every node that the turbine installer pods can be scheduled on to ensure snapshots work even if pod scheduling changes the host path directory must be read writable by the user\ group 1001 1001 host path cannot be used if your cluster requires pods to have resources, service account, affinity, node selectors, or tolerations defined this option creates a minio deployment in the namespace that turbine is installed under to handle passing the snapshot data to the host path swimlane does not support changing any of those settings for this deployment instructions change the destination drop down to host path set host path to the directory on the cluster nodes that the snapshots should be stored under storing snapshots on nfs requirements storing snapshots on nfs requires supports nfsv3 and nfsv4 host/ip authentication must be used as username and password authentication is not supported the nfs server must be configured to allow access from all the nodes in the cluster the nfs directory must be owned by the user\ group 1001 1001 the target directory needs to be read writable by the user\ group 1001 1001 all the nodes in the cluster must have the necessary nfs client packages installed to be able to communicate with the nfs server for example, the nfs common package is a common package used on ubuntu any firewalls must allow traffic between the nfs server and clients nfs cannot be used if your cluster requires pods to have resources, service account, affinity, node selectors, or tolerations defined this option creates a minio deployment in the namespace that turbine is installed under to handle passing the snapshot data to the host path and it is not currently supported to change any of those settings for this deployment instructions change the destination drop down to network file system (nfs) set server to the hostname or ip of the nfs server set path to the path on the nfs server that the snapshots should be stored under storing snapshots on other s3 compatible provider requirements storing snapshots on an s3 compatible provider requires an s3 compatible provider like min io https //min io/ the s3 compatible provider should be installed separately from the cluster nodes that turbine is installed on to ensure that snapshots are stored externally from the cluster so they can be retrieved in the event of a total cluster loss instructions change the destination drop down to other s3 compatible storage set bucket to the name of the s3 compatible bucket to store snapshots in set path to the path in the s3 compatible that the snapshots should be stored under set access key id and access key secret to the credentials required to access the storage provider set endpoint to the required value for your storage provider set region to the required value for your storage provider if you need to upload a third party issued ca certificate used for trusting the authenticity of the snapshot storage endpoint you may upload them using add a ca certificate only one file can be uploaded however, it is possible to concatenate multiple certificates into one file supported formats are pem, cer, crt, ca, and key restore from a partial (application) snapshot on the snapshots page, you can review a list of all of your application snapshots under the partial snapshots (application) menu click the circular icon to restore a certain snapshot to your turbine instance if you want to restore to the version of the snapshot, click restore from snapshot you are then prompted to enter the slug of the snapshot (confirming the slug name) enter turbine !> important! restoring to the version you've selected will remove any data since the snapshot was made in addition, during restoration, your turbine instance will not be available and you will not be able to use the turbine installer ui until the restore completes return to the main ui once your application status displays ready , then you know that both the ui and your turbine instance are back up and available again after a successful snapshot restore, the rabbitmq statefulset can enter a state where it cannot start and gets stuck in pending status to workaround this situation, delete the rabbitmq server statefulset and re create it with this command kubectl delete statefulset rabbitmq server the rabbitmq cluster operator will automatically re provision the rabbitmq server statefulset soon after deletion restore from a full (instance) snapshot in a non dr scenario instance snapshots can act as both instance level snapshots and as application level snapshots this section covers restoring the turbine application with an instance snapshot on the snapshots page, you can review a list of all of your instance snapshots under the full snapshots (instance) menu click the circular icon and select partial restore to restore a certain snapshot to your turbine instance you are then prompted to enter the slug of the snapshot (confirming the slug name) enter turbine !> important! restoring to the version you've selected removes any data since the snapshot was made in addition, during restoration, your turbine instance is not available and you will not be able to use the turbine installer until the restore completes return to the main ui once your application status displays ready , then you know that both the turbine installer ui and your turbine instance are back up and available again after a successful snapshot restore, the rabbitmq statefulset can enter a state where it cannot start and gets stuck in pending status to workaround this situation, delete the rabbitmq server statefulset and re create it with this command kubectl delete statefulset rabbitmq server the rabbitmq cluster operator will automatically re provision the rabbitmq server statefulset soon after deletion restore from a full (instance) snapshot in a dr scenario before continuing, you must ensure that your target cluster is ready for restoration this includes having velero installed as documented backup and restore on an existing cluster with snapshots docid\ nqvm65bghsoyrhs ydtkx this procedure also requires the kots kubectl plugin to be installed aws s3 other s3 nfs hostpath ensure that your snapshot volume mount is mounted and accessible on each individual node, then +> note if you are in an airgapped installation you also must provide the following arguments kotsadm namespace , kotsadm registry , registry password , registry username a process takes place after configuring the snapshot storage location that discovers which snapshots are available for restore after a few minutes, you can run the following to show the backups that are available select the backup you want to restore from the list and restore it via this command after a successful snapshot restore, the rabbitmq statefulset can enter a state where it cannot start and gets stuck in pending status to workaround this situation, delete the rabbitmq server statefulset and re create it with this command kubectl delete statefulset rabbitmq server the rabbitmq cluster operator will automatically re provision the rabbitmq server statefulset soon after deletion