Node and cluster maintenance on K8s Replicated (AKA) deployment

at some point, you'll need to carry out maintenance on a node (such as for a kernel upgrade, to apply a security patch, to upgrade the operating system, perform hardware maintenance, take a snapshot, etc ) which may require a single node or even a cluster wide shutdown or reboot it's critical that these events are handled gracefully in a kubernetes (k8s) replicated (aka) environment definition node a virtual or physical server can be a master or a worker cluster a group of interconnected nodes single node maintenance if additional nodes require maintenance then it is recommended to perform the steps below for each node one at a time or refer to the "multiple nodes maintenance" section for cluster wide maintenance connect (ssh) to the node which requires maintenance run the following shutdown script to gracefully evict or delete all pods running on the node and reschedule them onto another node $ sudo bash /opt/replicated/shutdown sh perform the necessary node maintenance restart the node (if necessary) once the node has restarted, connect to it and run the following start script $ sudo bash /opt/replicated/start sh multiple nodes maintenance it may become necessary to perform maintenance on all the nodes in a cluster together in such a scenario, follow the steps outlined below to create custom shutdown and startup scripts to gracefully stop k8s resources, perform the maintenance, and then resume k8s resources note this method can also be used if you need to completely shutdown all the nodes (for example to save development cost when using a cloud provider) step 1 set up the k8s shutdown and startup scripts ensure you're on a master node or another machine which has kubectl configured to access the cluster create a file , vars k8s sh, to store node names and cluster info in the example below, we use a 3 master and 3 worker nodes cluster note run the command kubectl get nodes to lookup the node names $ vi $home/vars k8s sh \## add the content below to var k8s sh n1=nodename master1 n2=nodename master2 n3=nodename master3 n4=nodename worker1 n5=nodename worker2 n6=nodename worker3 create the shutdown k8s sh script note the "cordoning nodes" section of the shutdown k8s sh is configured for a 6 node cluster please update the code to reflect the number of nodes in your cluster for example, if your cluster only has 3 nodes then remove any references to n4, n5, and n6 kubectl cordon $n1 & kubectl cordon $n2 & kubectl cordon $n3 & wait $ vi $home/shutdown k8s sh \## add the content below to the shutdown k8s sh script \#!/bin/bash \# e option causes the script to immediately exit on errors set e \# x option causes each shell command to be printed before it is executed set x source /vars k8s sh echo "stopping replicated application" replicated pod id=$(kubectl get pod l "app=replicated,tier=master" o name | sed 's/pod\\///') kubectl exec $replicated pod id c replicated replicatedctl app stop force attach || true echo "stopping replicated system" kubectl scale deploy replicated replicas=0 kubectl rollout status deploy replicated sleep 20 \# snapshotter won't exist if replicated wasn't fully started continue with shutdown if it doesn't exist echo 'delete replicated shared fs snapshotter deployment' kubectl delete deployments/replicated shared fs snapshotter || true echo "stopping ceph" kubectl n rook ceph system scale deploy rook ceph operator replicas=0 kubectl rollout status deployment extensions/rook ceph operator namespace rook ceph system echo "cordoning nodes" kubectl cordon $n1 & kubectl cordon $n2 & kubectl cordon $n3 & kubectl cordon $n4 & kubectl cordon $n5 & kubectl cordon $n6 & wait echo "successful shutdown of k8s" create the startup k8s sh script note the "waiting for node ready" section of the startup k8s sh is configured for a 6 node cluster please update the code to reflect the number of nodes in your cluster for example, if your cluster only has 3 nodes then remove any references to n4, n5, and n6 $ vi $home/startup k8s sh \## add the content below to the startup k8s sh script \#!/bin/bash \# e option causes the script to immediately exit on errors set e \# x option causes each shell command to be printed before it is executed set x source /vars k8s sh echo "uncordoning nodes" kubectl uncordon $n1 & kubectl uncordon $n2 & kubectl uncordon $n3 & kubectl uncordon $n4 & kubectl uncordon $n5 & kubectl uncordon $n6 echo "waiting for node ready" until (kubectl wait for=condition=ready node/$n1); do sleep 1; done until (kubectl wait for=condition=ready node/$n2); do sleep 1; done until (kubectl wait for=condition=ready node/$n3); do sleep 1; done until (kubectl wait for=condition=ready node/$n4); do sleep 1; done until (kubectl wait for=condition=ready node/$n5); do sleep 1; done until (kubectl wait for=condition=ready node/$n6); do sleep 1; done kubectl wait for=condition=ready node/$n2 & kubectl wait for=condition=ready node/$n3 \\ & kubectl wait for=condition=ready node/$n4 & kubectl wait for=condition=ready node/$n5 \\ & kubectl wait for=condition=ready node/$n6 & wait echo "waiting for kubernetes to become available for scaling ceph" set +e scale cmd="kubectl n rook ceph system scale deploy rook ceph operator replicas=1" attempts=0 $scale cmd result=$? until \[ $result eq 0 ] || \[ $attempts eq 60 ]; do $scale cmd result=$? attempts=$((attempts + 1)) echo "waiting for kubernetes to become available for scaling ceph ($attempts of 60)" sleep 2 done if \[ $attempts eq 60 ]; then echo "aborting unable to query kubernetes after $attempts attempts " echo exit 1 fi echo "waiting for ceph deployment to roll out" kubectl rollout status deployment extensions/rook ceph operator namespace rook ceph system echo "waiting for ceph to become available for querying" health cmd="kubectl exec n rook ceph i namespace rook ceph system $(kubectl get n rook ceph system pod l "app=rook ceph operator" o jsonpath='{ items\[0] metadata name}') ceph health" attempts=0 $health cmd result=$? until \[ $result eq 0 ] || \[ $attempts eq 60 ]; do $health cmd result=$? attempts=$((attempts + 1)) echo "waiting to query ceph ($attempts of 60)" sleep 2 done set e if \[ $attempts eq 60 ]; then echo "aborting unable to query ceph after $attempts attempts " echo exit 1 fi echo "waiting for ceph to be ready" attempts=0 ceph health=$($health cmd) until \[ "$ceph health" != "health ok" ] || \[ $attempts eq 60 ]; do ceph health=$($health cmd) attempts=$((attempts + 1)) echo $ceph health "(attempt $attempts of 60)" done if \[ $attempts eq 60 ]; then echo "aborting ceph health is not ok after $attempts attempts " echo exit 1 fi echo "ceph is ready" sleep 20 echo "starting replicated system" kubectl scale deploy replicated replicas=1 kubectl rollout status deploy replicated replicated pod id=$(kubectl get pod l "app=replicated,tier=master" o name | sed 's/pod\\///') until (kubectl exec $replicated pod id c replicated replicatedctl system status 2>/dev/null | grep q '"retraced" "ready"'); do sleep 1; done echo "starting replicated application" kubectl exec $replicated pod id c replicated replicatedctl app start attach echo "successful startup of k8s" step 2 shutdown the k8s cluster run the shutdown script to gracefully stop all k8s resources $ bash $home/shutdown k8s sh step 3 perform maintenance perform the necessary maintenance on all the nodes gracefully shutdown and power down all nodes step 4 start up the k8s cluster power up all the nodes run the start up script to resume k8s services $ bash $home/startup k8s sh step 5 confirmation use the following methods to confirm that the cluster and application is back online run kubectl get nodes run kubectl get pods a login to your replicated console login to swimlane application