Node and cluster maintenance on K8s Replicated (AKA) deployment

At some point, you'll need to carry out maintenance on a node (such as for a kernel upgrade, to apply a security patch, to upgrade the operating system, perform hardware maintenance, take a snapshot, etc.) which may require a single node or even a cluster-wide shutdown or reboot. It's critical that these events are handled gracefully in a Kubernetes (K8s) Replicated (AKA) environment.

Definition:

Node: a virtual or physical server

Can be a master or a worker

Cluster: A group of interconnected nodes

Single node maintenance

If additional nodes require maintenance then it is recommended to perform the steps below for each node one at a time or refer to the "Multiple nodes maintenance" section for cluster-wide maintenance.

Connect (SSH) to the node which requires maintenance.

Run the following shutdown script to gracefully evict or delete all pods running on the node and reschedule them onto another node.

$ sudo bash /opt/replicated/shutdown.sh

Perform the necessary node maintenance.

Restart the node (if necessary)

Once the node has restarted, connect to it and run the following start script.

$ sudo bash /opt/replicated/start.sh

Multiple nodes maintenance

It may become necessary to perform maintenance on all the nodes in a cluster together. In such a scenario, follow the steps outlined below to create custom shutdown and startup scripts to gracefully stop K8s resources, perform the maintenance, and then resume K8s resources. NOTE: This method can also be used if you need to completely shutdown all the nodes (for example to save development cost when using a cloud provider).

Step 1: Set up the K8s shutdown and startup scripts

Ensure you're on a master node or another machine which has kubectl configured to access the cluster.

Create a file , vars-k8s.sh, to store node names and cluster info. In the example below, we use a 3 master and 3 worker nodes cluster. NOTE: Run the command kubectl get nodes to lookup the node names.

$ vi $HOME/vars-k8s.sh ## Add the content below to var-k8s.sh N1=NodeName-master1 N2=NodeName-master2 N3=NodeName-master3 N4=NodeName-worker1 N5=NodeName-worker2 N6=NodeName-worker3

Create the shutdown-k8s.sh script NOTE: The "Cordoning Nodes" section of the shutdown-k8s.sh is configured for a 6 node cluster. Please update the code to reflect the number of nodes in your cluster. For example, if your cluster only has 3 nodes then remove any references to N4, N5, and N6 : kubectl cordon $N1 & kubectl cordon $N2 & kubectl cordon $N3 & wait

$ vi $HOME/shutdown-k8s.sh ## Add the content below to the shutdown-k8s.sh script #!/bin/bash # -e option causes the script to immediately exit on errors set -e # -x option causes each shell command to be printed before it is executed. set -x source ./vars-k8s.sh echo "Stopping Replicated Application" REPLICATED_POD_ID=$(kubectl get pod -l "app=replicated,tier=master" -o name | sed 's/pod\///') kubectl exec $REPLICATED_POD_ID -c replicated -- replicatedctl app stop --force --attach || true echo "Stopping Replicated System" kubectl scale deploy replicated --replicas=0 kubectl rollout status deploy replicated sleep 20 # Snapshotter won't exist if replicated wasn't fully started. Continue with shutdown if it doesn't exist. echo 'Delete replicated-shared-fs-snapshotter deployment' kubectl delete deployments/replicated-shared-fs-snapshotter || true echo "Stopping Ceph" kubectl -n rook-ceph-system scale deploy rook-ceph-operator --replicas=0 kubectl rollout status deployment.extensions/rook-ceph-operator --namespace rook-ceph-system echo "Cordoning Nodes" kubectl cordon $N1 & kubectl cordon $N2 & kubectl cordon $N3 & kubectl cordon $N4 & kubectl cordon $N5 & kubectl cordon $N6 & wait echo "Successful Shutdown of K8s"

Create the startup-k8s.sh script NOTE: The "Waiting for Node Ready" section of the startup-k8s.sh is configured for a 6 node cluster. Please update the code to reflect the number of nodes in your cluster. For example, if your cluster only has 3 nodes then remove any references to N4, N5, and N6.

$ vi $HOME/startup-k8s.sh ## Add the content below to the startup-k8s.sh script #!/bin/bash # -e option causes the script to immediately exit on errors set -e # -x option causes each shell command to be printed before it is executed. set -x source ./vars-k8s.sh echo "Uncordoning Nodes" kubectl uncordon $N1 & kubectl uncordon $N2 & kubectl uncordon $N3 & kubectl uncordon $N4 & kubectl uncordon $N5 & kubectl uncordon $N6 echo "Waiting for Node Ready" until (kubectl wait --for=condition=Ready node/$N1); do sleep 1; done until (kubectl wait --for=condition=Ready node/$N2); do sleep 1; done until (kubectl wait --for=condition=Ready node/$N3); do sleep 1; done until (kubectl wait --for=condition=Ready node/$N4); do sleep 1; done until (kubectl wait --for=condition=Ready node/$N5); do sleep 1; done until (kubectl wait --for=condition=Ready node/$N6); do sleep 1; done kubectl wait --for=condition=Ready node/$N2 & kubectl wait --for=condition=Ready node/$N3 \ & kubectl wait --for=condition=Ready node/$N4 & kubectl wait --for=condition=Ready node/$N5 \ & kubectl wait --for=condition=Ready node/$N6 & wait echo "Waiting for Kubernetes to become available for scaling Ceph" set +e SCALE_CMD="kubectl -n rook-ceph-system scale deploy rook-ceph-operator --replicas=1" ATTEMPTS=0 $SCALE_CMD RESULT=$? until [ $RESULT -eq 0 ] || [ $ATTEMPTS -eq 60 ]; do $SCALE_CMD RESULT=$? ATTEMPTS=$((ATTEMPTS + 1)) echo "Waiting for Kubernetes to become available for scaling Ceph ($ATTEMPTS of 60)" sleep 2 done if [ $ATTEMPTS -eq 60 ]; then echo "Aborting... Unable to query Kubernetes after $ATTEMPTS attempts." echo exit 1 fi echo "Waiting for Ceph deployment to roll out" kubectl rollout status deployment.extensions/rook-ceph-operator --namespace rook-ceph-system echo "Waiting for Ceph to become available for querying" HEALTH_CMD="kubectl exec -n rook-ceph -i --namespace rook-ceph-system $(kubectl get -n rook-ceph-system pod -l "app=rook-ceph-operator" -o jsonpath='{.items[0].metadata.name}') -- ceph health" ATTEMPTS=0 $HEALTH_CMD RESULT=$? until [ $RESULT -eq 0 ] || [ $ATTEMPTS -eq 60 ]; do $HEALTH_CMD RESULT=$? ATTEMPTS=$((ATTEMPTS + 1)) echo "Waiting to query Ceph ($ATTEMPTS of 60)" sleep 2 done set -e if [ $ATTEMPTS -eq 60 ]; then echo "Aborting... Unable to query ceph after $ATTEMPTS attempts." echo exit 1 fi echo "Waiting for Ceph to be Ready" ATTEMPTS=0 CEPH_HEALTH=$($HEALTH_CMD) until [ "$CEPH_HEALTH" != "HEALTH_OK" ] || [ $ATTEMPTS -eq 60 ]; do CEPH_HEALTH=$($HEALTH_CMD) ATTEMPTS=$((ATTEMPTS + 1)) echo $CEPH_HEALTH -- "(Attempt $ATTEMPTS of 60)" done if [ $ATTEMPTS -eq 60 ]; then echo "Aborting... Ceph Health is not OK after $ATTEMPTS attempts." echo exit 1 fi echo "Ceph is Ready" sleep 20 echo "Starting Replicated System" kubectl scale deploy replicated --replicas=1 kubectl rollout status deploy replicated REPLICATED_POD_ID=$(kubectl get pod -l "app=replicated,tier=master" -o name | sed 's/pod\///') until (kubectl exec $REPLICATED_POD_ID -c replicated -- replicatedctl system status 2>/dev/null | grep -q '"Retraced": "ready"'); do sleep 1; done echo "Starting Replicated Application" kubectl exec $REPLICATED_POD_ID -c replicated -- replicatedctl app start --attach echo "Successful Startup of K8s"

Step 2: Shutdown the K8s cluster

Run the shutdown script to gracefully stop all K8s resources.

$ bash $HOME/shutdown-k8s.sh

Step 3: Perform maintenance

Perform the necessary maintenance on all the nodes.

Gracefully shutdown and power down all nodes.

Step 4: Start up the K8s cluster

Power up all the nodes.

Run the start up script to resume K8s services.

$ bash $HOME/startup-k8s.sh

Step 5: Confirmation

Use the following methods to confirm that the cluster and application is back online.

Run kubectl get nodes

Run kubectl get pods -A