---
title: Node and cluster maintenance on K8s Replicated (AKA) deployment
slug: 10x-upgrade-guides/node-and-cluster-maintenance-on-k8s-replicated-aka-deployment
docTags: 
createdAt: 2025-12-15T07:36:07.804Z
---

At some point, you'll need to carry out maintenance on a node (such as for a kernel upgrade, to apply a security patch, to upgrade the operating system, perform hardware maintenance, take a snapshot, etc.) which may require a single node or even a cluster-wide shutdown or reboot. It's critical that these events are handled gracefully in a Kubernetes (K8s) Replicated (AKA) environment.

Definition:

- Node: a virtual or physical server 
  - Can be a master or a worker
- Cluster: A group of interconnected nodes

# Single node maintenance

If additional nodes require maintenance then it is recommended to perform the steps below for each node one at a time or refer to the "Multiple nodes maintenance" section for cluster-wide maintenance.

1. Connect (SSH) to the node which requires maintenance.
2. Run the following shutdown script to gracefully evict or delete all pods running on the node and reschedule them onto another node.

:::BlockQuote{indent="1"}
$ sudo bash /opt/replicated/shutdown.sh
:::

3. Perform the necessary node maintenance.
4. Restart the node (if necessary)
5. Once the node has restarted, connect to it and run the following start script.

:::BlockQuote{indent="1"}
$ sudo bash /opt/replicated/start.sh
:::

# Multiple nodes maintenance

It may become necessary to perform maintenance on all the nodes in a cluster together. In such a scenario, follow the steps outlined below to create custom shutdown and startup scripts to gracefully stop K8s resources, perform the maintenance, and then resume K8s resources.

**NOTE: **This method can also be used if you need to completely shutdown all the nodes (for example to save development cost when using a cloud provider).

### Step 1: Set up the K8s shutdown and startup scripts

1. Ensure you're on a master node or another machine which has kubectl configured to access the cluster.
2. Create a file , *vars-k8s.sh, *to store node names and cluster info.
   In the example below, we use a 3 master and 3 worker nodes cluster.

   **NOTE: **Run the command ***kubectl get nodes ***to lookup the node names.

:::BlockQuote{indent="1"}
$ vi $HOME/vars-k8s.sh

\## Add the content below to var-k8s.sh

N1=NodeName-master1
N2=NodeName-master2
N3=NodeName-master3
N4=NodeName-worker1
N5=NodeName-worker2
N6=NodeName-worker3
:::

3. Create the *shutdown-k8s.sh* script

   **NOTE**: The "Cordoning Nodes" section of the shutdown-k8s.sh is configured for a 6 node cluster. Please update the code to reflect the number of nodes in your cluster. For example, if your cluster only has 3 nodes then remove any references to N4, N5, and N6 : ***kubectl cordon $N1 & kubectl cordon $N2 & kubectl cordon $N3 & wait***

:::BlockQuote{indent="1"}
$ vi $HOME/shutdown-k8s.sh

\## Add the content below to the shutdown-k8s.sh script

\#!/bin/bash

\# -e option causes the script to immediately exit on errors
set -e

\# -x option causes each shell command to be printed before it is executed.
set -x

source ./vars-k8s.sh

echo "Stopping Replicated Application"
REPLICATED\_POD\_ID=$(kubectl get pod -l "app=replicated,tier=master" -o name | sed 's/pod\\///')
kubectl exec $REPLICATED\_POD\_ID -c replicated -- replicatedctl app stop --force --attach || true

echo "Stopping Replicated System"
kubectl scale deploy replicated --replicas=0
kubectl rollout status deploy replicated
sleep 20

\# Snapshotter won't exist if replicated wasn't fully started. Continue with shutdown if it doesn't exist. 
echo 'Delete replicated-shared-fs-snapshotter deployment'
kubectl delete deployments/replicated-shared-fs-snapshotter || true

echo "Stopping Ceph"
kubectl -n rook-ceph-system scale deploy rook-ceph-operator --replicas=0
kubectl rollout status deployment.extensions/rook-ceph-operator --namespace rook-ceph-system

echo "Cordoning Nodes"
kubectl cordon $N1 & kubectl cordon $N2 & kubectl cordon $N3 & kubectl cordon $N4 & kubectl cordon $N5 & kubectl cordon $N6 & wait

echo "Successful Shutdown of K8s"
:::

4. Create the *startup-k8s.sh* script

   **NOTE**: The "Waiting for Node Ready" section of the startup-k8s.sh is configured for a 6 node cluster. Please update the code to reflect the number of nodes in your cluster. For example, if your cluster only has 3 nodes then remove any references to N4, N5, and N6.

:::BlockQuote{indent="1"}
$ vi $HOME/startup-k8s.sh
\## Add the content below to the startup-k8s.sh script

\#!/bin/bash

\# -e option causes the script to immediately exit on errors
set -e

\# -x option causes each shell command to be printed before it is executed.
set -x

source ./vars-k8s.sh

echo "Uncordoning Nodes"
kubectl uncordon $N1 & kubectl uncordon $N2 & kubectl uncordon $N3 & kubectl uncordon $N4 & kubectl uncordon $N5 & kubectl uncordon $N6 

echo "Waiting for Node Ready"
until (kubectl wait --for=condition=Ready node/$N1); do sleep 1; done 
until (kubectl wait --for=condition=Ready node/$N2); do sleep 1; done 
until (kubectl wait --for=condition=Ready node/$N3); do sleep 1; done 
until (kubectl wait --for=condition=Ready node/$N4); do sleep 1; done 
until (kubectl wait --for=condition=Ready node/$N5); do sleep 1; done 
until (kubectl wait --for=condition=Ready node/$N6); do sleep 1; done 
kubectl wait --for=condition=Ready node/$N2 & kubectl wait --for=condition=Ready node/$N3 \\
  & kubectl wait --for=condition=Ready node/$N4 & kubectl wait --for=condition=Ready node/$N5 \\
  & kubectl wait --for=condition=Ready node/$N6 & wait

echo "Waiting for Kubernetes to become available for scaling Ceph"
set +e
SCALE\_CMD="kubectl -n rook-ceph-system scale deploy rook-ceph-operator --replicas=1"
ATTEMPTS=0
$SCALE\_CMD
RESULT=$?
until \[ $RESULT -eq 0 ] || \[ $ATTEMPTS -eq 60 ]; do
  $SCALE\_CMD
  RESULT=$?
  ATTEMPTS=$((ATTEMPTS + 1))
  echo "Waiting for Kubernetes to become available for scaling Ceph ($ATTEMPTS of 60)"
  sleep 2
done

if \[ $ATTEMPTS -eq 60 ]; then
    echo "Aborting... Unable to query Kubernetes after $ATTEMPTS attempts."
    echo exit 1
fi

echo "Waiting for Ceph deployment to roll out"
kubectl rollout status deployment.extensions/rook-ceph-operator --namespace rook-ceph-system

echo "Waiting for Ceph to become available for querying"
HEALTH\_CMD="kubectl exec -n rook-ceph -i --namespace rook-ceph-system $(kubectl get -n rook-ceph-system pod -l "app=rook-ceph-operator" -o jsonpath='\{.items\[0].metadata.name}') -- ceph health"
ATTEMPTS=0
$HEALTH\_CMD
RESULT=$?
until \[ $RESULT -eq 0 ] || \[ $ATTEMPTS -eq 60 ]; do
  $HEALTH\_CMD
  RESULT=$?
  ATTEMPTS=$((ATTEMPTS + 1))
  echo "Waiting to query Ceph ($ATTEMPTS of 60)"
  sleep 2
done
set -e

if \[ $ATTEMPTS -eq 60 ]; then
    echo "Aborting... Unable to query ceph after $ATTEMPTS attempts."
    echo exit 1
fi

echo "Waiting for Ceph to be Ready"
ATTEMPTS=0
CEPH\_HEALTH=$($HEALTH\_CMD)
until \[ "$CEPH\_HEALTH" != "HEALTH\_OK" ] || \[ $ATTEMPTS -eq 60 ]; do
    CEPH\_HEALTH=$($HEALTH\_CMD)
    ATTEMPTS=$((ATTEMPTS + 1))
    echo $CEPH\_HEALTH -- "(Attempt $ATTEMPTS of 60)"
done 

if \[ $ATTEMPTS -eq 60 ]; then
    echo "Aborting... Ceph Health is not OK after $ATTEMPTS attempts."
    echo exit 1
fi
echo "Ceph is Ready"
sleep 20

echo "Starting Replicated System"
kubectl scale deploy replicated --replicas=1
kubectl rollout status deploy replicated
REPLICATED\_POD\_ID=$(kubectl get pod -l "app=replicated,tier=master" -o name | sed 's/pod\\///')
until (kubectl exec $REPLICATED\_POD\_ID -c replicated -- replicatedctl system status 2>/dev/null | grep -q '"Retraced": "ready"'); do sleep 1; done 

echo "Starting Replicated Application"
kubectl exec $REPLICATED\_POD\_ID -c replicated -- replicatedctl app start --attach

echo "Successful Startup of K8s"
:::

### Step 2: Shutdown the K8s cluster

1. Run the shutdown script to gracefully stop all K8s resources.

:::BlockQuote{indent="1"}
$ bash $HOME/shutdown-k8s.sh
:::

### Step 3: Perform maintenance

1. Perform the necessary maintenance on all the nodes.
2. Gracefully shutdown and power down all nodes.

### Step 4: Start up the K8s cluster

1. Power up all the nodes.
2. Run the start up script to resume K8s services.

:::BlockQuote
$ bash $HOME/startup-k8s.sh
:::

### Step 5: Confirmation

Use the following methods to confirm that the cluster and application is back online.

1. Run ***kubectl get nodes***
2. Run ***kubectl get pods -A***
3. Login to your Replicated console
4. Login to Swimlane Application