--- title: Node and cluster maintenance on K8s Helm deployment slug: 10x-upgrade-guides/node-and-cluster-maintenance-on-k8s-helm-deployment docTags: createdAt: 2025-12-15T07:39:01.446Z --- At some point, you'll need to carry out maintenance on a node (such as for a kernel upgrade, to apply a security patch, to upgrade the operating system, perform hardware maintenance, take a vm snapshot, etc.) which may require a single node or even a cluster-wide shutdown or reboot. It's critical that these events are handled gracefully in a Helm deployment of Swimlane on Kubernetes (K8s). Definition: - Node: a virtual or physical server - Can be a master or a worker - Cluster: A group of interconnected nodes # Single node maintenance In brief, maintenance on a single node will include: - Gracefully shutdown K8s components - Perform node maintenance - Restart the node (if necessary) - Start K8s components again **NOTE:** If additional nodes require maintenance then it is recommended to perform the steps below for each node one at a time or refer to the "Multiple nodes maintenance" section for cluster-wide maintenance. 1. Connect (SSH) to the node which requires maintenance. 2. Drain the node. The *kubectl* drain operation reschedules all the pods from the node onto another node. To do so, it will 1) tag the node unschedulable in order to prevent new pods from being scheduled (equal to ***kubectl cordon \***) and 2) evicts or deletes all pods running on the node. For pods with a replica set (like mongo), the pod will be replaced by a new pod that will be scheduled to a new node. Draining a node won’t cause any downtime as long as Mongo’s resource utilization isn’t so high that it can’t run alongside another mongo pod during the maintenance window. **TIP**: Use ***kubectl get nodes -owide*** to lookup the node name. :::BlockQuote{indent="1"} $ kubectl drain \ --ignore-daemonsets --delete-local-data ::: :::Paragraph{indent="1"} **NOTE**: The flags 'ignore-daemonsets' and 'delete-local-data' deletes data that is not persisted to an actual disk somewhere such as emptyDir. The command above may also need '--force' if there exists pods not managed by ReplicaSet, Job, DaemonSet or StatefulSet. ::: 3. Perform the necessary node maintenance (restart if needed) 4. Make the node schedulable again :::Paragraph{indent="1"} Run the following command to uncordon the node which tags the node to be schedulable again. ::: :::BlockQuote{indent="1"} $ kubectl uncordon \ ::: 5. Repeat the process above for any additional nodes that need maintenance work. # Multiple nodes maintenance It may become necessary to perform maintenance on all the nodes in a cluster together. In such a scenario, follow the steps outlined below to gracefully stop K8s resources, perform the maintenance, and then restart K8s resources. **NOTE: **This method can also be used if you need to completely shutdown all the nodes (for example to save development cost when using a cloud provider). ### Step 1: Shutdown 1. Ensure you're on a master node or another machine which has kubectl configured to access the cluster. 2. Stop the Swimlane Application **TIP: ** \ : Run the command ***helm ls ***to lookup the Helm release name. \ : will be *swimlane/swimlane* if working with the chart remotely otherwise *swimlane* :::BlockQuote{indent="1"} $ helm upgrade \ \ -f \ --set api.replicaCount=0,tasks.replicaCount=0,web.replicaCount=0,mongodb-replicaset.replicas=0 \# Example using a chart remotely helm upgrade swimlane swimlane/swimlane -f values.yaml --set api.replicaCount=0,tasks.replicaCount=0,web.replicaCount=0,mongodb-replicaset.replicas=0 \# Example using a chart locally helm upgrade swimlane swimlane -f swimlane/values.yaml --set api.replicaCount=0,tasks.replicaCount=0,web.replicaCount=0,mongodb-replicaset.replicas=0 ::: 3. Make nodes unschedulable The ***kubectl cordon \*** command will tag the nodes unschedulable and prevent any new pods from being scheduled on other nodes. **NOTE**: The commands below is for a 6 node cluster. Please update the code to reflect the number of nodes in your cluster. :::BlockQuote{indent="1"} $ kubectl cordon \ $ kubectl cordon \ $ kubectl cordon \ $ kubectl cordon \ $ kubectl cordon \ $ kubectl cordon \ ::: 1. Drain the nodes Gracefully terminate all pods running on the node **NOTE**: The commands below is for a 6 node cluster. Please update the code to reflect the number of nodes in your cluster. :::BlockQuote{indent="1"} $ kubectl drain \ --ignore-daemonsets --delete-local-data $ kubectl drain \ --ignore-daemonsets --delete-local-data $ kubectl drain \ --ignore-daemonsets --delete-local-data $ kubectl drain \ --ignore-daemonsets --delete-local-data $ kubectl drain \ --ignore-daemonsets --delete-local-data $ kubectl drain \ --ignore-daemonsets --delete-local-data ::: ### Step 2: Perform maintenance 1. Perform the necessary maintenance on all the nodes 2. Gracefully shutdown and power down all nodes ### Step 3: Startup 1. Power up all the nodes 2. Ensure you're on a master node or another machine which has kubectl configured to access the cluster 3. Make nodes schedulable again The ***kubectl uncordon \*** command will tag the nodes schedulable and ready to take on new pods :::BlockQuote{indent="1"} $ kubectl uncordon \ $ kubectl uncordon \ $ kubectl uncordon \ $ kubectl uncordon \ $ kubectl uncordon \ $ kubectl uncordon \ ::: 1. Start the Swimlane Application **TIP: ** \ : Run the command ***helm ls ***to lookup the Helm release name. \ : will be *swimlane/swimlane* if working with the chart remotely otherwise *swimlane* :::BlockQuote{indent="1"} $ helm upgrade \ \ -f \ \# Example using a chart remotely helm upgrade swimlane swimlane/swimlane -f values.yaml \# Example using a chart locally helm upgrade swimlane swimlane -f swimlane/values.yaml ::: ### Step 4: Confirmation Use the following methods to confirm that the cluster and application is back online. 1. Run ***kubectl get nodes*** 2. Run ***kubectl get pods -A*** 3. Login to Swimlane Application