Node Maintenance on K8s Turbine Platform Installer (TPI) deployment
At some point, you'll need to carry out maintenance on a node (i.e. a kernel upgrade, apply a security patch, upgrade the operating system, perform hardware maintenance, take a snapshot) which may require a single node shutdown or reboot. It's critical that these events are handled gracefully in a Kubernetes (K8s) Turbine Platform Installer (TPI) environment. Applies to: TPI Standalone and High Availability (HA) deployment type.
PIOTU9-ERWY3456780Β Definitions:
- Node: a virtual machine or physical server
- Cluster: A group of interconnected nodes
NOTE: The steps below are supported for maintenance on one node at a time. Support for cluster-wide shutdown is currently NOT supported.
STEPS
- Pick a node to be taken down for maintenance.
kubectl get nodes
- Perform a health check of critical services:
# The namespace used is "default" in case of an embedded cluster deployment
export NS=<swimlane namespace>
# The MONGO_PREFIX is either "swimlane-sw-" for SPI or empty for TPI
export MONGO_PREFIX=
export MONGO_ADMIN_PASSWORD=<DEFINE_PASSWORD>
# The JSON response includes key of type list called "members", note down the "name" and "stateStr" for each MongoDB server
# There should be 3 members with "stateStr" showing one "PRIMARY" and two "SECONDARY"
kubectl -n $NS exec ${MONGO_PREFIX}mongo-0 -- mongosh -u Admin -p $MONGO_ADMIN_PASSWORD --authenticationDatabase admin --tls --tlsAllowInvalidCertificates --tlsAllowInvalidHostnames admin --eval="rs.status()"
# Check the replication lag specifically for the secondaries, it should show a maximum lag of a few seconds max
kubectl -n $NS exec ${MONGO_PREFIX}mongo-0 -- mongosh -u Admin -p $MONGO_ADMIN_PASSWORD --authenticationDatabase admin --tls --tlsAllowInvalidCertificates --tlsAllowInvalidHostnames admin --eval="rs.printSecondaryReplicationInfo()"
# Check the current state of the PostgreSQL cluster according to the replication manager Patroni
kubectl exec -n $NS postgresql-0 -- patronictl list
# Output should return a table with all PostgreSQL cluster members
# TL column should show the same value across all rows - this means that each server is on the correct timeline/epoch
# Lag column should show 0 across all rows and no value for the Leader - this means that there is no observed lag in replication, value otherwise shown in MB
# State should not show stopped in any row, it should be Streaming
export ELASTIC_PASSWORD=$(kubectl -n $NS get secret turbine-es-elastic-user -o jsonpath='{.data.elastic}' | base64 -d)
# Run an API call to the Elasticsearch service to check for the cluster's status
kubectl exec -n $NS turbine-es-default-0 -c elasticsearch -- curl -k "https://elastic:${ELASTIC_PASSWORD}@localhost:9200/_cluster/health?wait_for_status=green"
# This should return a JSON response
# The "status" key should say "green"
# The "relocating_shards" key should say 0
# https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html#cluster-health-api-response-body
# example JSON response: {"cluster_name":"turbine","status":"green","timed_out":false,"number_of_nodes":3,"number_of_data_nodes":3,"active_primary_shards":63,"active_shards":127,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}
If any of these checks do not return 'OK,' retry after some time and note any changes in the situation. If any of the services indicate that they are not progressing toward a healthy state, reach out to support.
- To minimize unwarranted disruption to MongoDB, check if the current node selected for maintenance is hosting the MongoDB pod with the PRIMARY role. The following command returns the hostname of the node where that pod is scheduled.
# Returns which Kubernetes node is running the MongoDB primary role
kubectl -o jsonpath='{.spec.nodeName}' get pod -n $NS $(kubectl -n $NS exec ${MONGO_PREFIX}mongo-0 -- mongosh -u Admin -p $MONGO_ADMIN_PASSWORD --authenticationDatabase admin --tls --tlsAllowInvalidCertificates --tlsAllowInvalidHostnames admin --eval="rs.isMaster().primary" | tail -n 1 | cut -d"." -f1)
If the node is the same as the one picked for maintenance, run this command to trigger a re-election and switch the PRIMARY role to a different node.
export MONGO_PRIMARY=$(kubectl -n $NS exec ${MONGO_PREFIX}mongo-0 -- mongosh -u Admin -p $MONGO_ADMIN_PASSWORD --authenticationDatabase admin --tls --tlsAllowInvalidCertificates --tlsAllowInvalidHostnames admin --eval="rs.isMaster().primary" | tail -n 1 | cut -d"." -f1)
kubectl -n $NS exec $MONGO_PRIMARY -- mongosh -u Admin -p $MONGO_ADMIN_PASSWORD --authenticationDatabase admin --tls --tlsAllowInvalidCertificates --tlsAllowInvalidHostnames admin --eval="rs.stepDown()"
Run this command again and confirm that it is not the same node that is currently picked to undergo maintenance:
kubectl -o jsonpath='{.spec.nodeName}' get pod -n $NS $(kubectl -n $NS exec ${MONGO_PREFIX}mongo-0 -- mongosh -u Admin -p $MONGO_ADMIN_PASSWORD --authenticationDatabase admin --tls --tlsAllowInvalidCertificates --tlsAllowInvalidHostnames admin --eval="rs.isMaster().primary" | tail -n 1 | cut -d"." -f1)
- As root, use the shutdown script to delete pods on the node
/opt/ekco/shutdown.sh
You may need to run this command to drain the node from all existing workloads if the prior command hasn't completed.
kubectl drain <NodeName> --ignore-daemonsets --delete-emptydir-data --grace-period=600 --force
- Ensure the drain node command completes/returns successfully. It should return error-free and the exit code is 0. If the command gets stuck or takes a long time to finish, then press control+c and continue to next steps.
- Perform maintenance on the affected node and/or reboot.
- If the node was not rebooted, run these commands on the node to bring it back into service:
/opt/ekco/startup.sh
kubectl uncordon <NodeName>
- Allow the workload to redistribute across all nodes in the cluster. Set the Swimlane namespace. For embedded clusters, the namespace is default.
export NS=<swimlane namespace>
kubectl rollout restart deployment -n $NS
- Then check if all services are operational (you can also run through the procedure outlined in Step 2):
# TPI Admin console and Swimlane components
kubectl get pods
# Remaining cluster pods
kubectl get pods --all-namespaces
- Log in to the TPI Admin console and Swimlane application to confirm they are back online. The status should display a green icon and a 'Ready' message.
- Repeat the steps on the next node that needs maintenance.