Troubleshooting MongoDB Pod 2/3 Initialization Issue

Summary and overview

1.1 Purpose

This document describes the procedure followed to troubleshoot and resolve the MongoDB 2/3 initialization issue, where MongoDB member 2 became disconnected from the MongoDB replica set. The objective of this procedure is to restore MongoDB member 2 by triggering a fresh synchronization with the primary MongoDB member. The recovery process involves identifying the affected MongoDB pod and its associated Kubernetes node, isolating the node from new workload scheduling, verifying the associated Persistent Volume Claim (PVC), clearing the contents of the affected local persistent storage directory, and allowing the MongoDB pod to restart and synchronize with the primary member.

1.2 Scope

This runbook covers:

Identifying the MongoDB pod experiencing connectivity or initialization issues.

Checking the MongoDB pod's node placement.

Identifying and verifying the associated PVC.

Cordoning the Kubernetes node before performing storage operations.

Clearing the contents of the affected MongoDB local storage directory.

Uncordoning the node.

Monitoring MongoDB pod initialization and replica synchronization.

Prerequisites

Before starting the recovery procedure, ensure the following prerequisites are met:

Access to the Kubernetes cluster using kubectl.

Appropriate permissions to cordon and uncordon Kubernetes nodes.

Access to the affected Kubernetes worker node.

Permission to inspect and modify the local OpenEBS storage directory.

Knowledge of the MongoDB replica set configuration.

Confirmation of the affected MongoDB member and its corresponding pod.

Confirmation that the primary MongoDB member and the remaining replica set members are healthy.

Approval from the database/platform team before performing destructive storage operations.

Important Safety Considerations

⚠️ WARNING: This procedure deletes data from the local storage directory of the affected MongoDB member. Before executing the deletion command:

Confirm that MongoDB member 2 is the affected secondary and not the primary.

Verify that the remaining replica set members contain the required data.

Confirm that the primary is healthy and can serve as the synchronization source.

Verify that the PVC belongs exclusively to the affected MongoDB pod.

Ensure the MongoDB pod is stopped or otherwise safely detached from the volume before modifying its contents.

Confirm the recovery approach is compatible with the MongoDB operator and storage configuration.

Do not proceed if the replica set has lost quorum, the primary is unavailable, or the source data cannot be verified.

Recovery Procedure

Step 1: Identify the MongoDB Pod and Its Assigned Node

First, identify the MongoDB pods running in the Kubernetes cluster and determine which node is hosting the affected MongoDB member. Execute:

kubectl get pods -o wide | grep mongo

This command displays MongoDB-related pods along with their associated Kubernetes nodes. Review the output and identify the MongoDB member that is experiencing the initialization or connectivity issue.

Expected Outcome

The MongoDB pod list is displayed.

The affected MongoDB member (MongoDB 2) is identified.

The Kubernetes node hosting the affected pod is identified.

Record the following details:

Affected MongoDB pod name.

Kubernetes namespace.

Assigned Kubernetes node.

Current pod status.

Step 2: Cordon the Affected Kubernetes Node

Once the node hosting the affected MongoDB pod has been identified, cordon the node.

kubectl cordon {node-name}

Replace <Node-name> with the actual Kubernetes node name. Purpose Cordoning prevents new pods from being scheduled onto the selected node. This helps isolate the node during the recovery procedure. Verify the node status:

kubectl get nodes

The affected node should display the SchedulingDisabled status.

Expected Outcome

The affected node is successfully cordoned.

New workloads cannot be scheduled onto the node.

Existing workloads remain running unless separately stopped or evicted.

Note: Cordon does not stop existing pods or detach their storage volumes. Ensure the affected MongoDB process is no longer using the PVC before modifying its contents.

Step 3: Identify the MongoDB PVC

Next, identify the Persistent Volume Claim associated with the affected MongoDB pod.

kubectl get pvc | grep mongo

Review the output and identify the PVC associated with MongoDB member 2. To verify the PVC details, execute:

kubectl describe pvc {mongo-pvc-name}

Check the following information:

PVC name.

PVC status.

Bound Persistent Volume.

Storage class.

Namespace.

Associated workload or volume claim reference.

Ensure that the PVC belongs to the affected MongoDB member. Expected Outcome The correct PVC associated with MongoDB member 2 is identified.

Step 4: Locate the PVC Directory on the Affected Node

After identifying the PVC, access the Kubernetes node hosting the affected MongoDB storage. Navigate to the local OpenEBS storage directory:

ls -lah /var/openebs/local

This command lists the available local persistent volume directories. Identify the directory corresponding to the PVC associated with MongoDB member 2. Verify the directory name carefully against the PVC/PV mapping.

Expected Outcome The local storage directory corresponding to the affected MongoDB PVC is identified.

Step 5: Clear the Contents of the Affected MongoDB PVC

Once the correct PVC directory has been identified and all safety checks have been completed, clear the contents of the affected local storage directory.

rm -rf /var/openebs/local/{mongo-pvc-id}/*

Replace <mongo-pvc-id> with the actual PVC directory name.

Important Precaution Do not delete the entire PVC directory. Only the contents inside the identified PVC directory should be removed, as described in the recovery command. Before executing the command, verify the target directory one final time:

ls -lah /var/openebs/local/{mongo-pvc-id}

The deletion must be restricted to the confirmed affected MongoDB member's storage. Purpose Clearing the affected member's local database files allows the MongoDB member to perform a fresh synchronization from the primary replica, provided the MongoDB deployment or operator is configured to support this recovery behavior. Expected Outcome

The contents of the affected PVC directory are cleared.

The PVC directory itself remains present.

The affected member can initialize its local database storage again.

Step 6: Uncordon the Kubernetes Node

After completing the storage cleanup, return to the Kubernetes environment and uncordon the affected node.

kubectl uncordon {node-name}

Replace <Node-name> with the actual Kubernetes node name. Verify the node status:

kubectl get nodes

Expected Outcome

The node returns to the Ready state, assuming the node is otherwise healthy.

The SchedulingDisabled status is removed.

Kubernetes can schedule eligible workloads onto the node again.

Step 7: Monitor the MongoDB Pod Status

After uncordoning the node, monitor the affected MongoDB pod to observe its initialization and recovery progress.

kubectl get pods -o wide | grep mongo

kubectl get pods -w

kubectl describe pod {mongo-pod-name}

kubectl logs {mongo-pod-name} --follow

kubectl logs {mongo-pod-name} -c bootstrap --follow

Observe the pod status and look for successful initialization. Expected Behavior The affected MongoDB pod may take some time to initialize and synchronize with the primary member. The synchronization duration depends on factors such as:

Total database size.

Amount of data that needs to be synchronized.

Network connectivity between MongoDB members.

Available CPU, memory, and disk I/O.

MongoDB replication and initial sync activity.

Avoid restarting the pod repeatedly while synchronization is progressing.

Post-Recovery Validation

After the MongoDB pod starts, validate the MongoDB replica set health from the MongoDB.

rs.status()

Review the replica set member states. Important states to observe include:

PRIMARY – The primary replica set member.

SECONDARY – A healthy secondary member.

STARTUP2 – A member performing initial synchronization.

RECOVERING – A member recovering and not yet available as a normal secondary.

The affected member may initially appear in STARTUP2 while it performs the initial synchronization. This is an expected intermediate state during the recovery process. Continue monitoring until the affected member successfully transitions to SECONDARY.

Kubernetes-Level Validation

kubectl get pods -o wide | grep mongo

Confirm that:

The affected MongoDB pod is running.

The pod is not repeatedly restarting.

No persistent storage mounting errors are reported.

The pod remains stable.

MongoDB-Level Validation

rs.status()

Confirm that:

The primary member is healthy.

The affected MongoDB member has successfully joined the replica set.

The affected member transitions to SECONDARY.

No unexpected replica set errors are reported.

Troubleshooting and Monitoring Guidelines

Scenario 1: MongoDB Pod Remains in STARTUP2

If the MongoDB member remains in STARTUP2, it may still be performing initial synchronization. Recommended actions:

Monitor MongoDB pod logs.

Check the primary member's health.

Verify network connectivity between replica set members.

Check available storage capacity.

Review MongoDB replication and initial sync progress.

Do not assume that STARTUP2 indicates a failure immediately. Allow sufficient time for synchronization based on the database size and available resources.

Scenario 2: MongoDB Pod Enters CrashLoopBackOff

If the affected pod enters CrashLoopBackOff, investigate the pod logs and events.

kubectl describe pod {mongo-pod-name}

kubectl logs {mongo-pod-name} --previous

Check for:

Storage mounting issues.

Permission errors.

MongoDB configuration problems.

Container startup failures.

Insufficient resources.

Replica set configuration issues.

Scenario 3: MongoDB Member Does Not Rejoin the Replica Set

If the member fails to rejoin the replica set:

Verify primary member availability.

Check replica set configuration.

Validate inter-member network connectivity.

Review MongoDB logs for authentication or replication errors.

Check whether the initial sync is failing or restarting.

Escalate to the Swimlane Technical Support team if the member cannot complete synchronization.

Scenario 4: PVC or Storage Issues

If the MongoDB pod fails to mount its PVC:

Verify the PVC status.

Check the associated PV.

Inspect OpenEBS storage-related pod logs.

Verify that the local storage directory exists.

Check node-level disk availability and filesystem health.

Do not manually recreate or delete the PVC/PV without confirming the storage recovery procedure with the platform team.

Observed Result

Following the recovery procedure, we confirmed that the affected MongoDB member had entered the STARTUP2 state. This indicated that the member was in the initial synchronization phase and was expected to take additional time to synchronize with the primary MongoDB member. The synchronization duration was expected to depend on the overall database size and the amount of data being transferred.

Important Operational Notes

Always identify the exact MongoDB member before performing storage cleanup.

Never perform this procedure against the primary MongoDB member.

Ensure the remaining replica set members are healthy before initiating a fresh synchronization.

Cordon the node before beginning the recovery operation.

Remember that cordoning alone does not stop an existing MongoDB pod.

Confirm that the MongoDB process is no longer accessing the affected storage before deleting its contents.

Never delete the entire PVC directory when the intended operation is to clear only its contents.

Monitor the MongoDB pod and replica set status throughout the synchronization process.

Allow sufficient time for initial synchronization to complete.

Consider the recovery complete only after the affected MongoDB member has successfully transitioned to SECONDARY and replica set health has been validated.

Recovery Completion Criteria

The recovery procedure should be considered successfully completed only when all the following conditions are satisfied:

The affected MongoDB pod is running.

The pod is stable and not repeatedly restarting.

The MongoDB member has completed initial synchronization.

The member has transitioned from STARTUP2 to SECONDARY.

The primary MongoDB member remains healthy.

The replica set reports healthy membership.

No unexpected replication errors are observed.

The Kubernetes node is in Ready state, and scheduling is enabled.

Conclusion

The troubleshooting activity addressed the MongoDB 2/3 initialization issue by initiating a fresh synchronization of the disconnected MongoDB member. The recovery involved isolating the affected Kubernetes node, identifying the corresponding PVC and local OpenEBS storage directory, clearing the affected member's local storage contents, and allowing the MongoDB pod to initialize again. At the end of the troubleshooting session, MongoDB member 2 was observed in the STARTUP2 state, indicating that initial synchronization had started. The next operational step is to continue monitoring the MongoDB member until synchronization completes and the member transitions to the SECONDARY state. This runbook can be used as a reference for similar MongoDB replica recovery scenarios, subject to validation of the cluster topology, replica set health, and storage configuration before execution.