--- title: Troubleshooting PostgreSQL Replication Lag Causing Pods Restarts slug: troubleshotting/troubleshooting-postgresql-replication-lag-causing-pods-restarts createdAt: 2026-11-13T11:32:05.041Z --- ## Overview and Symptoms In high-availability (HA) Swimlane Turbine environments that use a Patroni-managed PostgreSQL cluster, an unhealthy standby replica can contribute to instability in application services. ## Common symptoms include: - turbine-api or swimlane-tasks pods intermittently fail readiness or liveness checks, restart, or show elevated restart counts. - Tasks and playbook actions intermittently fail or time out, particularly when read queries are routed to an out-of-date replica. - Database query latency increases across backend services. ## Root Cause A PostgreSQL standby replica may fall behind the cluster leader or lose its streaming replication connection. Its replication lag can then increase as it fails to replay the leader's Write-Ahead Log (WAL) records. ### In a severely degraded condition: - The replica may be unable to retrieve or replay the required WAL segments. - The replica may report a different timeline from the current leader, which can indicate that it is not following the leader's current history. - The replica may no longer report a streaming state. Read traffic routed to it may return stale data or encounter delays and connection timeouts. If application requests depend on the affected replica, these database errors or delays may cause application health checks to fail and trigger pod restarts. Confirm the replication condition and application logs before concluding that replication lag is the sole cause of the restarts. ## Step 1 — Diagnose PostgreSQL Cluster Status Run patronictl list from a PostgreSQL pod that has the Patroni configuration and credentials available. The example below uses postgresql-0: `kubectl exec -it postgresql-0 -- patronictl list` If the PostgreSQL pods are in a non-default namespace, add the namespace option: `kubectl exec -n -it postgresql-0 -- patronictl list` ### Example: Unhealthy Replica The following is an illustrative example in which postgresql-1 is lagging: | Member | Host | Role | State | TL | Lag in MB | | ------------ | -------- | ------- | --------- | -- | --------- | | postgresql-0 | 10.x.x.x | Leader | running | 80 | — | | postgresql-1 | 10.x.x.x | Replica | running | 55 | 368 | | postgresql-2 | 10.x.x.x | Replica | streaming | 80 | 0 | ## Indicators to Investigate - Role: the affected member should be identified as a Replica. - State: a healthy replica normally reports streaming. A replica reporting running rather than streaming warrants investigation; interpret the state together with the other fields and Patroni logs. - Timeline (TL): compare the replica timeline with the current leader timeline. A mismatch can indicate timeline divergence and should be investigated. - Lag in MB: a large or continuously increasing lag value indicates that the replica is not keeping up with the leader. ## Step 2 — Reinitialize the Lagging Replica If the replica cannot recover through normal WAL streaming, reinitializing it may be required. Patroni reinitialization removes and rebuilds the selected replica's data directory, typically by cloning data from the leader using a base backup. | IMPORTANT: Reinitialization is destructive to the target replica's local database data. Confirm the correct member before proceeding. Do not run this command against the leader, and do not reinitialize multiple replicas at the same time. Ensure the leader and at least one other healthy replica are available where possible. | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ## 2.1 Identify the leader and target member Use the patronictl list output to identify the current Leader and the lagging Replica. In the example above, postgresql-0 is the leader and postgresql-1 is the target. ## 2.2 Open a shell in the leader pod `kubectl exec -it postgresql-0 -- bash` If required, include the namespace with -n \. Run the Patroni command from a pod where the appropriate Patroni configuration and credentials are available. ## 2.3 Reinitialize only the affected replica **Syntax:** `patronictl reinit ` **For this example:** `patronictl reinit postgresql postgresql-1` When prompted to confirm reinitialization of postgresql-1, verify the member name, then enter Y and press Enter. **Are you sure you want to reinitialize members postgresql-1? \[y/N]:** ## 2.4 Monitor the base backup The rebuild duration depends on database size, storage performance, and network throughput between the PostgreSQL pods. Monitor the target pod logs from another terminal: `kubectl logs -f postgresql-1 -c postgresql` If the pod is in a custom namespace, add -n \ to the kubectl logs command. Wait for the reinitialization to finish before taking further action. ## Step 3 — Verify Replication and Application Stability ## 3.1 Verify Patroni cluster health After the replica has finished rebuilding, run the following command: patronictl list Expected result (illustrative): | Member | Host | Role | State | TL | Lag in MB | | ------------ | -------- | ------- | --------- | -- | --------- | | postgresql-0 | 10.x.x.x | Leader | running | 80 | — | | postgresql-1 | 10.x.x.x | Replica | streaming | 80 | 0 | | postgresql-2 | 10.x.x.x | Replica | streaming | 80 | 0 | Confirm that both replicas are streaming and follow the leader's timeline. Replication lag should be zero or very low and stable under normal write activity; a small, transient non-zero value is expected while the cluster processes writes. ## 3.2 Verify application pod stability Check the relevant application pods and confirm that restart counts are no longer increasing and that readiness/liveness failures have stopped: **kubectl get pods -l 'for swimlane-tasks,turbine-api'** If the command returns no pods, verify the labels used by your deployment with 'kubectl get pods --show-labels' and adjust the selector accordingly. Also review application and PostgreSQL logs for any remaining database connection or query errors. ## Notes and Best Practices ## Check available disk space Before reinitialization, verify that the target replica has sufficient free space. For example, **run df -h** inside the target pod or inspect the mounted data volume. If the volume is full, resolve the storage issue or expand the PVC before retrying. ## Reinitialize one replica at a time Do not reinitialize multiple replicas simultaneously. Maintain cluster redundancy and confirm that the remaining members are healthy before proceeding. ## Investigate recurring replication lag If the same replica repeatedly falls behind after reinitialization, investigate the underlying cause rather than rebuilding it repeatedly. **Areas to review include:** - Network latency, packet loss, or bandwidth saturation between Kubernetes nodes. - Storage latency and available capacity on the PostgreSQL volumes. - PostgreSQL WAL retention and checkpoint-related settings, including max\_wal\_size and wal\_keep\_size, in the context of the deployed PostgreSQL and Patroni configuration. - PostgreSQL and Patroni logs for replication disconnects, WAL availability errors, or repeated recovery failures. ## Completion Checklist - The current leader has been identified and is healthy. - The correct lagging replica has been selected for reinitialization. - The target replica has sufficient storage capacity. - Reinitialization and base backup have completed successfully. - All replicas report streaming and the expected timeline. - Replication lag is zero or low and stable. - turbine-api and swimlane-tasks pods are Ready, with no continuing increase in restarts. - Application and PostgreSQL logs show no continuing replication-related errors.