Troubleshooting PostgreSQL Replication Lag Causing Pods Restarts

In high-availability (HA) Swimlane Turbine environments that use a Patroni-managed PostgreSQL cluster, an unhealthy standby replica can contribute to instability in application services.

A PostgreSQL standby replica may fall behind the cluster leader or lose its streaming replication connection. Its replication lag can then increase as it fails to replay the leader's Write-Ahead Log (WAL) records.

If application requests depend on the affected replica, these database errors or delays may cause application health checks to fail and trigger pod restarts. Confirm the replication condition and application logs before concluding that replication lag is the sole cause of the restarts.

Run patronictl list from a PostgreSQL pod that has the Patroni configuration and credentials available. The example below uses postgresql-0:

kubectl exec -it postgresql-0 -- patronictl list

If the PostgreSQL pods are in a non-default namespace, add the namespace option:

kubectl exec -n <namespace> -it postgresql-0 -- patronictl list

The following is an illustrative example in which postgresql-1 is lagging:

If the replica cannot recover through normal WAL streaming, reinitializing it may be required. Patroni reinitialization removes and rebuilds the selected replica's data directory, typically by cloning data from the leader using a base backup.

Use the patronictl list output to identify the current Leader and the lagging Replica. In the example above, postgresql-0 is the leader and postgresql-1 is the target.

kubectl exec -it postgresql-0 -- bash

If required, include the namespace with -n <namespace>. Run the Patroni command from a pod where the appropriate Patroni configuration and credentials are available.

Syntax:

patronictl reinit <cluster-name> <target-member>

For this example:

patronictl reinit postgresql postgresql-1

When prompted to confirm reinitialization of postgresql-1, verify the member name, then enter Y and press Enter.

Are you sure you want to reinitialize members postgresql-1? [y/N]:

The rebuild duration depends on database size, storage performance, and network throughput between the PostgreSQL pods. Monitor the target pod logs from another terminal:

kubectl logs -f postgresql-1 -c postgresql

If the pod is in a custom namespace, add -n <namespace> to the kubectl logs command. Wait for the reinitialization to finish before taking further action.

After the replica has finished rebuilding, run the following command:

patronictl list

Expected result (illustrative):

Confirm that both replicas are streaming and follow the leader's timeline. Replication lag should be zero or very low and stable under normal write activity; a small, transient non-zero value is expected while the cluster processes writes.

Check the relevant application pods and confirm that restart counts are no longer increasing and that readiness/liveness failures have stopped:

kubectl get pods -l 'for swimlane-tasks,turbine-api'

If the command returns no pods, verify the labels used by your deployment with 'kubectl get pods --show-labels' and adjust the selector accordingly. Also review application and PostgreSQL logs for any remaining database connection or query errors.

Before reinitialization, verify that the target replica has sufficient free space. For example, run df -h inside the target pod or inspect the mounted data volume. If the volume is full, resolve the storage issue or expand the PVC before retrying.

Do not reinitialize multiple replicas simultaneously. Maintain cluster redundancy and confirm that the remaining members are healthy before proceeding.

If the same replica repeatedly falls behind after reinitialization, investigate the underlying cause rather than rebuilding it repeatedly. Areas to review include: