Troubleshooting Playbook Failures Due to Internal CA Mismatch.

Overview & Scope

Purpose	Diagnose and recover Swimlane/Turbine environments where playbooks stop starting or fail after an application upgrade because RabbitMQ and its clients trust certificates issued by different internal certificate authorities (CAs).
Scope	Applies when playbooks fail broadly across tenants post-upgrade, `turbine-api` or `turbine-content-library` cannot connect to RabbitMQ over AMQPS, and logs display `SELF_SIGNED_CERT_IN_CHAIN` errors.
Namespace Note	Examples use the `default` namespace. Confirm your specific namespace before running commands (newer deployments may use separate `core` and `rabbitmq` namespaces).

Symptom & Impact Analysis

Customer-Facing Symptoms

Execution Failures: Manually triggered playbooks do not start; scheduled playbooks stop creating runs. Queue Stagnation: Playbook runs remain queued or pending, or fail prior to executing actions. Service Failures: Content-library operations fail, though the main UI remains accessible. Misleading K8s Status: Kubernetes pods show Running and Ready despite broken underlying AMQP connections. Business & Technical Impact `turbine-api` relies on RabbitMQ for playbook and platform messaging. If the AMQPS connection fails, API endpoints stay responsive, but asynchronous workflows fail completely. Existing Turbine Engine or agent processes may temporarily process previously queued items over older open connections, masking the full extent of the outage.

Diagnostic Findings & Root Cause Analysis

Key Observations

Turbine API AMQP Connection Failure: All `turbine-api` replicas repeatedly log certificate verification failures every 5 seconds.

# Log location: turbine/logs/turbine-api-*/turbine-api.log
Failed to connect to AMQP
Error: self-signed certificate in certificate chain
code: SELF_SIGNED_CERT_IN_CHAIN

Content Library Errors: `turbine-content-library` replicas log identical `SELF_SIGNED_CERT_IN_CHAIN` errors, ruling out a single-pod scope. Root CA Re-issuance: `swimlane-selfsigned-ca-cert` was updated to revision 2 during the upgrade, generating a brand new internal root CA. Partial Leaf Certificate Renewal: Only `turbine-api-cert` and `turbine-content-library-cert` were renewed under the new CA. RabbitMQ and most other components kept older leaf certificates.

Root Cause Summary

During the application upgrade, `cert-manager` updated the `swimlane-selfsigned-ca-cert` resource, generating revision 2 of the root CA. However, leaf certificate renewals were not coordinated across all services: API and Content Library certificates were renewed under New Root CA. RabbitMQ and engine/agent workers remained on leaf certificates tied to Old Root CA. When renewed Turbine clients loaded the new CA bundle, they failed to validate RabbitMQ's older server certificate, stopping TLS handshakes before AMQP communication began.

Step-by-Step Diagnostic Procedures

1. Inspect Pod & Log Status

# Check pod readiness
kubectl get pods -n default | egrep 'turbine-(api|engine|agent|content-library)|rabbitmq'

# Search turbine-api logs for AMQP TLS errors
kubectl logs -n default -l app=turbine-api \
  --all-containers --since=2h --prefix | \
  egrep -i 'AMQP|SELF_SIGNED|certificate'

# Search turbine-content-library logs for AMQP TLS errors
kubectl logs -n default -l app=turbine-content-library \
  --all-containers --since=2h --prefix | \
  egrep -i 'AMQP|SELF_SIGNED|certificate'

2. Compare Certificate Revisions & Issuers

# List revision numbers and issuance dates across all certificates
kubectl get certificate -n default \
  -o custom-columns='NAME:.metadata.name,REVISION:.status.revision,NOT-BEFORE:.status.notBefore,NOT-AFTER:.status.notAfter,READY:.status.conditions[-1].status'

# Inspect specific certificate details
kubectl get certificate -n default \
  swimlane-selfsigned-ca-cert rabbitmq turbine-api-cert turbine-content-library-cert \
  -o yaml

# Confirm issuer configuration
kubectl get issuer -n default swimlane-ca-issuer -o yaml

3. Verify RabbitMQ Cluster Health

# Verify RabbitMQ status and cluster status
kubectl get pods -n default -l app.kubernetes.io/name=rabbitmq
kubectl exec -n default rabbitmq-server-0 -- rabbitmqctl cluster_status

Resolution & Remediation Plan

Safety Requirements & Pre-checks

Schedule a maintenance window and take a full KOTS/Velero backup before changing certificates. Export existing CA and certificate configurations for safe rollback:

kubectl get secret -n default root-ca-cert -o yaml > root-ca-cert-backup.yaml
kubectl get certificate,issuer -n default -o yaml > certificate-resources-backup.yaml

⚠️ PROHIBITED ACTIONS Do NOT delete `root-ca-cert` (this will regenerate another CA and compound the mismatch). Do NOT renew only RabbitMQ or only `turbine-api` in isolation. Do NOT disable TLS verification as a permanent workaround. Do NOT restart all RabbitMQ replicas simultaneously.

Execution: Complete CA Rotation

Renew all leaf certificates issued by `swimlane-ca-issuer` using `cmctl` and restart consuming workloads in a controlled sequence.

# Step 1: List affected certificates issued by swimlane-ca-issuer
kubectl get certificate -n default -o json | \
  jq -r '.items[] | select(.spec.issuerRef.name == "swimlane-ca-issuer") | [.metadata.name, .status.revision, .status.notBefore] | @tsv'

# Step 2: Renew all filtered leaf certificates
kubectl get certificate -n default -o json | \
  jq -r '.items[] | select(.spec.issuerRef.name == "swimlane-ca-issuer") | .metadata.name' | \
  xargs -n 20 cmctl renew -n default

# Step 3: Wait for certificate issuance readiness
kubectl wait -n default --for=condition=Ready certificate --all --timeout=15m

# Step 4: Verify certificate issuance against root CA
kubectl get certificate -n default \
  -o custom-columns='NAME:.metadata.name,REVISION:.status.revision,NOT-BEFORE:.status.notBefore,READY:.status.conditions[-1].status'

# Step 5: Check certificate chain matching
kubectl get secret -n default rabbitmq-cert -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject -issuer -dates
kubectl get secret -n default turbine-api-cert -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject -issuer -dates

Projected Secret updates do not guarantee that long-running processes reload certificates. Restart the affected workloads after all renewals complete.

Restart Workloads

Redeploy the application version from the KOTS Admin Console to restart workloads cleanly with updated secrets. If executing manual restarts, perform a rolling restart of RabbitMQ replicas first, followed by dependent Turbine services.

# Perform rolling restarts for deployments and statefulsets in sequence
kubectl rollout restart statefulset -n default -l app.kubernetes.io/name=rabbitmq
kubectl rollout status statefulset -n default -l app.kubernetes.io/name=rabbitmq --timeout=10m

kubectl rollout restart deployment -n default -l 'app in (turbine-api,turbine-content-library,turbine-engine)'
kubectl rollout status deployment -n default -l 'app in (turbine-api,turbine-content-library,turbine-engine)' --timeout=10m

# General command to trigger rolling restarts across all deployments and statefulsets
kubectl rollout restart deploy,sts -n default

Validation Checklist

Validation Area	Success Criteria
1. RabbitMQ Connectivity	• No `SELF_SIGNED_CERT_IN_CHAIN` errors in `turbine-api` logs. • AMQP initialization logs succeed without repeating retries.
2. Pod Health & Stability	• All pod replicas reach `Ready` status. • Restart counts stabilize across all namespaces.
3. Playbook Execution	• Test playbook runs transition past `Pending` state. • Actions complete across native and container engines.
4. Queue Backlog	• Queue depth in RabbitMQ drains steadily without worker failures.

Escalation & Summary

If issues persist after completing certificate renewal, gather the following diagnostic outputs and escalate to Swimlane:

# Gather certificate, request, and issuer details
kubectl get certificate,certificaterequest,issuer -n default -o yaml > escalation-certs.yaml

# Export recent cert-manager logs
kubectl logs -n default deployment/cert-manager --since=1h > cert-manager.log