Troubleshooting-playbook-execution-fails-with-http-500-polly-circuitbreaker

{ "Date": "2026-10-22 08:31:59.3262", "HostName": "swimlane-api-6c7dff8d5-44wff", "LayoutName": "FullDebug", "Level": "Error", "UserName": "[email protected]", "MethodName": "Api.Infrastructure.Filters.GlobalExceptionFilter.OnException", "LineNumber": "33", "Message": "Unhandled exception occurred while processing API request: The circuit is now open and is not allowing calls.", "Scope": {"TraceId":"0HNPFRMNC2KU7:00000001","RequestId":"29369-1-1792657918.421"}, "ExceptionMessage": "The circuit is now open and is not allowing calls.", "ExceptionMethod": "Void OnActionPreExecute()", "ExceptionType":
The circuit is now open and is not allowing calls.
Polly.CircuitBreaker.BrokenCircuitException

Origin: swimlane-api pod, container api. In this incident, the error appeared in:

swimlane-api container

The failure occurred in OrchestrationRestProvider.GetPlaybookByIdAsync while retrieving a playbook before execution.

1. Check pod status and API logs

kubectl get pods -n default | grep -E 'swimlane-api|turbine-api'

kubectl logs -n default <swimlane-api-pod> -c api --since=30m \
  | grep -Ei 'BrokenCircuitException|circuit is now open|GetPlaybookByIdAsync'

The circuit breaker error is a symptom of earlier failed requests; it does not by itself prove a proxy issue.

2. Check the orchestration URL and proxy bypass settings

kubectl exec -n default <swimlane-api-pod> -c api -- printenv \
  SWIMLANE_Orchestration__BaseUrl NO_PROXY no_proxy

For this environment, the orchestration URL was:

http://turbine-api.default.svc.cluster.local:3000/

If a proxy is configured, ensure the destination hostname is covered by NO_PROXY. The short entry turbine-api alone does not cover this fully qualified hostname. If lowercase no_proxy also exists, check it for conflicting settings.

3. Correct missing bypass entries

kubectl edit deployment swimlane-api -n default

Under container api → env → NO_PROXY, append these entries to the existing comma-separated value:

turbine-api.default.svc.cluster.local,.svc.cluster.local

Preserve all existing entries. Saving the container environment change automatically triggers a rollout.

4. Verify recovery

kubectl rollout status deployment/swimlane-api -n default

Retry the affected playbook and check the new API pod’s logs for recurring errors. In this incident, playbook execution succeeded after the change.

5. Persist the fix

Add the same bypass entries to the Admin Console’s proxy configuration so future deployments and upgrades retain them.

If the issue continues: Collect fresh swimlane-api and turbine-api logs, including the failures preceding the circuit breaker error. Investigate service connectivity, DNS, authentication, and downstream errors rather than assuming every circuit breaker error has the same cause.