Troubleshooting Record Event Triggers Stop Working After RabbitMQ Restarts

Purpose Diagnose and recover a Swimlane/Turbine cluster where record-event playbooks stop running after RabbitMQ restarts, even though the record itself still saves and the pods stay `Running`. The failure is a stuck AMQP connection in `swimlane-records`. The service keeps trying to publish `SensorEventV2` on a connection RabbitMQ already closed with `vhost '/' is down`. Restarting `swimlane-records` after RabbitMQ is healthy opens a new connection and restores the trigger path.

Scope

Use this runbook when all of the following are true:

Record-event playbooks do not start when a record is created or updated. Case-management and other application records are the usual report.

The record save itself succeeds. The UI does not show a save error.

`swimlane-records` logs `AlreadyClosedException` with AMQP code `541` and `vhost '/' is down`.

`turbine.engine.sensor-event` exists, but few or no `turbine.record.update` / `turbine.record.create` events arrive at `turbine-engine`.

RabbitMQ pods are `Running`. The records pods were not restarted after the last RabbitMQ restart. Do not use this runbook when the engine itself cannot connect to RabbitMQ, or when logs show `SELF_SIGNED_CERT_IN_CHAIN`. That is a certificate mismatch. See `runbooks/post-upgrade-playbook-failures-ca-mismatch.md`. Customer-facing issue Playbooks that should start from a record create or record update do not run. The customer often describes this as record-event triggers not working, or as case-management records not driving the expected workflow. Likely visible effects:

A record is created or updated, including by another playbook, and the follow-on record-event playbook never starts.

Case-management tracking IDs (`CM-*`) and similar application records are affected. In the example incident, ITSM, DF, TI, IF, UDB, and HDB records failed the same way.

Scheduled playbooks and webhook playbooks can keep running. Only the record-event publish from `swimlane-records` is broken.

The Swimlane UI stays up. Kubernetes shows `swimlane-records`, `turbine-engine`, and `rabbitmq-server-*` as `Running`.

Impact `swimlane-records` publishes the record-event signal while it saves the record. The call is `SensorEventPublisher.Publish(SensorEventV2)`, from `EventService.SendRecordUpdateEventAsync`, inside `RecordRepositoryProcessor.OnSaveAsync` / `PatchAsync`. When that publish throws, the service logs the error and still marks the record job `success`. The record is stored. The signal never reaches `turbine.engine.sensor-event`, so the engine cannot start the playbook. Events that failed during the outage are not replayed. Records saved in that window have to be saved again if their playbooks must run.

Environment for this incident

Item	Value
Diagnostics bundle	`turbine_diagnostics_turbine_20261106_154121.tar.gz`
Bundle window	6 Nov 2026, records logs about 13:00–14:41 UTC
Namespace	`turbine`
Application	Swimlane / Turbine `26.1.4`
RabbitMQ	`4.2.6` (`rabbitmq:4.2.6-management-alpine`), Khepri metadata store
Kubernetes	v1.30.6
Control plane	`lptrbc01`, `lptrbc02`, `lptrbc03`
Workers	`lptrbw01`–`lptrbw05`
Account	`37789c38-3ae0-4f96-b304-5772dffba9bb`
Tenant	`1a682c58-3aa4-4d15-9815-1683146dc34e`
Records label	`service=swimlane-records` (`app.kubernetes.io/name` is `turbine`, not `swimlane-records`)

RabbitMQ placement at collection time:

Pod	Node	File handles at startup	Age
`rabbitmq-server-0`	`lptrbw03`	about 99903 (89910 sockets)	25h
`rabbitmq-server-1`	`lptrbw02`	1024 (927 usable, 832 sockets). Khepri metadata leader.	25h
`rabbitmq-server-2`	`lptrbw05`	about 99903 (89910 sockets)	25h

`rabbitmq-server-1` logged this at boot on 5 Nov 2026 12:52 UTC:

Limiting to approx 927 file handles (832 sockets)
Available file handles: 1024. Please consider increasing system limits

The other two brokers logged about 99903 file handles. RabbitMQ expects at least 65536.

Findings from the example support bundle

1. Every record patch failed while publishing the trigger `swimlane-records` had been up for 135 days (`swimlane-records-7fd7b6b556-*`). In the captured hour and a half it logged only one error, repeated on every patch:

Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Peer,
code=541, text='INTERNAL_ERROR - access to vhost '/' refused for user 'turbine': vhost '/' is down'

Exception type: `RabbitMQ.Client.Exceptions.AlreadyClosedException`. Stack:

RabbitMQ.Client.Framing.AutorecoveringConnection.CreateChannelAsync
  at SensorEventPublisher.GetChannelAsync (SensorEventPublisher.cs:101)
  at SensorEventPublisher.Publish(SensorEventV2)
  at EventService.SendPlaybookRunsEventAsync (EventService.cs:380)
  at EventService.SendRecordUpdateEventAsync (EventService.cs:317)
  at RecordRepositoryProcessor.OnSaveAsync (RecordRepositoryProcessor.cs:941)

The log line is `RecordRepositoryProcessor.PatchAsync` line 327. The actor is `swimlane_system`, so these are playbook and system updates, not only clicks in the UI. Counts in the bundle window (6 Nov 2026, 13:00–14:40 UTC):

Pod	Errors	Notable tracking IDs
`swimlane-records-7fd7b6b556-8g4sn`	94	`CM-15623`, `ITSM-1784`, `CM-15573`, `ITSM-1799`
`swimlane-records-7fd7b6b556-dslsw`	88	`ITSM-1801`, `CM-15573`, `CM-15623`
`swimlane-records-7fd7b6b556-vz7km`	36	`CM-15623`, `ITSM-1799`

On the first pod, 58 of 59 patched records hit this error. Tracking prefixes were `CM`, `ITSM`, `DF`, `TI`, `IF`, `UDB`, and `HDB`. Application IDs with the most failures were `aBllm1llxhhK0TnMV`, `aD1YLTirk6uKnjSbW`, `aC4iPeFXeYUpiHxvl`, and `aDfRdhhP7pVLBLnj1`. A typical sequence for one correlation ID:

Patching record aBkm58WuXB5eNKJaj using the V2 patching algorithm...
Core.Events.DataUpdated event processed in 2.4804 ms
ERROR  Already closed ... vhost '/' is down
Finished processing job ... publishing job result...
Job result published   JobStatus=success

The in-process `DataUpdated` event is not the playbook trigger. The playbook trigger is the AMQP `SensorEventV2` publish, and that is the call that failed. The job status stayed `success` for all 218 errors, which is why the UI still showed a saved record.

2. RabbitMQ had restarted, and the records pods had not RabbitMQ pods were 25 hours old. They booted on 5 Nov 2026:

`rabbitmq-server-2` at 12:50 UTC

`rabbitmq-server-1` at 12:52 UTC

`rabbitmq-server-0` at 12:53 UTC - `rabbitmq-server-0` at 12:53 UTC During that boot the vhost was down. Other services logged the same `vhost '/' is down` text and then recovered:

`identity-api`, `notifications-api`, `security-intelligence`, and the elastic-ingestion pods logged it only around 5 Nov 10:53–12:52 UTC, while the broker was restarting.

`swimlane-records` was still logging it on 6 Nov at 14:40 UTC. - `swimlane-records` was still logging it on 6 Nov at 14:40 UTC. MassTransit clients retried and reconnected. The RabbitMQ .NET client inside `swimlane-records` stayed on the closed connection. `CreateChannelAsync` kept throwing `AlreadyClosedException` instead of opening a new channel. At 6 Nov 14:00 UTC, RabbitMQ also logged many long-lived AMQP connections closing after about one day (`client unexpectedly closed TCP connection`). That lines up with `swimlane-api`, `swimlane-tasks`, `turbine-api`, and `turbine-engine` being rolled about 40 minutes before the bundle was collected. Those new pods got fresh connections. `swimlane-records` was not rolled.

3. The engine was healthy and was waiting for events `turbine-engine` `26.1.4` was `Running` (6/6). After its own restart it logged:

Declaring SensorEventMessageHandler
Activating SensorEventMessageHandler consumer

`turbine.engine.sensor-event` is the queue those consumers use. The engine was able to match `turbine.record.update` events that did arrive (some from `swimlane-api`, which had a live connection). It could not run playbooks for the publishes `swimlane-records` dropped.

4. Two sets of queues were already missing Two sets of queues were already missing These names were absent after the 5 Nov broker restart. Clients checked for them with a passive `queue.declare` and got `404 NOT_FOUND`. They did not create the queues.

no queue 'turbine.command.type.api' in vhost '/'
no queue '1a682c58-3aa4-4d15-9815-1683146dc34e.swimlane.success' in vhost '/'
no queue '1a682c58-3aa4-4d15-9815-1683146dc34e.swimlane.fail' in vhost '/'
no queue '1a682c58-3aa4-4d15-9815-1683146dc34e.swimlane.reject' in vhost '/'

`turbine.command.type.api` was logged once per `turbine-api` pod at startup on 6 Nov 14:01 UTC (`turbine-api-5698fc49c8-*`). The API then continued and received playbook-run updates, so this channel error did not take the API down. The tenant `swimlane.success`, `.fail`, and `.reject` names were logged by RabbitMQ at 5 Nov 12:55 UTC (minutes after the broker boot) and again at 6 Nov 14:01 UTC when `swimlane-tasks-d55b6cc58-*` started. The tasks pods stayed `Running`.

5. Change-stream sync was not the trigger failure Change-stream sync was not the trigger failure `swimlane-data-changestream` logged `ElasticDataSync feature flag is disabled` at startup on 6 Nov 13:53 UTC and then only listened on port 8080. That flag controls Mongo-to-Elasticsearch sync inside that worker. It does not publish `SensorEventV2`. Do not treat that warning as the cause of missing record-event playbooks.

Root cause RabbitMQ restarted on 5 Nov 2026 around 12:50 UTC. While the vhost was down it closed client connections with AMQP `541` / `vhost '/' is down`. `swimlane-records` did not reconnect. Its pods were 135 days old and kept using the closed `AutorecoveringConnection`. Every later record save that had orchestration tasks called `SensorEventPublisher.Publish`, threw `AlreadyClosedException`, and dropped the record-event signal. The record write still completed. A second condition makes a repeat more likely. `rabbitmq-server-1` on `lptrbw02` is the Khepri metadata leader and started with only 1024 file handles. The other brokers had about 99903. Under Turbine's connection count, that node can refuse new sockets or stop the vhost and hand clients the same `541` close again.

Why Kubernetes health can look normal

`swimlane-records`, `turbine-engine`, and all three `rabbitmq-server` pods were `1/1 Running` with 0 restarts in the bundle.

Record jobs were logged as `JobStatus=success`.

The engine was consuming `turbine.engine.sensor-event` and running cron playbooks.

The failure is one publisher holding a dead connection. Readiness probes do not check that the records publisher can open an AMQP channel.

Diagnosis Run these from a host that can use `kubectl` against the Turbine namespace. This incident used `turbine`.

1. Confirm the records pods and the label `app.kubernetes.io/name` on these pods is `turbine`. The selector that matches records is `service=swimlane-records`.

kubectl -n turbine get pods -l service=swimlane-records -o wide
kubectl -n turbine get pods -l service=swimlane-records --show-labels
kubectl -n turbine get pods -l app.kubernetes.io/name=rabbitmq -o wide

2. Count the closed-connection errors Count the closed-connection errors

for p in $(kubectl -n turbine get pods -l service=swimlane-records -o name); do
  echo "$p: $(kubectl -n turbine logs $p -c records --since=60m | grep -c -E "is down|AlreadyClosed|code=541")"
done

A count of `0` only helps if the pods have actually saved records in that window. If the count is non-zero, print one error and confirm the stack mentions `SensorEventPublisher`:

kubectl -n turbine logs deploy/swimlane-records -c records --since=60m \
  | grep -E "is down|AlreadyClosed|code=541" \
  | head -5

3. Confirm the trigger queue and its consumers

kubectl -n turbine exec rabbitmq-server-0 -c rabbitmq -- \
  rabbitmqctl list_queues name type consumers messages \
  | grep -E 'sensor-event|command.type.api|swimlane\.(success|fail|reject)'

4. See whether the engine is receiving record events `kubectl logs deploy/turbine-engine` reads one pod. Check more than one if the first pod is quiet.

kubectl -n turbine logs deploy/turbine-engine -c turbine-engine --since=30m \
  | grep -E 'turbine.record.update|turbine.record.create|matched'

`matched 1 playbook` (or more), followed by `Published queued playbook run` and a playbook run id: the trigger path is working for that event. Interpretation:

`matched 0 playbook(s)` and `Completed event ... with 0 playbook runs`: the event was delivered. No enabled playbook trigger matched that record. This is not the dead-connection failure.

No `turbine.record.*` lines while `swimlane-records` is logging `code=541`: publishes are being dropped.

5. Check the file-handle limit on every RabbitMQ pod

for i in 0 1 2; do
  echo "== rabbitmq-server-$i"
  kubectl -n turbine logs rabbitmq-server-$i -c rabbitmq \
    | grep -E "Limiting to approx|Available file handles"
done

```

kubectl -n turbine exec rabbitmq-server-0 -c rabbitmq -- rabbitmqctl cluster_status

Resolution

1. Make sure RabbitMQ is up Do this in order. Do not roll `turbine-engine` unless step 4 shows it is not consuming. A rolling restart of the engine interrupts playbook runs that are already active. If a broker pod is unhealthy, or you are recovering from the low file-handle limit, fix the node limit before recycling that pod.

ulimit -n
systemctl show containerd -p LimitNOFILE

kubectl -n turbine delete pod rabbitmq-server-1
kubectl -n turbine exec rabbitmq-server-0 -c rabbitmq -- rabbitmqctl cluster_status
kubectl -n turbine logs rabbitmq-server-1 -c rabbitmq | grep "Limiting to approx"

2. Restart swimlane-records The new log line must show tens of thousands of file handles, not 927. If RabbitMQ is already `Running` and the only problem is the stuck records connection, you still need the records restart in the next step. Deleting a RabbitMQ pod alone does not repair a client that is already holding a closed connection. In this incident, both were done: one RabbitMQ pod was deleted, then `swimlane-records` was restarted.

kubectl -n turbine rollout restart deploy/swimlane-records
kubectl -n turbine rollout status deploy/swimlane-records
kubectl -n turbine get pods -l service=swimlane-records

3. Do not restart the other deployments to recreate queues

# Does not create the missing queues. Do not run this for that purpose.
kubectl -n turbine rollout restart deploy/swimlane-api deploy/swimlane-tasks deploy/turbine-api deploy/turbine-engine

4. Re-list the trigger queue `turbine-api` and `swimlane-tasks` passive-declare `turbine.command.type.api` and `<tenant>.swimlane.success|fail|reject`. After a restart, they log `NOT_FOUND` again. `turbine-engine` is already the consumer of `turbine.engine.sensor-event`. Restarting it drops in-flight playbook runs and does not fix the records publisher. Leave those queues alone until you know which component declares them and what the queue arguments are. Creating them by hand with the wrong type returns `PRECONDITION_FAILED` and the consumers still fail.

kubectl -n turbine exec rabbitmq-server-0 -c rabbitmq -- \
  rabbitmqctl list_queues name type consumers messages \
  | grep -E 'sensor-event|command.type.api|swimlane\.(success|fail|reject)'

turbine.engine.sensor-event     quorum  6       0

Validation

Records publisher

for p in $(kubectl -n turbine get pods -l service=swimlane-records -o name); do
  echo "$p: $(kubectl -n turbine logs $p -c records --since=15m | grep -c -E "is down|AlreadyClosed|code=541")"
done

pod/swimlane-records-ff7bfddd4-5xhlw: 0
pod/swimlane-records-ff7bfddd4-7l6z6: 0
pod/swimlane-records-ff7bfddd4-9bt8r: 0

Engine received a record event An empty loop means the label matched no pods. Check `service=swimlane-records` before treating silence as success.

Event turbine.record.update ("f33ea18d-a689-4b66-8c92-6e47864e62cc") matched 1 playbook(s) to trigger (68751c05ead1abb1f2622022
Playbook run document ... triggering playbook "68751c05ead1abb1f2622022" inserted id "6aedefeb0b58aefd7bc91252"
Published queued playbook run "6aedefeb0b58aefd7bc91252" from event "turbine.record.update"

```

Starting playbook run "6aedf02bc3e231c7e019b6ce" ("solution_aFBzaC73DTx6Ty4sv_playbook_adj6q") triggered by event "turbine.record.update"
Published success playbook run "6aedf03032b0854966850d2e" from event "turbine.record.update" (94feec01-1fdd-4c0e-84c8-fb1f12efb83b)
Completed event "turbine.record.update" (94feec01-1fdd-4c0e-84c8-fb1f12efb83b) with 1 playbook runs

Customer check Two other `turbine.record.update` events at 15:20:07 and 15:20:12 UTC were delivered and matched 0 playbooks (`5f429317-ca67-4488-a46a-3c1078c52db2`, `646bdbb0-4152-4e0a-8294-892212956e8c`). Those records had no matching enabled trigger. They are not leftovers of the `541` failure. Cron events in the same log (`turbine.schedule.cron` matching playbooks `687dfda4db943ae20a2541fa` and `687dfda5db943ae20a2541fb`) show the engine was also consuming scheduled work.

Escalation data

kubectl -n turbine get pods -o wide | grep -E 'swimlane-records|rabbitmq-server|turbine-engine|swimlane-api|turbine-api|swimlane-tasks'
kubectl -n turbine logs deploy/swimlane-records -c records --since=30m | grep -E "is down|AlreadyClosed|code=541" | tail -20
for i in 0 1 2; do kubectl -n turbine logs rabbitmq-server-$i -c rabbitmq | grep -E "Limiting to approx|Available file handles|vhost"; done
kubectl -n turbine exec rabbitmq-server-0 -c rabbitmq -- rabbitmqctl cluster_status
kubectl -n turbine exec rabbitmq-server-0 -c rabbitmq -- rabbitmqctl list_queues name type consumers messages state

```

Incident summary template

Summary:
Record-event playbooks stopped after RabbitMQ restarted. swimlane-records kept a closed AMQP connection and dropped SensorEventV2 publishes. Record saves still returned success.

Customer symptom:
Record-event triggers did not run for case-management and other application records. The records themselves were saved.

Cause:
RabbitMQ restarted (this incident: 5 Nov 2026 ~12:50 UTC) and closed clients with AMQP 541, vhost '/' is down. swimlane-records (up 135 days) never opened a new connection. rabbitmq-server-1 on lptrbw02 had also started with only 1024 file handles.

Fix:
Confirmed RabbitMQ was up, restarted deploy/swimlane-records, and verified turbine.engine.sensor-event had 6 consumers and a new turbine.record.update event queued a playbook run.

Not fixed by this change:
Missed events were not replayed. turbine.command.type.api and the tenant swimlane.success/fail/reject queues were still absent. Restarting swimlane-api, swimlane-tasks, turbine-api, or turbine-engine does not create them.

Validation:
swimlane-records code=541 count was 0 on all three new pods. turbine-engine logged turbine.record.update matched 1 playbook and a later record-event playbook run completed success.