--- title: How to Clean Up Turbine Playbook Runs Data when the Openebs directory is filled up slug: turbine-user-guide/how-to-clean-up-turbine-playbook-runs-data-when-the-openebs-directory-is-filled-up docTags: createdAt: 2026-10-23T14:18:24.577Z --- - Playbook run GridFS files and chunks - Input/output GridFS files and chunks ## Cleanup scripts (attached below) Run the scripts in the following order: 1. `PlaybookRunsCleanup.js` 2. `PlaybookRunsJobsCleanup.js` 3. `PlaybookRunsFilesChunksCleanup.js` 4. `InputOutputDataFilesCleanup.js` Do not change the execution order. # 1. Preparation and safety checks ## 1.1 Schedule a maintenance window The deletion and compaction operations can generate significant MongoDB load, disk I/O, and replication traffic. Perform this activity during a maintenance window. Before proceeding: - Avoid running a platform snapshot or MongoDB backup at the same time. - Confirm that all MongoDB replicas are healthy. - Confirm that sufficient free disk space is available. - Take a supported backup before deleting production data. - Ensure the cleanup scripts are available on the machine from which `kubectl` is being run. ## 1.2 Set the namespace ```shell export NS=default ``` ## 1.3 Check MongoDB replica health ```shell kubectl -n "$NS" get pods | grep mongo ``` All MongoDB pods should be in the `Running` state and show all containers as ready. Example: ```javascript mongo-0 2/2 Running mongo-1 2/2 Running mongo-2 2/2 Running ``` # 2. Identify the MongoDB primary Run the following command: ```shell export NS=default && kubectl -n $NS exec mongo-0 -- mongosh -quiet -u Admin -p $(kubectl -n $NS get secret mongo-admin -o jsonpath="{.data.password}" | base64 --decode) --authenticationDatabase admin --tls --tlsAllowInvalidHostnames --tlsAllowInvalidCertificates admin --eval="rs.isMaster();" | grep primary ``` Example output: ```javascript Defaulted container "mongo" out of: mongo, copy-config (init), install (init), bootstrap (init) primary: 'mongo-0.mongo.default.svc.cluster.local:27017', ``` In this example, the primary replica is: ```javascript mongo-0 ``` You can also check the complete replica-set status: ```shell export NS=default && kubectl -n "$NS" exec mongo-0 -- mongosh -quiet -u Admin -p "$(kubectl -n "$NS" get secret mongo-admin -o jsonpath='{.data.password}' | base64 --decode)" --authenticationDatabase admin --tls --tlsAllowInvalidHostnames --tlsAllowInvalidCertificates admin --quiet --eval='rs.status().members.forEach(function(m) { print(m.name + " : " + m.stateStr) })' ``` Expected output will be similar to: ```javascript mongo-0.mongo.default.svc.cluster.local:27017 : PRIMARY mongo-1.mongo.default.svc.cluster.local:27017 : SECONDARY mongo-2.mongo.default.svc.cluster.local:27017 : SECONDARY ``` Record which pod is the primary and which pods are secondaries. # 3. Connect to the MongoDB primary Replace `mongo-0` with the primary pod identified in the previous step: ```shell export NS=default && kubectl -n $NS exec -it mongo-0 -- mongosh -u Admin -p $(kubectl -n $NS get secret mongo-admin -o jsonpath="{.data.password}" | base64 --decode) --authenticationDatabase admin --ssl --sslAllowInvalidHostnames --sslAllowInvalidCertificates Swimlane ``` # 4. Identify the large Turbine database Inside `mongosh`, run: ```javascript show dbs ``` Review the size of each database. ![](https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/nsTHU4uJ5HwwLcYtPa4GC_image.png) The relevant tenant databases usually have names similar to: ```javascript SwimlaneTurbine_XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX ``` Select the database that is consuming significant storage: ```javascript use SwimlaneTurbine_XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX ``` # 5. Review collection sizes Run the following function inside the selected database: ```javascript function getReadableFileSizeString(fileSizeInBytes) { var i = -1; var byteUnits = [' kB', ' MB', ' GB', ' TB', 'PB', 'EB', 'ZB', 'YB']; do { fileSizeInBytes = fileSizeInBytes / 1024; i++; } while (fileSizeInBytes > 1024); return Math.max(fileSizeInBytes, 0.1).toFixed(1) + byteUnits[i]; }; var collectionNames = db.getCollectionNames(), stats = []; collectionNames.forEach(function (n) { stats.push(db.getCollection(n).stats()); }); stats = stats.sort(function(a, b) { return b['size'] - a['size']; }); for (var c in stats) { print(stats[c]['ns'] + ": " + getReadableFileSizeString(stats[c]['size']) + " (" + getReadableFileSizeString(stats[c]['storageSize']) + ")"); } ``` The first value represents the logical document size. The value in parentheses represents the allocated collection storage. ![](https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/fCols5RcI5yjQuqPyQJ8O_image.png) Pay particular attention to: ```javascript PlaybookRuns PlaybookRuns.Jobs PlaybookRuns.files PlaybookRuns.chunks InputOutputData.files InputOutputData.chunks ``` Save the output so it can be compared after cleanup and compaction. Exit `mongosh` when finished: ```javascript exit ``` # 6. Configure the retention period Extract the cleanup package: ```shell unzip CleanupScripts2.zip -d CleanupScripts2 cd CleanupScripts2 ``` The four required files are: ```javascript PlaybookRunsCleanup.js PlaybookRunsJobsCleanup.js PlaybookRunsFilesChunksCleanup.js InputOutputDataFilesCleanup.js ``` ## 6.1 Edit `PlaybookRunsCleanup.js` Open the file: ```shell vi PlaybookRunsCleanup.js ``` Change: ```javascript const daysToKeep = 1; ``` For example, to retain the most recent 30 days: ```javascript const daysToKeep = 30; ``` The script deletes `PlaybookRuns` documents whose `created` value is older than the calculated threshold. ## 6.2 Edit `PlaybookRunsJobsCleanup.js` Open the file: ```shell vi PlaybookRunsJobsCleanup.js ``` Change: ```javascript const daysToKeep = 30; ``` Set it to the same retention period: ```javascript const daysToKeep = 30; ``` This script deletes `PlaybookRuns.Jobs` documents whose `start` value is older than the threshold. ## 6.3 Edit `PlaybookRunsFilesChunksCleanup.js` Open the file: ```shell vi PlaybookRunsFilesChunksCleanup.js ``` Change: ```javascript const DAYS_TO_KEEP = 30; ``` Set the required retention: ```javascript const DAYS_TO_KEEP = 30; ``` This script deletes historical files from: ```javascript PlaybookRuns.files ``` It also deletes their related chunks from: ```javascript PlaybookRuns.chunks ``` ## 6.4 Edit `InputOutputDataFilesCleanup.js` Open the file: ```shell vi InputOutputDataFilesCleanup.js ``` Change: ```javascript const DAYS_TO_KEEP = 30; ``` Set it to the required retention period: ```javascript const DAYS_TO_KEEP = 30; ``` This script deletes historical orphaned input/output files and associated chunks. ## Critical scope warning The supplied scripts contain logic similar to: ```javascript dbName.startsWith("SwimlaneTurbine_") ``` Therefore, they will process **all databases beginning with `SwimlaneTurbine_`**, not only the database specified at the end of the `mongosh` command. Before execution, confirm that the selected retention period is appropriate for every `SwimlaneTurbine_` database in the cluster. **NOTE:** Do not run the supplied scripts unchanged when only one tenant database should be cleaned. # 7. Confirm the deletion threshold For example, when retaining 30 days, preview the threshold using: ```shell date -u -d '30 days ago' ``` Before running the scripts, the number of records eligible for deletion can be reviewed from `mongosh`. Connect to the primary and select the target database: ```javascript use SwimlaneTurbine_XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX ``` Count old playbook runs: ```javascript db.PlaybookRuns.find( {"start" : { $lte: ISODate('2026-07-29T00:00Z')}} ).count(); ``` Count old playbook jobs: ```javascript db.PlaybookRuns.Jobs.find( {"start" : { $lte: ISODate('2026-07-29T00:00Z')}} ).count(); ``` These are validation counts only. They do not delete data. # 8. Run the cleanup scripts Confirm the MongoDB primary immediately before running the scripts because the primary may have changed. ```shell export NS=default && kubectl -n $NS exec mongo-0 -- mongosh -u Admin -p $(kubectl -n $NS get secret mongo-admin -o jsonpath="{.data.password}" | base64 --decode) --authenticationDatabase admin --ssl --sslAllowInvalidHostnames --sslAllowInvalidCertificates admin --eval="rs.isMaster();" | grep primary ``` ## 8.1 Run `PlaybookRunsCleanup.js` ```shell kubectl exec -it -- mongosh -u Admin -p $(kubectl get secret mongo-admin -o jsonpath="{.data.password}" | base64 --decode) --authenticationDatabase admin --ssl --sslAllowInvalidHostnames --sslAllowInvalidCertificates SwimlaneTurbine_XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX < PlaybookRunsCleanup.js ``` Monitor the output for messages similar to: ```javascript Processing database: SwimlaneTurbine_... Removing Runs older than ... Runs deletion: ... ``` Do not proceed until the script finishes. ## 8.2 Run `PlaybookRunsJobsCleanup.js` ```shell kubectl exec -it -- mongosh -u Admin -p $(kubectl get secret mongo-admin -o jsonpath="{.data.password}" | base64 --decode) --authenticationDatabase admin --ssl --sslAllowInvalidHostnames --sslAllowInvalidCertificates SwimlaneTurbine_XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX < PlaybookRunsJobsCleanup.js ``` Monitor for: ```javascript Processing database: SwimlaneTurbine_... Removing Jobs older than ... Jobs deletion: ... ``` ## 8.3 Run `PlaybookRunsFilesChunksCleanup.js` ```shell kubectl exec -it -- mongosh -u Admin -p $(kubectl get secret mongo-admin -o jsonpath="{.data.password}" | base64 --decode) --authenticationDatabase admin --ssl --sslAllowInvalidHostnames --sslAllowInvalidCertificates SwimlaneTurbine_XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX < PlaybookRunsFilesChunksCleanup.js ``` Monitor for: ```javascript Deleted ... files and associated chunks. Processed ... files. ``` ## 8.4 Run `InputOutputDataFilesCleanup.js` ```shell kubectl exec -it -- mongosh -u Admin -p $(kubectl get secret mongo-admin -o jsonpath="{.data.password}" | base64 --decode) --authenticationDatabase admin --ssl --sslAllowInvalidHostnames --sslAllowInvalidCertificates SwimlaneTurbine_XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX < InputOutputDataFilesCleanup.js ``` Monitor for: ```javascript Deleted ... files and associated chunks. Processed ... files. ``` # 9. Verify cleanup completion Reconnect to the MongoDB primary: ```shell export NS=default && kubectl -n $NS exec mongo-0 -- mongosh -u Admin -p $(kubectl -n $NS get secret mongo-admin -o jsonpath="{.data.password}" | base64 --decode) --authenticationDatabase admin --ssl --sslAllowInvalidHostnames --sslAllowInvalidCertificates admin --eval="rs.isMaster();" | grep primary ``` Inside `mongosh`, run: ```javascript show dbs ``` ![](https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/-6pc2Zeo8i5Pkf_N4nzJj_image.png) Review the size of each database. The relevant tenant databases usually have names similar to: ```javascript SwimlaneTurbine_XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX ``` Select the database again ```javascript use SwimlaneTurbine_XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX ``` Run the following function inside the selected database: ```javascript function getReadableFileSizeString(fileSizeInBytes) { var i = -1; var byteUnits = [' kB', ' MB', ' GB', ' TB', 'PB', 'EB', 'ZB', 'YB']; do { fileSizeInBytes = fileSizeInBytes / 1024; i++; } while (fileSizeInBytes > 1024); return Math.max(fileSizeInBytes, 0.1).toFixed(1) + byteUnits[i]; }; var collectionNames = db.getCollectionNames(), stats = []; collectionNames.forEach(function (n) { stats.push(db.getCollection(n).stats()); }); stats = stats.sort(function(a, b) { return b['size'] - a['size']; }); for (var c in stats) { print(stats[c]['ns'] + ": " + getReadableFileSizeString(stats[c]['size']) + " (" + getReadableFileSizeString(stats[c]['storageSize']) + ")"); } ``` You may see the relevant DB has been reduced in size. # 10. Understand deletion versus disk reclamation Deleting MongoDB documents does not normally reduce the operating-system size of the MongoDB data files immediately. After deletion: - The removed space becomes reusable by MongoDB. - Filesystem usage may not decrease until collection compaction is performed. - Compaction is separate from deletion. **Run compaction only after all four cleanup scripts have completed successfully and replica health is normal.** # 11. Compact affected collections ## Important compaction considerations - Perform compaction during a maintenance window. - Run it on only one MongoDB member at a time. - Start with a secondary. - Wait for that member to recover fully before proceeding. - Do not compact multiple members simultaneously. - Continuously monitor replication health and disk availability. - Reconfirm primary and secondary roles before every node. - Do not assume that `mongo-0`, `mongo-1`, or `mongo-2` will retain the same role throughout the activity. Affected collections: ```javascript PlaybookRuns PlaybookRuns.Jobs PlaybookRuns.files PlaybookRuns.chunks InputOutputData.files InputOutputData.chunks ``` ## 11.1 List the current replica roles ```shell export NS=default && kubectl -n "$NS" exec mongo-0 -- mongosh -u Admin -p "$(kubectl -n "$NS" get secret mongo-admin -o jsonpath='{.data.password}' | base64 --decode)" --authenticationDatabase admin --ssl --sslAllowInvalidHostnames --sslAllowInvalidCertificates admin --quiet --eval='rs.status().members.forEach(function(m) { print(m.name + " : " + m.stateStr) })' ``` Identify the two secondary pods. Example: ```javascript mongo-0.mongo.default.svc.cluster.local:27017 : PRIMARY mongo-1.mongo.default.svc.cluster.local:27017 : SECONDARY mongo-2.mongo.default.svc.cluster.local:27017 : SECONDARY ``` ## 11.2 Connect directly to the first secondary Example: ```shell export SECONDARY_1=mongo-1 && kubectl -n "$NS" exec -it "$SECONDARY_1" -- mongosh -u Admin -p "$(kubectl -n "$NS" get secret mongo-admin -o jsonpath='{.data.password}' | base64 --decode)" --authenticationDatabase admin --ssl --sslAllowInvalidHostnames --sslAllowInvalidCertificates SwimlaneTurbine_XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX ``` ## 11.3 Run the compaction commands Run each command individually and wait for the response before proceeding: ```javascript db.runCommand({ compact: "PlaybookRuns", force: true }); ``` ```javascript db.runCommand({ compact: "PlaybookRuns.Jobs", force: true }); ``` ```javascript db.runCommand({ compact: "PlaybookRuns.files", force: true }); ``` ```javascript db.runCommand({ compact: "PlaybookRuns.chunks", force: true }); ``` ```javascript db.runCommand({ compact: "InputOutputData.files", force: true }); ``` ```javascript db.runCommand({ compact: "InputOutputData.chunks", force: true }); ``` A successful result should include: ```javascript { bytesFreed: Long('66464616448'), ok: 1, '$clusterTime': { clusterTime: Timestamp({ t: 1779114197, i: 27 }), signature: { hash: Binary.createFromBase64('edoM6vmsYg3xGHODXlClcWzfkWY=', 0), keyId: Long('7588083953530044420') } }, operationTime: Timestamp({ t: 1779114197, i: 27 }) } ``` Exit: ```javascript exit ``` ## 11.4 Compact the second secondary Repeat the same process against the second secondary as above. ## 11.5 Compact the current primary Only proceed after both secondaries are healthy. Recheck the current primary: ```shell export NS=default && kubectl -n $NS exec mongo-0 -- mongosh -u Admin -p $(kubectl -n $NS get secret mongo-admin -o jsonpath="{.data.password}" | base64 --decode) --authenticationDatabase admin --ssl --sslAllowInvalidHostnames --sslAllowInvalidCertificates admin --eval="rs.isMaster();" | grep primary ``` # 12. Post-cleanup validation ## 12.1 Verify MongoDB replica health ```shell export NS=default && kubectl -n "$NS" exec mongo-0 -- mongosh -u Admin -p "$(kubectl -n "$NS" get secret mongo-admin -o jsonpath='{.data.password}' | base64 --decode)" --authenticationDatabase admin --ssl --sslAllowInvalidHostnames --sslAllowInvalidCertificates admin --quiet --eval='rs.status().members.forEach(function(m) { print(m.name + " | " + m.stateStr + " | health=" + m.health) })' ``` ![](https://api.archbee.com/api/optimize/n8uUzk0MM4uC57-zlHNtY/96UlAeOzUD7P82FBHCRCw_image.png) **Expected:** - One `PRIMARY` - Remaining members `SECONDARY` - `health=1` for all members ## 12.2 Verify MongoDB pod status ```shell export NS=default && kubectl -n "$NS" get pods | grep mongo ``` ## 12.3 Check filesystem space Run on each Kubernetes node: ```shell df -h | head -20 ``` If MongoDB data is under `/var/openebs`, also check: ```shell du -sh /var/openebs/* ``` The exact host-level reduction depends on MongoDB storage behavior, storage engine allocation, and whether compaction completed successfully. ## 12.4 Review the collection sizes again Reconnect to the database and rerun the collection-size function from Step 5. Compare: - Document count - Logical collection size - Storage size - Filesystem usage ## 12.5 Resume activity gradually After confirming that: - MongoDB replication is healthy - MongoDB pods are stable - No unexpected database errors are present - Disk utilization is acceptable Resume paused playbooks and scheduled operations. Avoid starting a large snapshot immediately while MongoDB is still stabilizing after compaction. # 13. Rollback and stop conditions Stop the procedure and investigate before continuing if any of the following occurs: - No MongoDB primary is available. - A secondary enters `RECOVERING`, `ROLLBACK`, `UNKNOWN`, or `DOWN`. - Replication lag continues to increase. - Available disk space becomes critically low. - A cleanup script exits unexpectedly. - A compact command returns `ok: 0`. - MongoDB pods restart or become unready. - Application errors increase significantly. Deleted documents cannot be restored without a valid backup. Do not manually delete MongoDB files from the underlying filesystem. # 14. Recommended execution record Record the following for the change ticket: - Maintenance start and end times - MongoDB primary before cleanup - Selected retention period - Databases processed - Counts before cleanup - Counts after cleanup - Script start and completion times - Output from each cleanup script - Replica health before and after compaction - Collection sizes before and after compaction - Filesystem utilization before and after compaction - Any errors or warnings encountered