How to purge the Hangfire job queue
When large numbers (perhaps thousands) of relatively low-value jobs are Enqueued within Hangfire, and only a few are Processing slowly at any one time (which can happen for a variety of reasons), it may be necessary to purge the job queue in order for newer jobs of higher value to process in a typical rapid way. Follow the instructions below to purge the queue:
- Swimlane Platform Installer (Standalone and HA). Embedded : <namespace> is default Existing : <namespace> depends on environment
kubectl scale deployment swimlane-tasks --replicas=0 -n default
kubectl scale deployment swimlane-api --replicas=0 -n default
- Follow the instructions in MongoDB: Accessing the Mongo shell and other CLI tools with SSL to open a command-line query interface for MongoDB.
- Check the current indexes first before dropping the collections. Run this query again after dropping the collections to ensure all of the indexes were recreated. NOTE: The numbers of indexes depend on the Swimlane 10.x version used.
db.getCollectionNames().forEach(function(collection) {
if (collection.indexOf("hangfire") > -1){
indexes = db[collection].getIndexes();
print("Indexes for " + collection + ":");
print(indexes.length);
indexes.forEach(function(item){
print(item.name);
});
}
});
- Note the number of Recurring Jobs in the Hangfire Dashboard before dropping Hangfire collections:
After dropping the Hangfire collections, confirm the Recurring Jobs count is the same. If it isn't, restart the tasks pods again to re-initialize the Recurring Jobs.
- Issue the following commands to drop all hangfire collections in MongoDB
use Swimlane;
db.getCollectionNames().forEach(function(collname){
if(collname.includes('hangfire')){
print('delete collection: ' + collname);
db.getCollection(collname).drop()
}
});
- Start Swimlaneβs Task service again using:
Standalone:
kubectl scale deployment swimlane-tasks --replicas=1
kubectl scale deployment swimlane-api --replicas=1
HA:
kubectl scale deployment swimlane-tasks --replicas=3
kubectl scale deployment swimlane-api --replicas=3
- Confirm that the several Hangfire queues' numbers are accurate.