SSDEEP tasks hanging and generating a large Hangfire queue

SSDEEP tasks are taking longer to process (up to over 24 hours). The following errors are seen from the Hangfire page:

ProxyError: HTTPSConnectionPool(host='swimlane.corpzone.internalzone.com', port=443): Max retries exceeded with url: /api/user/authorize (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x0000000003232C88>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed

Dropping the Hangfire JobGraph collection db.hangfire.jobGraph.drop() does not resolve the issue.

Restarting the Swimlane task service on Windows seems to help resolve the issue for just a few minutes, then the issue starts again.

Restarting the Tasks server on a Linux system with docker stop sw_tasks and docker start sw_tasks also helps for a few minutes, then the issue starts again.

The issue lies in the SSDEEP integration task python script. Workaround: Modify the following in the python integration script:

The Loopback time:

The loopback_time is used to create a record filter. Then a “for loop” will run through the filter and process the data. If there are thousands of records that come up in the search filter, this may cause the task to be stuck in a loop for hours, causing a huge queue.

The loopback_time variable is set to 24 hours (1 day) by default . Set this to fewer hours between 1 and 12 hours. 2. Comment out the lines for Deduped Email and Tracking Id lines in the fuzzymatch():

At times, depending on what you get from the logs, you may need to comment out the lines for NOT_EQ for Deduped Email and Tracking Id:

records.filter('Deduped Email', NOT_EQ, 'Yes') records.filter('Tracking Id', NOT_EQ, trackingId)

Mongo profiler logs indicated that using the NOT_EQ boolean was causing the indexing to fail and was slowing mongo lookup in the records in Hangfire, causing a huge processing queue. Comment the lines out and replace both lines with the “for loop” statements below:

for record in records: if record.tacking_Id == trackingId: continue if record['deduped Alert'] == 'YES': continue

3. Sleep_timer: Limiting the sleep times in the code should help the overall performance:

Update all sleep timer in the code to sleep for less than 3 seconds. The default may be (0, 3) which will indicate sleep between 0 and 3 seconds. Reduce all to (0.5, 1) 4. Restart task and API pods