Resolve SSDEEP Tasks in Hangfire

when your ssdeep tasks are taking longer to process (up to over 24 hours), and you receive errors like this one from the hangfire page proxyerror httpsconnectionpool(host='swimlane corpzone internalzone com', port=443) max retries exceeded with url /api/user/authorize (caused by proxyerror('cannot connect to proxy ', newconnectionerror('\<urllib3 connection httpsconnection object at 0x0000000003232c88> failed to establish a new connection \[errno 11001] getaddrinfo failed you may have also attempted to drop the hangfire jobgraph collection db hangfire jobgraph drop() but that still does not resolve the issue then you will need to modify the loopback time in the python integration script the loopback time is used to create a record filter then a “for loop” will run through the filter and process the data if there are thousands of records that come up in the search filter, this may cause the task to be stuck in a loop for hours, causing a huge queue the loopback time variable is set to 24 hours (1 day) by default set this to fewer hours, like between 1 and 12 hours then, comment out the lines for deduped email and tracking id lines in the fuzzymatch at times, depending on what you get from the logs, you may need to comment out the lines for not eq for deduped email and tracking id records filter('deduped email', not eq, 'yes') records filter('tracking id', not eq, trackingid) mongo profiler logs indicate that using the not eq boolean was causing the indexing to fail and was slowing mongo lookup in the records in hangfire, causing a huge processing queue comment the lines out and replace both lines with the “for loop” statements below for record in records if record tacking id == trackingid continue if record\['deduped alert'] == 'yes' continue limiting the sleep times in the code should help the overall performance, as well update all sleep timer in the code to sleep for less than 3 seconds the default may be (0, 3) which will indicate sleep between 0 and 3 seconds reduce all to (0 5, 1)