Eliminating Orphan Workers

[npgoossen]

Issue #1602 mentions that if a worker crashes with a work horse still running, the work horse process, when the job finishes will attempt to update stats.
By default Redis creates keys with no expiration, resulting in a new worker key being created that makes it so you can't re-register a worker with the same key. The worker is also in an odd state, neither alive or dead.

Mentioned in the issue is a suggesting to optimize this to avoid an extra Redis call by checking if the value was set to 1, but I'm not sure this will work as that will also catch the situation of the first job a worker runs, resulting in the number never being incremented.

[selwin]

Mentioned in the issue is a suggesting to optimize this to avoid an extra Redis call by checking if the value was set to 1, but I'm not sure this will work as that will also catch the situation of the first job a worker runs, resulting in the number never being incremented.

I'm not sure this will work because if connection.exists(self.key): will be pipelined. I have recently changed my way of thinking, I think we should simply set a TTL to the worker key after the increment calls are successful. This way, no worker keys would be orphaned forever.

Mind making this change?

[npgoossen]

@selwin - Thanks for the comment/review. Changed it to set the expiration when these values get set. Shouldn't have any adverse affects on normal operation as the heartbeat keeps increasing it.

[selwin]

@npgoossen can you check the failing tests? They all need to pass before I can merge this in.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (8965b4c) 93.84% compared to head (5a3a8ce) 93.84%.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1993   +/-   ##
=======================================
  Coverage   93.84%   93.84%           
=======================================
  Files          29       29           
  Lines        3897     3900    +3     
=======================================
+ Hits         3657     3660    +3     
  Misses        240      240           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.