Add new deferred_ttl job attribute and cleanup for deferred jobs

[raymondguo-db]

Currently there is no cleanup for deferred jobs in the DeferredJobRegistry. The only way for jobs to leave the DeferredJobRegistry is if the jobs' dependencies succeed. If dependencies fail or never execute, then deferred jobs are not cleaned up.

In my use case, we saw that our DeferredJobRegistry was getting too large, hitting our Redis instance limits. Our workaround so far for this was to make a periodic job that clears the DeferredJobRegistry.

This PR simplifies this process by adding an optional deferred_ttl attribute to deferred jobs. By default, the value for this is +inf (the current behavior, so existing workflows aren't affected). deferred_ttl in effect is a time limit for deferred jobs before they stop waiting and are considered failed, cleaned up by the DeferredJobRegistry.

[raymondguo-db]

cc @rauchy

[selwin]

Hey, this is a good idea, but I'd like this to be implemented via the newly added Dependency class. This is how the API should look like:

queue.enqueue(foo, depends_on=Dependency(bar, ttl=90))

Mind making this change? I think the change shouldn't be too big and it makes the API cleaner by grouping related args together.

[selwin]

A reminder to make this change when you're free :).