Config Object

[rpkak]

This PR adds a config object as a context manager.

The user can update the config by doing:

from rq.config import Config

with Config(worker_class='rq.SimpleWorker', serializer_class='rq.serializers.JSONSerializer'):
    # Do things where the default worker class is SimpleWorker and the serializer is json

In RQ this data can be accessed using:

from rq.config import get_configuration

serializer_class = get_configuration('serializer_class')

closes #1548
closes #1417

[codecov]

Codecov Report

Merging #1552 (00e69e1) into master (e71fcb9) will decrease coverage by 0.20%.
The diff coverage is 96.25%.

❗ Current head 00e69e1 differs from pull request most recent head 8c382cf. Consider uploading reports for the commit 8c382cf to get more accurate results

@@            Coverage Diff             @@
##           master    #1552      +/-   ##
==========================================
- Coverage   95.59%   95.39%   -0.21%     
==========================================
  Files          46       46              
  Lines        7061     6902     -159     
==========================================
- Hits         6750     6584     -166     
- Misses        311      318       +7     

Impacted Files	Coverage Δ
rq/cli/helpers.py	87.50% <75.00%> (-1.10%)	⬇️
rq/config.py	82.75% <82.75%> (ø)
rq/scheduler.py	96.10% <87.50%> (-0.15%)	⬇️
tests/test_scheduler.py	94.97% <92.68%> (-0.42%)	⬇️
rq/registry.py	97.10% <96.96%> (-0.12%)	⬇️
tests/test_worker.py	97.51% <97.05%> (-0.01%)	⬇️
rq/cli/cli.py	92.96% <97.61%> (-0.14%)	⬇️
rq/worker.py	88.39% <97.91%> (-0.26%)	⬇️
rq/job.py	98.00% <98.36%> (-0.19%)	⬇️
rq/__init__.py	100.00% <100.00%> (ø)
... and 27 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e71fcb9...8c382cf. Read the comment docs.

[selwin]

Implementation wise, this config object shouldn't be accessed as though this is a global or thread local object like

from rq.config import get_configuration
serializer_class = get_configuration('serializer_class')

I have plans to deprecate RQ's usage of thread locals. So this config object should be explicitly passed to Worker, Queue, Job or Scheduler. So it should be something like this:

worker = Worker(config=config)
worker.config.get('serializer_class')

[rpkak]

I have plans to deprecate RQ's usage of thread locals. So this config object should be explicitly passed to Worker, Queue, Job or Scheduler. So it should be something like this:

worker = Worker(config=config)
worker.config.get('serializer_class')

The problem of this is that we always need to use the config as an arg and so we could forget to implement it (For Example in functions of this project which are called from multiple other functions that don't need config now but in the future).

That's because I used context manager and a global config stack to make the config accessable from everywhere.

So @selwin can you just write me why you don't like globals?

[selwin]

The problem of this is that we always need to use the config as an arg and so we could forget to implement it (For Example in functions of this project which are called from multiple other functions that don't need config now but in the future).

Correct, but the goal would be to put everything, including connection information into this config object.

So @selwin can you just write me why you don't like globals?

I like it when everything is passed around so there's no ambiguity. There are many threads discussing why globals shouldn't be used, this is simply one of them https://stackoverflow.com/questions/19158339/why-are-global-variables-evil

To me personally it's easier to maintain a code base where everything is mostly passed from one function to another. No surprises.

[rpkak]

Ok @selwin it's used as an arg, that is passed now and I implemented the cli.

It is very ugly yet in the cli because there is no possibility to pass the config yet.
That will get better when I implement worker, queue and job.

[selwin]

I'd like to have a more explicitly define config object so we can have an API like this:

config = get_config()
config.serializer_class
config.connection

[selwin]

The most important thing about this PR is that it breaks backwards compatibility. I'm not willing to make a backwards incompatible change without advanced warning.

This means that config should be an additional kwarg on top of currently existing ones. We can raise a DeprecationWarning for arguments that will be deprecated in version 2.X, but the actual deprecation will happen after the warning has been in place for a few years.

[selwin]

I don't think we can get rid of connection for these reasons:

1. Storage must be explicitly defined
2. I want users to be able to easily use RQ with default options without having to worry about config

So the minimum way to define a queue would be queue = Queue(connection=redis).

[rpkak]

I don't know why this text fails. send_stop_job_command works on when I tried it, but not in the test. @selwin Do you know why this hasn't worked.

[selwin]

@rpkak I'm not sure. Does the test work on your local machine?

[rpkak]

@rpkak I'm not sure. Does the test work on your local machine?

No, it doesn't, but during manual testing, it worked fine.

[ccrvlh]

Hey @rpkak did you end up doing some more work on this?
I got here while thinking about namespaces (eg myns:job instead of rq:job).

His has appeared in a couple of issues around here. A very trivial implementation for namespaces would be to read an envvar RQ_NAMESPACE and then make all the hardcoded namespaces read from a DEFAULT_NAMESPACE = os.getenv("RQ_NAMESPACE", "rq"). I run a couple of tests and this works, but would allow for an inconsistent state (the worker is running without the envvar for example).

I was though of having a config being stored within the datastore in rq:config, which would allow for a centralized configuration and all resources could read it. So the config object being passed to the queue could add data to rq:config.