FEA parallel_config context manager to allow more fine-grained control

[jeremiedbb]

Fixes #912
Fixes #979
related to #1376

adds a new context_manager parallel_config that should be used in place of parallel_backend. It exposes more parameters and does not enforce setting a backend.

It allows to do something like:

from joblib.parallel import parallel_config
from joblib.memory import Memory
from joblib import Parallel, delayed
 
with parallel_config(prefer="threads", n_jobs=3):
    res = Parallel(verbose=100)(delayed(lambda :1)() for _ in range(10))
 
# [Parallel(n_jobs=3)]: Using backend ThreadingBackend with 3 concurrent workers.
# [Parallel(n_jobs=3)]: Done   1 tasks      | elapsed:    0.0s
# [Parallel(n_jobs=3)]: Done   2 tasks      | elapsed:    0.0s
# [Parallel(n_jobs=3)]: Done   3 tasks      | elapsed:    0.0s
# [Parallel(n_jobs=3)]: Done   4 tasks      | elapsed:    0.0s
# [Parallel(n_jobs=3)]: Done   5 tasks      | elapsed:    0.0s
# [Parallel(n_jobs=3)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s
# [Parallel(n_jobs=3)]: Done   7 out of  10 | elapsed:    0.0s remaining:    0.0s
# [Parallel(n_jobs=3)]: Done   8 out of  10 | elapsed:    0.0s remaining:    0.0s
# [Parallel(n_jobs=3)]: Done  10 out of  10 | elapsed:    0.0s remaining:    0.0s
# [Parallel(n_jobs=3)]: Done  10 out of  10 | elapsed:    0.0s finished

It's using the threading backend instead of the default loky, with 3 jobs instead of the default 1.

The config parameters of the context are used only if they are not explicitely set in Parallel. It makes us forced to use sentinels for the default values, adding a little bit of complexity.

TODO

• raise an error if backend constructor params are set but backend is left to None
• add tests

[codecov]

Codecov Report

Patch coverage: 97.12% and project coverage change: -0.03 ⚠️

Comparison is base (41b70ff) 94.78% compared to head (4aa3509) 94.75%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1392      +/-   ##
==========================================
- Coverage   94.78%   94.75%   -0.03%     
==========================================
  Files          44       44              
  Lines        7207     7306      +99     
==========================================
+ Hits         6831     6923      +92     
- Misses        376      383       +7     

Impacted Files	Coverage Δ
joblib/_utils.py	95.45% <83.33%> (-4.55%)	⬇️
joblib/test/test_parallel.py	96.20% <97.34%> (+0.12%)	⬆️
joblib/parallel.py	97.24% <97.75%> (+0.17%)	⬆️
joblib/__init__.py	100.00% <100.00%> (ø)

... and 1 file with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[tomMoral]

From a quick scan, the setup looks nice :)
Just a few first comments.

[jjerphan]

To me, this is a valuable contribution for users. Thank you, @jeremiedbb.

Here is a review from an outsider (I am starting getting around joblib).

[ogrisel]

@jeremiedbb There are a few conflicts to resolve after the merge of the return_generator branch but it does not look too complicated.

[tomMoral]

I did a pass but the code looks super nice!! Good job @jeremiedbb :)

[tomMoral]

LGTM! thanks @jeremiedbb for making this :)

[trendelkampschroer]

Thanks a lot for this feature.