Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FIX python-lz4 version comparison #847

Merged
merged 4 commits into from May 20, 2019

Conversation

sveitser
Copy link
Contributor

@sveitser sveitser commented Feb 16, 2019

The python-lz4 package has its version string prefixed with "v". As a result on nixos a test fails.

This commit changes to pkg_resources.parse_version which normalizes
the two schemas correctly and seems to be the recommended way to
compare versions in python at the moment.

Nixos build log https://hydra.nixos.org/build/88820470/nixlog/2.

Failing test:


=================================== FAILURES ===================================
_____________________________ test_lz4_compression _____________________________

tmpdir = local('/build/pytest-of-nixbld/pytest-0/test_lz4_compression0')

    @with_lz4
    def test_lz4_compression(tmpdir):
        # Check that lz4 can be used when dependency is available.
        import lz4.frame
        compressor = 'lz4'
        assert compressor in _COMPRESSORS
        assert _COMPRESSORS[compressor].fileobj_factory == lz4.frame.LZ4FrameFile
    
        fname = tmpdir.join('test.pkl').strpath
        data = 'test data'
>       numpy_pickle.dump(data, fname, compress=compressor)

joblib/test/test_numpy_pickle.py:1047: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
joblib/numpy_pickle.py:498: in dump
    compress_level)) as f:
joblib/numpy_pickle_utils.py:192: in _write_fileobject
    filename, compresslevel=compresslevel)
joblib/compressor.py:234: in compressor_file
    self._check_versions()
joblib/compressor.py:229: in _check_versions
    if lz4 is None or LooseVersion(lz4.__version__) < LooseVersion('0.19'):
/nix/store/6la3fblc497pacyy0vxhnjq1l1wlb5w7-python3-3.7.2/lib/python3.7/distutils/version.py:52: in __lt__
    c = self._cmp(other)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = LooseVersion ('v2.1.2'), other = LooseVersion ('0.19')

    def _cmp (self, other):
        if isinstance(other, str):
            other = LooseVersion(other)
    
        if self.version == other.version:
            return 0
>       if self.version < other.version:
E       TypeError: '<' not supported between instances of 'str' and 'int'

The python-lz4 package has its version string prefixed with "v".
As a result on nixos a test fails.

This commit changes to pkg_resources.parse_version which normalizes
the two schemas correctly and seems to be the recommended way to
compare versions in python at the moment.

Nixos build log https://hydra.nixos.org/build/88820470/nixlog/2.
@codecov
Copy link

codecov bot commented Feb 16, 2019

Codecov Report

Merging #847 into master will decrease coverage by 1.91%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #847      +/-   ##
==========================================
- Coverage   94.55%   92.63%   -1.92%     
==========================================
  Files          45       45              
  Lines        6407     6407              
==========================================
- Hits         6058     5935     -123     
- Misses        349      472     +123
Impacted Files Coverage Δ
joblib/compressor.py 90.93% <100%> (-2.59%) ⬇️
joblib/backports.py 93.75% <100%> (+54.16%) ⬆️
joblib/numpy_pickle_compat.py 49% <0%> (-42%) ⬇️
joblib/test/test_numpy_pickle.py 88.5% <0%> (-9.86%) ⬇️
joblib/testing.py 87.5% <0%> (-7.5%) ⬇️
joblib/_dask.py 91.52% <0%> (-3.96%) ⬇️
joblib/numpy_pickle.py 94.58% <0%> (-3.95%) ⬇️
joblib/_multiprocessing_helpers.py 76.92% <0%> (-3.85%) ⬇️
joblib/numpy_pickle_utils.py 90.32% <0%> (-3.23%) ⬇️
... and 15 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a99273b...884c92c. Read the comment docs.

@codecov
Copy link

codecov bot commented Feb 16, 2019

Codecov Report

Merging #847 into master will increase coverage by 11.2%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #847      +/-   ##
==========================================
+ Coverage   84.15%   95.36%   +11.2%     
==========================================
  Files          45       45              
  Lines        6406     6407       +1     
==========================================
+ Hits         5391     6110     +719     
+ Misses       1015      297     -718
Impacted Files Coverage Δ
joblib/compressor.py 93.52% <100%> (+11%) ⬆️
joblib/backports.py 93.75% <100%> (+58.33%) ⬆️
joblib/test/test_memmapping.py 99.41% <0%> (+0.29%) ⬆️
joblib/_parallel_backends.py 93.6% <0%> (+0.79%) ⬆️
joblib/memory.py 95.69% <0%> (+1.34%) ⬆️
joblib/test/test_hashing.py 98.88% <0%> (+1.67%) ⬆️
joblib/logger.py 86.84% <0%> (+2.63%) ⬆️
joblib/parallel.py 97.28% <0%> (+2.71%) ⬆️
joblib/_memmapping_reducer.py 95% <0%> (+2.77%) ⬆️
joblib/my_exceptions.py 98.11% <0%> (+3.77%) ⬆️
... and 22 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3169ca6...6dc4a7c. Read the comment docs.

@FlorianFranzen
Copy link

Would be nice if we can get this merged.

@aabadie
Copy link
Contributor

aabadie commented Mar 18, 2019

LGTM

@ogrisel any idea why the scikit-learn tests are failing ? I tried with 0.20.3 on my fork but have another failing test, see https://travis-ci.org/aabadie/joblib/jobs/507830463 and aabadie@17b3ee3
This is also broken on master btw.

@ogrisel
Copy link
Contributor

ogrisel commented May 20, 2019

I merged @lesteve's #861 to master and merged master into this branch to check whether that fixes the problem.

@ogrisel
Copy link
Contributor

ogrisel commented May 20, 2019

The windows error is unrelated, let's merge.

@ogrisel ogrisel merged commit efe896a into joblib:master May 20, 2019
pierreglaser pushed a commit to pierreglaser/joblib that referenced this pull request Jun 12, 2019
@cgohlke cgohlke mentioned this pull request Dec 4, 2019
ogrisel added a commit to ogrisel/joblib that referenced this pull request Dec 10, 2019
ogrisel added a commit that referenced this pull request Dec 10, 2019
* Revert "FIX python-lz4 version comparison (#847)"

This reverts commit efe896a.

* Manually deal with LZ4 version issues

* Changelog update

* Ignore vscode configuration
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Dec 14, 2019
Release 0.14.1

Configure the loky workers' environment to mitigate oversubsription with nested multi-threaded code in the following case:

allow for a suitable number of threads for numba (NUMBA_NUM_THREADS);
enable Interprocess Communication for scheduler coordination when the nested code uses Threading Building Blocks (TBB) (ENABLE_IPC=1)
joblib/joblib#951

Fix a regression where the loky backend was not reusing previously spawned workers. joblib/joblib#968

Revert joblib/joblib#847 to avoid using pkg_resources that introduced a performance regression under Windows: joblib/joblib#965
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants