-
Notifications
You must be signed in to change notification settings - Fork 26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CIVIS-1508] DEP upgrade joblib dep #429
Conversation
joblib is now past 1.1.x and we should be able to install later version.
This is so a newer minor version cannot be downloaded on accident.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the contribution! I have a high-level question:
- Assuming joblib follows semantic versioning, the jump from joblib version
0.*
to1.*
could mean breaking changes from joblib. If there are any, would these changes affect civis-python in any way?
Also, it looks like you might have started this work before some of the most recent PRs got merged into master. Some of the no-longer-valid changes show up in your PR here and should be dropped.
I've taken the liberty to update this PR's title to link this work to the relevant JIRA ticket, and added Closes #423
to the PR description so that GitHub will close the issue once this PR is merged.
This is so this dependency needs to be updated less.
The breaking change is identified here. with joblib <1.x, pickling joblib.Memory objects was backwards compatible. Newer versions of numpy changed how numpy.dtypes was pickled which caused objects that should be the same to create different hashes. Essentially, if you're loading in joblib.Memory pickle objects, they are no longer compatible across versions (hence the major version bump). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for tracking down joblib's breaking changes. Looks like joblib 1.* (and numpy >= 1.20) have been out for about a year, which shouldn't cause issues for us.
LGTM!
Features
This PR upgrades the joblib dependency to allow versions 1.1.x which provides several enhancements and bug fixes around pickling which impacted previous runs.
Closes #423.