Use importlib to load numba extensions #5209

svrakitin · 2020-02-06T23:31:53Z

esc · 2020-02-07T09:26:27Z

@svrakitin thanks for submitting this and congrats on submitting your first PR to Numba. I have queued it for review now.

esc

LGMT just two small remarks.

buildscripts/condarecipe.local/meta.yaml

svrakitin · 2020-02-07T12:52:14Z

This one adds importlib_metadata as dependency for python versions < 3.8.

stuartarchibald · 2020-02-07T17:22:12Z

numba/core/entrypoints.py

+try:
+    from importlib import metadata as importlib_metadata
+except ImportError:
+    import importlib_metadata


What if this fails? I'm wondering if there ought to be an explicit Python>=3.8 branch which just imports it and then another which tries and fails gracefully if the package is missing.

It will only fail on Python < 3.8 if importlib_metadata is missing (it is listed as requirement).

We tend to assume nothing about how Numba was built, it's entirely possible someone just grabbed the source and some limited set of bootstrap packages available for their unusual architecture and built it (this actually happens in practice, we see these people on the issue tracker when strange things happen!). As a result, we tend to err on the side of trying to be as helpful as possible and catch things like this and say something about it being needed.

Further, this will need adding to the dependencies section in the docs: https://github.com/numba/numba/blob/13ece9b97e6f01f750e870347f231282325f60c3/docs/source/user/installing.rst#dependency-list

stuartarchibald · 2020-02-07T17:26:41Z

Note to reviewers: Someone needs to go and manually test this for Python 3.7 and 3.8 against numba-scipy to make sure it works fine in practice.

esc

I think the new dependency should still be added to http://numba.pydata.org/numba-doc/latest/user/installing.html#dependency-list and also potentially setup.py. Looks good beyond that.

esc · 2020-02-10T16:32:28Z

BTW: how hard is this dependency?

svrakitin · 2020-02-10T18:12:58Z

@esc It is non-optional run time dependency.

esc · 2020-02-11T10:23:24Z

@esc It is non-optional run time dependency.

In that case, I believe it should be added to setup.py too. The only other two dependencies that are listed there are llvmlite and numpy.

esc · 2020-02-11T10:24:20Z

Actually setuptools is in there too.

esc · 2020-02-11T14:21:07Z

@svrakitin thanks for submitting the fixes, I have now marked it as waiting for reviewer input. @stuartarchibald @sklam what say you?

stuartarchibald

Thanks for this. There's a couple of tiny things to resolve (we should use conda packages ahead of pip!).

Otherwise, this PR seems technically fine and gets approval from me. More generally the core developers will need to agree on accepting importlib_metadata as a hard dependency but I do not anticipate this being a problem.

@esc any chance you could please test run this with numba-scipy to make sure it works ok in practice, thanks.

stuartarchibald · 2020-02-12T12:43:25Z

buildscripts/incremental/setup_conda_environment.cmd

@@ -32,6 +32,7 @@ call activate %CONDA_ENV%
 if %PYTHON% LSS 3.4 (%CONDA_INSTALL% enum34)
 if %PYTHON% LSS 3.4 (%PIP_INSTALL% singledispatch)
 if %PYTHON% LSS 3.3 (%CONDA_INSTALL% -c numba funcsigs)
+if %PYTHON% LSS 3.8 (%PIP_INSTALL% importlib-metadata)


Perhaps get this from the Anaconda distro?!

stuartarchibald · 2020-02-12T12:43:28Z

buildscripts/incremental/setup_conda_environment.sh

@@ -80,6 +80,8 @@ $CONDA_INSTALL -c numba llvmlite
 # Install enum34 and singledispatch for Python < 3.4
 if [ $PYTHON \< "3.4" ]; then $CONDA_INSTALL enum34; fi
 if [ $PYTHON \< "3.4" ]; then $PIP_INSTALL singledispatch; fi
+# Install importlib-metadata for Python < 3.8
+if [ $PYTHON \< "3.8" ]; then $PIP_INSTALL importlib-metadata; fi


Perhaps get this from the Anaconda distro?!

stuartarchibald

Thanks for the fixes. Code changes approved. This now needs manual and build farm testing.

stuartarchibald · 2022-01-25T10:49:24Z

buildscripts/incremental/setup_conda_environment.sh

@@ -98,6 +98,8 @@ fi
 # Install latest llvmlite build
 $CONDA_INSTALL -c numba/label/dev llvmlite

+# Install importlib-metadata for Python < 3.8


Suggested change

# Install importlib-metadata for Python < 3.8

# Install importlib-metadata for Python < 3.9

stuartarchibald

Thanks for updating and testing this @gmarkall . Few minor things to look at else I think this is ready for testing on the farm. Thanks again!

stuartarchibald · 2022-01-25T11:09:08Z

buildscripts/condarecipe.local/meta.yaml

    # On channel https://anaconda.org/numba/
+    - importlib_metadata       # [py<39]


Suggested change

# On channel https://anaconda.org/numba/

- importlib_metadata # [py<39]

- importlib_metadata # [py<39]

# On channel https://anaconda.org/numba/

looks like this comment is now on the incorrect line.

stuartarchibald · 2022-01-25T11:09:59Z

buildscripts/condarecipe.local/meta.yaml

    # On channel https://anaconda.org/numba/
    - llvmlite >=0.39.0dev0,<0.39
+    - importlib_metadata       # [py<39]


Suggested change

# On channel https://anaconda.org/numba/

- llvmlite >=0.39.0dev0,<0.39

- importlib_metadata # [py<39]

- importlib_metadata # [py<39]

# On channel https://anaconda.org/numba/

- llvmlite >=0.39.0dev0,<0.39

stuartarchibald · 2022-01-25T11:14:04Z

numba/tests/test_entrypoints.py

-            self.assertEqual(counters['init'], 1)
+                # was our init function called?
+                mod.init_func.assert_called_once()


Seems like a similar change can also be made to test_entrypoint_tolerance as part of this clean up.

Similar change made - note this also involves configuring the mock with a side effect for init_func().

This also removes the Python 2-specific workaround :-)

gmarkall · 2022-01-26T10:31:24Z

@stuartarchibald Many thanks for the reviews - the most recent comments should now be addressed.

stuartarchibald · 2022-01-26T10:52:49Z

@stuartarchibald Many thanks for the reviews - the most recent comments should now be addressed.

Thanks @gmarkall, the updates look good. If you are happy to:

approve 724990f which is derived from my patch here Use importlib to load numba extensions #5209 (comment)
run any CUDA testing you think necessary

Then I'm happy to approve this patch. Many thanks!

gmarkall

Approved re 724990f which is authored by @stuartarchibald.

gmarkall · 2022-01-26T11:49:49Z

gpuci run tests.

stuartarchibald · 2022-01-26T11:52:31Z

Approved re 724990f which is authored by @stuartarchibald.

Thanks @gmarkall

stuartarchibald

Thanks for all the effort on this! Looks good.

stuartarchibald · 2022-02-07T13:19:29Z

Buildfarm ID: numba_smoketest_cpu_yaml_74.

stuartarchibald · 2022-02-07T14:54:41Z

Buildfarm ID: numba_smoketest_cpu_yaml_74.

Passed.

gmarkall · 2022-02-08T11:10:45Z

@svrakitin Many thanks for your efforts here! 🎉

svrakitin · 2022-02-08T11:30:10Z

@gmarkall Hah, thanks for getting this to the end. :)

flying-sheep · 2022-08-18T11:01:27Z

Actually setuptools is in there too.

Good catch @esc, it should have been removed from there in this PR.

Follow-up PR to close the deal: #8366

Stepan Rakitin added 3 commits February 7, 2020 00:29

Use importlib to load numba extensions

2b672f2

Optimize imports

bedf8da

Add missing deps on importlib-metadata backport

4999274

esc added the 3 - Ready for Review label Feb 7, 2020

stuartarchibald added the Pending BuildFarm For PRs that have been reviewed but pending a push through our buildfarm label Feb 7, 2020

stuartarchibald added this to the Numba 0.49 RC milestone Feb 7, 2020

esc requested changes Feb 7, 2020

View reviewed changes

buildscripts/condarecipe.local/meta.yaml Outdated Show resolved Hide resolved

buildscripts/condarecipe.local/meta.yaml Outdated Show resolved Hide resolved

Fix selector and backport name in conda recipe

22f4f04

svrakitin requested a review from esc February 7, 2020 15:26

stuartarchibald reviewed Feb 7, 2020

View reviewed changes

Add more meaningful message for missing importlib_metadata

22e91be

esc requested changes Feb 10, 2020

View reviewed changes

Update run time dependencies in installing.rst

8f3c285

Add importlib_metadata dep to install_requires in setup.py

760ae11

esc added 4 - Waiting on reviewer Waiting for reviewer to respond to author and removed 3 - Ready for Review labels Feb 11, 2020

esc previously approved these changes Feb 11, 2020

View reviewed changes

stuartarchibald reviewed Feb 12, 2020

View reviewed changes

Install importlib_metadata using conda in build scripts

c3d6990

svrakitin requested a review from stuartarchibald February 12, 2020 15:13

stuartarchibald previously approved these changes Feb 12, 2020

View reviewed changes

stuartarchibald removed the 4 - Waiting on reviewer Waiting for reviewer to respond to author label Feb 12, 2020

gmarkall added the 4 - Waiting on reviewer Waiting for reviewer to respond to author label Jan 25, 2022

stuartarchibald reviewed Jan 25, 2022

View reviewed changes

stuartarchibald added the Effort - long Long size effort needed label Jan 25, 2022

gmarkall added 4 commits January 26, 2022 09:21

Merge remote-tracking branch 'numba/master' into fix/4927-slow-import

fbd66c3

Fix comments in build scripts (PR numba#5209 feedback)

b5f75f6

Use a mock for test_entrypoint_tolerance (PR numba#5209 feedback)

59fa60d

This also removes the Python 2-specific workaround :-)

Move comment in build script

4456c37

gmarkall approved these changes Jan 26, 2022

View reviewed changes

stuartarchibald added 4 - Waiting on CI Review etc done, waiting for CI to finish Pending BuildFarm For PRs that have been reviewed but pending a push through our buildfarm and removed 4 - Waiting on reviewer Waiting for reviewer to respond to author labels Jan 26, 2022

gmarkall mentioned this pull request Feb 5, 2022

CUDA Dispatcher refactor 2: inherit from dispatcher.Dispatcher #7815

Merged

2 tasks

stuartarchibald approved these changes Feb 7, 2022

View reviewed changes

sklam merged commit fa5c9e1 into numba:main Feb 7, 2022

svrakitin deleted the fix/4927-slow-import branch February 7, 2022 23:54

flying-sheep mentioned this pull request Aug 18, 2022

Remove setuptools as runtime dependency #8366

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use importlib to load numba extensions #5209

Use importlib to load numba extensions #5209

svrakitin commented Feb 6, 2020

esc commented Feb 7, 2020

esc left a comment

svrakitin commented Feb 7, 2020

stuartarchibald Feb 7, 2020

svrakitin Feb 7, 2020

stuartarchibald Feb 7, 2020

stuartarchibald commented Feb 7, 2020

esc left a comment

esc commented Feb 10, 2020

svrakitin commented Feb 10, 2020

esc commented Feb 11, 2020

esc commented Feb 11, 2020

esc commented Feb 11, 2020

stuartarchibald left a comment

stuartarchibald Feb 12, 2020

stuartarchibald Feb 12, 2020

stuartarchibald left a comment

stuartarchibald Jan 25, 2022

stuartarchibald left a comment

stuartarchibald Jan 25, 2022

stuartarchibald Jan 25, 2022

stuartarchibald Jan 25, 2022

gmarkall Jan 26, 2022

gmarkall commented Jan 26, 2022

stuartarchibald commented Jan 26, 2022 •

edited

gmarkall left a comment

gmarkall commented Jan 26, 2022

stuartarchibald commented Jan 26, 2022

stuartarchibald left a comment

stuartarchibald commented Feb 7, 2022

stuartarchibald commented Feb 7, 2022

gmarkall commented Feb 8, 2022

svrakitin commented Feb 8, 2022

flying-sheep commented Aug 18, 2022

	# Install importlib-metadata for Python < 3.8
	# Install importlib-metadata for Python < 3.9

		# On channel https://anaconda.org/numba/
		- importlib_metadata # [py<39]

Use importlib to load numba extensions #5209

Use importlib to load numba extensions #5209

Conversation

svrakitin commented Feb 6, 2020

esc commented Feb 7, 2020

esc left a comment

Choose a reason for hiding this comment

svrakitin commented Feb 7, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuartarchibald commented Feb 7, 2020

esc left a comment

Choose a reason for hiding this comment

esc commented Feb 10, 2020

svrakitin commented Feb 10, 2020

esc commented Feb 11, 2020

esc commented Feb 11, 2020

esc commented Feb 11, 2020

stuartarchibald left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuartarchibald left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuartarchibald left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gmarkall commented Jan 26, 2022

stuartarchibald commented Jan 26, 2022 • edited

gmarkall left a comment

Choose a reason for hiding this comment

gmarkall commented Jan 26, 2022

stuartarchibald commented Jan 26, 2022

stuartarchibald left a comment

Choose a reason for hiding this comment

stuartarchibald commented Feb 7, 2022

stuartarchibald commented Feb 7, 2022

gmarkall commented Feb 8, 2022

svrakitin commented Feb 8, 2022

flying-sheep commented Aug 18, 2022

stuartarchibald commented Jan 26, 2022 •

edited