Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Further optimize distribution() searching. #272

Merged
merged 3 commits into from
Jan 10, 2021
Merged

Conversation

anntzer
Copy link
Contributor

@anntzer anntzer commented Jan 6, 2021

Locally, this speeds up distribution("ipython") on a large-ish
environment by nearly 2-fold. (The GHA benchmark goes from ~1.6ms
to ~0.8ms.)

All the following changes matter for the performance gain:

  • avoid needlessly reinstanciating Prepareds.
  • switch from os.path.splitext (very slow) to str.rpartition
  • avoid having to add back a dot to the ext part given by rpartition
  • precompute egg_prefix and versionless_egg_name

(See https://gitlab.com/python-devs/importlib_metadata/-/issues/95 for
rationale why performance matters.)

@jaraco
Copy link
Member

jaraco commented Jan 10, 2021

Thanks for this. It's excellent. The only real wish I have (for future consideration) is if these optimizations had been committed and pushed separately, we could see the impact of each change independently.

Any idea why the Python 3.6 tests are timing out?

@jaraco
Copy link
Member

jaraco commented Jan 10, 2021

Any idea why the Python 3.6 tests are timing out?

Oh, dear. I see the Python 3.6 tests have been timing out for some time, although apparently only intermittently. I'll bet it's some bug with pips resolver and selective dependencies on self (importlib_metadata). I'll deal with that in another issue/pr.

@anntzer
Copy link
Contributor Author

anntzer commented Jan 10, 2021

if these optimizations had been committed and pushed separately, we could see the impact of each change independently.

Done.

@jaraco jaraco closed this Jan 10, 2021
@jaraco jaraco reopened this Jan 10, 2021
@jaraco
Copy link
Member

jaraco commented Jan 10, 2021

Looks like the second and third commits had the most impact:

importlib_metadata HEAD $ git checkout -q c1af1d6~1; tox -q -e perf
200 loops, best of 5: 1.47 msec per loop
________________________________________________________________ summary ________________________________________________________________
  perf: commands succeeded
  congratulations :)

importlib_metadata HEAD $ git checkout -q c1af1d6; tox -q -e perf
200 loops, best of 5: 1.45 msec per loop
________________________________________________________________ summary ________________________________________________________________
  perf: commands succeeded
  congratulations :)

importlib_metadata HEAD $ git checkout -q c9c0909; tox -q -e perf
200 loops, best of 5: 1.14 msec per loop
________________________________________________________________ summary ________________________________________________________________
  perf: commands succeeded
  congratulations :)

importlib_metadata HEAD $ git checkout -q 7fc4f11; tox -q -e perf
200 loops, best of 5: 928 usec per loop
________________________________________________________________ summary ________________________________________________________________
  perf: commands succeeded
  congratulations :)

@jaraco jaraco closed this Jan 10, 2021
@jaraco jaraco reopened this Jan 10, 2021
@jaraco jaraco merged commit c93a9f7 into python:main Jan 10, 2021
@anntzer anntzer deleted the moreperf branch January 10, 2021 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants