From b89f88c23099159f26347c2f70d5b15806ce35e0 Mon Sep 17 00:00:00 2001 From: "David L. Jones" Date: Fri, 18 Sep 2020 19:36:35 -0700 Subject: [PATCH] Remove uses of pkg_resources in non-namespace packages. In #713 and #1296, the `google` package in protobuf sources was found to cause conflicts with other Google projects, because it was not properly configured as a namespace package [1]. The initial fix in 786f80f addressed part of the issue, and #1298 fixed the rest. However, 786f80f (the initial fix) also made `google.protobuf` and `google.protobuf.pyext` into namespace packages. This was not correct: they are both regular, non-namespace, sub-subpackages. However (still), the follow-up #1298 did not nominate them as namespace packages, so the namespace registration behavior has remained, but without benefit. This change removes the unnecessary namespace registration, which has substantial overhead, thus reducing startup time substantially when using protobufs. Because this change affects the import internals, quantifying the overhead requires a full tear-down/start-up of the Python interpreter. So, to capture the full cost for every run, I measured the time to launching a _fresh_ Python instance in a subprocess, varying the imports and code under test. In other words, I used `timeit` to measure the time to launch a _fresh_ Python subprocess which actually performs the imports. * Reference: normal Python startup (i.e., don't import protobuf at all). ``` % python3 -m timeit -s 'import subprocess' -r 3 -n 10 'subprocess.call(["python3", "-c", "pass"])' 10 loops, best of 3: 27.1 msec per loop ``` * Baseline: cost to import `google.protobuf.descriptor`, with extraneous namespace packages. ``` % python3 -m timeit -s 'import subprocess' -r 3 -n 10 'subprocess.call(["python3", "-c", "import google.protobuf.descriptor"])' 10 loops, best of 3: 133 msec per loop ``` * This change: cost to import `google.protobuf.descriptor`, without extraneous namespace packages. ``` % python3 -m timeit -s 'import subprocess' -r 3 -n 10 'subprocess.call(["python3", "-c", "import google.protobuf.descriptor"])' 10 loops, best of 3: 43.1 msec per loop ``` [1]: https://packaging.python.org/guides/packaging-namespace-packages/ --- python/google/protobuf/__init__.py | 6 ------ python/google/protobuf/pyext/__init__.py | 4 ---- 2 files changed, 10 deletions(-) diff --git a/python/google/protobuf/__init__.py b/python/google/protobuf/__init__.py index 4e0d0e3906a3..512d64105a91 100644 --- a/python/google/protobuf/__init__.py +++ b/python/google/protobuf/__init__.py @@ -31,9 +31,3 @@ # Copyright 2007 Google Inc. All Rights Reserved. __version__ = '3.13.0' - -if __name__ != '__main__': - try: - __import__('pkg_resources').declare_namespace(__name__) - except ImportError: - __path__ = __import__('pkgutil').extend_path(__path__, __name__) diff --git a/python/google/protobuf/pyext/__init__.py b/python/google/protobuf/pyext/__init__.py index 558561412299..e69de29bb2d1 100644 --- a/python/google/protobuf/pyext/__init__.py +++ b/python/google/protobuf/pyext/__init__.py @@ -1,4 +0,0 @@ -try: - __import__('pkg_resources').declare_namespace(__name__) -except ImportError: - __path__ = __import__('pkgutil').extend_path(__path__, __name__)