Skip to content

Commit

Permalink
Remove uses of pkg_resources in non-namespace packages. (#7902)
Browse files Browse the repository at this point in the history
In #713 and #1296, the `google` package in protobuf sources was found
to cause conflicts with other Google projects, because it was not
properly configured as a namespace package [1]. The initial fix in
786f80f addressed part of the issue, and #1298 fixed the rest.

However, 786f80f (the initial fix) also made `google.protobuf` and
`google.protobuf.pyext` into namespace packages. This was not correct:
they are both regular, non-namespace, sub-subpackages.

However (still), the follow-up #1298 did not nominate them as
namespace packages, so the namespace registration behavior has
remained, but without benefit.

This change removes the unnecessary namespace registration, which has
substantial overhead, thus reducing startup time substantially when
using protobufs.

Because this change affects the import internals, quantifying the
overhead requires a full tear-down/start-up of the Python interpreter.
So, to capture the full cost for every run, I measured the time to
launching a _fresh_ Python instance in a subprocess, varying the
imports and code under test. In other words, I used `timeit` to
measure the time to launch a _fresh_ Python subprocess which actually
performs the imports.

* Reference: normal Python startup (i.e., don't import protobuf at all).
  ```
   % python3 -m timeit -s 'import subprocess' -r 3 -n 10 'subprocess.call(["python3", "-c", "pass"])'
  10 loops, best of 3: 27.1 msec per loop
  ```

* Baseline: cost to import `google.protobuf.descriptor`, with
  extraneous namespace packages.
  ```
  % python3 -m timeit -s 'import subprocess' -r 3 -n 10 'subprocess.call(["python3", "-c", "import google.protobuf.descriptor"])'
  10 loops, best of 3: 133 msec per loop
  ```

* This change: cost to import `google.protobuf.descriptor`, without
  extraneous namespace packages.
  ```
  % python3 -m timeit -s 'import subprocess' -r 3 -n 10 'subprocess.call(["python3", "-c", "import google.protobuf.descriptor"])'
  10 loops, best of 3: 43.1 msec per loop
  ```

[1]:  https://packaging.python.org/guides/packaging-namespace-packages/
  • Loading branch information
dlj-NaN committed Sep 23, 2020
1 parent cae85a9 commit 66e3562
Show file tree
Hide file tree
Showing 2 changed files with 0 additions and 10 deletions.
6 changes: 0 additions & 6 deletions python/google/protobuf/__init__.py
Expand Up @@ -31,9 +31,3 @@
# Copyright 2007 Google Inc. All Rights Reserved.

__version__ = '3.13.0'

if __name__ != '__main__':
try:
__import__('pkg_resources').declare_namespace(__name__)
except ImportError:
__path__ = __import__('pkgutil').extend_path(__path__, __name__)
4 changes: 0 additions & 4 deletions python/google/protobuf/pyext/__init__.py
@@ -1,4 +0,0 @@
try:
__import__('pkg_resources').declare_namespace(__name__)
except ImportError:
__path__ = __import__('pkgutil').extend_path(__path__, __name__)

0 comments on commit 66e3562

Please sign in to comment.