Commits on Sep 21, 2020

1. Remove uses of pkg_resources in non-namespace packages. …

   In protocolbuffers#713 and protocolbuffers#1296, the `google` package in protobuf sources was found
   to cause conflicts with other Google projects, because it was not
   properly configured as a namespace package [1]. The initial fix in
   786f80f addressed part of the issue, and protocolbuffers#1298 fixed the rest.
   
   However, 786f80f (the initial fix) also made `google.protobuf` and
   `google.protobuf.pyext` into namespace packages. This was not correct:
   they are both regular, non-namespace, sub-subpackages.
   
   However (still), the follow-up protocolbuffers#1298 did not nominate them as
   namespace packages, so the namespace registration behavior has
   remained, but without benefit.
   
   This change removes the unnecessary namespace registration, which has
   substantial overhead, thus reducing startup time substantially when
   using protobufs.
   
   Because this change affects the import internals, quantifying the
   overhead requires a full tear-down/start-up of the Python interpreter.
   So, to capture the full cost for every run, I measured the time to
   launching a _fresh_ Python instance in a subprocess, varying the
   imports and code under test. In other words, I used `timeit` to
   measure the time to launch a _fresh_ Python subprocess which actually
   performs the imports.
   
   * Reference: normal Python startup (i.e., don't import protobuf at all).
     ```
      % python3 -m timeit -s 'import subprocess' -r 3 -n 10 'subprocess.call(["python3", "-c", "pass"])'
     10 loops, best of 3: 27.1 msec per loop
     ```
   
   * Baseline: cost to import `google.protobuf.descriptor`, with
     extraneous namespace packages.
     ```
     % python3 -m timeit -s 'import subprocess' -r 3 -n 10 'subprocess.call(["python3", "-c", "import google.protobuf.descriptor"])'
     10 loops, best of 3: 133 msec per loop
     ```
   
   * This change: cost to import `google.protobuf.descriptor`, without
     extraneous namespace packages.
     ```
     % python3 -m timeit -s 'import subprocess' -r 3 -n 10 'subprocess.call(["python3", "-c", "import google.protobuf.descriptor"])'
     10 loops, best of 3: 43.1 msec per loop
     ```
   
   [1]:  https://packaging.python.org/guides/packaging-namespace-packages/

   

   dlj-NaN committed Sep 21, 2020
   Copy the full SHA

   b89f88c View commit details

   Browse the repository at this point in the history