macOS: How to build a wrapper that works with different kinds of Python installations (official, Homebrew, ...)?

[petermbauer]

I have built a wrapper library using Python installed with the official Installer. Thus, the .so file contains the following information

otool -l wrapper.cpython-310-darwin.so
...
Load command 11
          cmd LC_LOAD_DYLIB
      cmdsize 88
         name /Library/Frameworks/Python.framework/Versions/3.10/Python (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 3.10.0
compatibility version 3.10.0

Now i would like to use this wrapper from a Homebrew Python installation but even when running python -c "import wrapper" with the Homebrew Python, it loads the Python library from the "official" installation.

What is the recommended way of building and distributing such wrapper libraries for macOS?

[henryiii]

Do not link to Python (except on Windows). See https://pybind11.readthedocs.io/en/stable/compiling.html and our three example repos, which include build systems, wheels, and cibuildwheel.

[petermbauer]

Thanks a lot for the quick reply!
I am trying to understand why Windows is different here, is there some more detailed description available somewhere?

Does this mean that under macOS and Linux there is no need to build separately with all Python versions that should be supported?

[petermbauer]

After digging a bit more into pybind11_add_module() i found that it uses Python_add_library() instead of the plain add_library() to create the wrapper library target. This in turn does not link the wrapper to the Python library but just adds the include directory containing the Python header files and adds the linker argument -undefined dynamic_lookup. Due to this linker flag, the missing symbols are ignored by the linker and provided later on when the wrapper library is loaded by Python.

Regarding building for multiple Python versions, i am still not sure if this needs to be done for non-Windows platforms.

[henryiii]

You need to build per Python version. If you use the Limited ABI, which pybind11 doesn't support yet, and nanobind only supports for 3.12+, then you can build once and reuse on later Python versions (this is true for all OS's). You use undefined symbols on linux and macOS, though macOS is better if you can generate the symbol list and give it to the linker. nanobind does this, but pybind11 doesn't currently. And windows you have to give it to the linker, though I think the linker uses the symbol list but doesn't make a hard link. It helps the linker to know if there are undefined symbols when loaded from Python, but you don't want to make a hard link, otherwise you'll depend on the exact patch release of Python, which you do not want to do.

https://packaging.python.org/en/latest/guides/packaging-binary-extensions/#binary-extensions-for-windows probably should cover this, but under week linking there's a FIXME: This section needs to be fleshed out.