bzlmod: Mechanism for optionally using a system-provided library in place of a version fetched by bazel

[jbms]

The existing mechanism in tensorstore and tensorflow uses an environment variable that is checked by a special repository rule.

@fmeum Proposed using a build_setting instead. Regardless of how the selection is made, though, the following functionality is needed:

• Selection needs to apply transitively, e.g if I select that system zlib should be used, and non-system libcurl is used which depends on zlib, libcurl should use system zlib.
• Ideally there would be a way to specify the minimum required version, and possibly required compile-time "features", for the system library.

See discussion on Slack here: https://bazelbuild.slack.com/archives/C014RARENH0/p1702789572436649?thread_ts=1702671095.223429&cid=C014RARENH0

[meteorcloudy]

I did some work several years ago to package Bazel for Debian, which required us to build Bazel with Debian system libraries. Unfortunately, the build is not maintained and probably no longer works, but I can share what I did to make it work and maybe we check if the solution will work with Bzlmod.

1. Created a list of system_repo that symlink to paths containing the system dependencies.
2. Added BUILD files for each system_repo so that Bazel knows how to build with those targets.
3. Added a few distribution_rules to switch between regular targets and system targets.
4. Replace regular rules with distribution rules in BUILD files of the dependencies.
5. Build with --define=distribution=debian to enable system dependencies.

To make it a general solution, I think the main problems are:

• How are we going to generate/maintain the system_repos and their BUILD files.
• We'll have to update each module to adopt the distribution rules to support this.

[jbms]

In the case of cmake, some of FindXXX scripts are distributed with cmake itself and otherwise usually with the package that uses it.

Sometimes a package does arrange to install a Find script as part of its normal installation (in which case it will probably just contain the absolute paths directly, rather than do any finding), similar to how it might install a pkg-config file. But that will only happen if the package itself uses cmake, and even then I think for distribution-installed packages it is rare to include a Find script.

In the case of Bazel it seems to me that the rules for generating the system repo could just be in the registry.

[jbms]

As an example, boringssl provides its own BUILD files, but we may want to substitute it with the system-installed openssl.

[jbms]

It seems to me that there are a few questions to resolve:

• Does the resolution of system vs. built happen during the loading phase (in which case I think an environment variable would be the only mechanism for configuring it) or during the analysis phase (in which case a normal build_setting could be used to configure it)? A build_setting would probably be more elegant, but means that we will need alias rules for each target.
• Should there just be a single zlib package in the bazel central registry, or should there be 3 different packages ---- zlib, zlib-system, and zlib-maybe-system, where zlib is always built from source, zlib-system always is the system library, and zlib-maybe-system selects between them according to some sort of configuration option?

[SomeoneSerge]

Sometimes a package does arrange to install a Find script as part of its normal installation (in which case it will probably just contain the absolute paths directly, rather than do any finding)
[...] even then I think for distribution-installed packages it is rare to include a Find script.

install(EXPORT ...) generates a xxxxConfig.cmake file; at least in Nixpkgs, pretty much all cmake-built projects ship those together with e.g. the headers, and it's also ensured those are automatically picked up by find_package. I presume it must be more or less the same with other distributions

zlib, zlib-system, and zlib-maybe-system, where zlib is always built from source, zlib-system always is the system library, and zlib-maybe-system selects between them according to some sort of configuration option?

Would this mean that downstream projects need to specify zlib-maybe-system instead of zlib as their dependency? I'd say, dependency injection-friendly behaviour (like with find_package/FetchContent_MakeAvailable(... FIND_PACKAGE_ARGS)) should be the default, otherwise it's hard to achieve consistent package sets and avoid the "combinatorial explosion" kind of picture.

Instead of hard-coding zlib vs zlib-maybe-system in projects, maybe we want CLI flags constraining Bazel to specific behaviours: something like --system-first ("[for all dependencies, ]try find_package() first, else try the source build, else there's probably no network available and it's time to fail") and a flag for --hermetic

[SomeoneSerge]

though we could have utilities to simplify this for packages that follow a common pattern, e.g. by invoking pkg-config

A behaviour like that of meson's dependency() is maybe a decent reference, I believe it can ingest both pkg-config and cmake exports