Bazel should come with a hermetic, multi-platforms cc toolchain by default

[sluongng]

Problem statement

The current cc toolchain configurations packaged with Bazel rely on the host system to already have the necessary tools pre-installed. This approach might work for bigger corporate environments where things are managed by a centralized team, however for smaller orgs and standalone users, knowing which cc toolchain to install for a particular project could be a pain.

This is not a new problem, we have seen this before with both Java and Python toolchains. The solutions was to provide remotejdk in Bazel for Java toolchains, and rules_python now includes a pre-compiled, multi-platforms python toolchain (https://github.com/bazelbuild/rules_python#toolchain-registration-with-bzlmod).

Potential impact

So where does a cc toolchain is need today? Here are some use cases I have personally observed when interacting with the Bazel community:

1. Cross compiling CGo binary from MacOS laptops targeting production environment.
   Uber blog post goes into detail the pain points https://www.uber.com/en-GB/blog/bootstrapping-ubers-infrastructure-on-arm64-with-zig/

2. Cross compiling C/C++ targeting different platforms.
   https://github.com/bazelembedded/bazel-embedded/ is an impressive example of such needs

3. Hermetic compilation / link action against system libraries
   https://github.com/grailbio/bazel-toolchain pioneer the approach of using a custom tarball as a hermetic sysroot configured in cc toolchain. This would allow folks to cross compile against linux system dependencies from MacOS or Windows

4. Build Python packages with C/C++ dependencies
   See https://github.com/jvolkman/rules_pycross/blob/main/pycross/private/cc_toolchain_util.bzl for some examples

5. Compile / link Rust binaries against C libraries
   There are many issues like Support building pure-Rust targets without a C++ toolchain rules_rust#1601 and in #rust slack channel

Suggest action

Similar to Bazel's remotejdk, Bazel should provide a sane, multi-platforms, hermetic cc toolchain out of the box.
I would recommend picking LLVM over gcc for better MacOS support.

Additional goals would be to make the cc toolchain configurable:

• Multi versions, distros
• Custom sysroot
• Additional platforms and toolings

[lberki]

Yes, this would be nice, but realistically, it's not something the Bazel team can take on. The world of C++ is way more complicated than that of Java: for example, compiled Java code is platform-agnostic and compiled C++ code s not, there are more C++ compilers than Java compilers, etc...

So I don't think we can commit to that. Unfortunately. I'd love for such a thing to exist, though.

[sluongng]

Perhaps, a more tangible short-term result we could aim for is documenting the current behavior of cc_toolchains default to the local host toolchain.

And provide a list of projects that supply a hermetic cc toolchain where the clang/glibc etc... are downloaded remotely from a checksum-verified external source. That way, users could pick the project they need.

WDYT?

[kjteske]

+1 to starting with documentation / a list of projects with examples. https://github.com/aspect-build/gcc-toolchain was hugely helpful for us.

[jfirebaugh]

If it's not something the Bazel team can take on (which is entirely reasonable), then I would say that the long term outcome should be that Bazel does not come with a default cc toolchain at all, but instead clearly documents the various hermetic toolchains available in the ecosystem (and of course they should all be easily integrated via bzlmod).

I echo @sluongng's problem statement above -- the fact that the default cc toolchain is non-hermetic is one of the most frustrating aspects of Bazel because it feels like a way in which Bazel does not live up to its fundamental promise to be correct by default.

[fzakaria]

I actually think it's OK to rely on system provided tooling but deepen integrations maybe with something like Nix to provide the shell environment.

This pairing would be very powerful and works well already with tools such as Meson