Multi-platform remote caching and execution: brainstorming

[tjgq]

Hi all,

I'd like to share a document where I'm brainstorming potential solutions for multi-platform remote caching and execution (that is, across execution platforms, not target platforms).

I'd like to hear about the community's use cases for multi-platform builds, and whether any of proposed solutions would be a good match for them. Comments are welcome (preferentially on the document since it's easier to tie them to individual passages, but general comments here on GitHub are okay too).

[lberki]

My general opinion is that the simpler, the better. It looks like there isn't any perfect solution and all of the alternatives given in the document are at best the "hopefully it will work" category and therefore spending more complexity on them doesn't seem to have a good return on investment.

In particular:

• I dislike ambient toolchains because it doesn't even attempt to guarantee correctness
• I dislike cache key rewriting because it looks hideously complex and there is a lot of room to get things wrong in both directions (too much caching and not enough caching)

[sluongng]

A few observations I have here:

1. First, I was a bit confused by your "Tool inputs" section. Mostly because I think there is a disconnection between how I often think of toolchain dependencies and not knowing how to translate that into your model.
   I think of Bazel toolchains in 2 categories: host toolchain and remote toolchain(or hermetic toolchain).

   • Host toolchain is a group of tools that need to be prepared and be available in the underlying system before the build happens, we simply declare to Bazel to trust us that they exist (i.e. @local_config_cc//:toolchain).
   • The remote toolchain is the set of files and archives that are prepared by Bazel during loading phases, often by downloading them from a remote URL with attached checksum validation (I.e. remotejdk).

   Do these 2 concepts map correspondingly to your "source inputs" and "derived inputs"?

2. Recently I raised in rules_go a similar issue about the current Go toolchain (aka. Go SDK) setup. Provide a macro to register multiple go toolchains rules_go#3540
   Would something that combines multiple toolchains(multiple go SDK archives) for different execution platforms be a "universal" toolchain as referred to in your doc?

3. There are many references about "keeping tools outside of workspace" and I think both "Ambient toolchain" and "Versioned toolchain" are solutions for my concept of "host toolchain"?

4. Cache key rewriting is essentially what we are having today if we have a "remote toolchain" correct? The tools will be included as action inputs and thus, their hashes will be part of the action cache key calculation. When the action is uploaded to the action cache, we would want to programmatically generate different versions of the same action, one for each platform... or to rewrite the input action into a "universal" action that's gonna match similar action on all platforms (?).

5. Universal toolchain, to me, is an effort to shift the platform-specific logic into the toolchain itself. First of all, we gonna need a bigger toolchain that gona support all platforms. At first, I was thinking of the effort of rewriting everything with https://github.com/jart/cosmopolitan, but from your experimental branch master...experimental-universal-android-toolchain, it seems like we simply need an if statement to switch the logic based on platform detection. This increases the existing toolchain complexity, which may be acceptable for a smaller toolchain such as Go SDK, but could be a nightmare for Java/CC(?).

6. This comment

   If Bazel had a way to lazily fetch the contents source artifacts, and to direct remote executors to fetch them, it would alleviate the issue since only toolchain files used by local actions would have to be fetched.

   Reminds me of how buck2 is currently doing http_archive as part of the build graph instead of in a different loading phase like Bazel.
   There is a concept of sub_target and dynamic outputs that would let you run an analyzer after the download and declare targets inside the archive. (Similar to providing a prepared BUILD file).
   Either Bazel would need something like this, or we need to revisit the FUSE discussion in Mark "Remote Output Service..." as approved. proposals#311

Based on my very limited understanding here, I tend to agree with you, that universal toolchain is a good direction to head down to. But depending on how much complexity/burden it would put on the toolchain maintainers, versioned toolchain approach would be a better/cheaper short-term solution.

[tjgq]

1. First, I was a bit confused by your "Tool inputs" section. [...] I think of Bazel toolchains in 2 categories: host toolchain and remote toolchain(or hermetic toolchain) [...] Do these 2 concepts map correspondingly to your "source inputs" and "derived inputs"?

No, source inputs and derived inputs are both "hermetic". They have paths rooted at the workspace; appear in action inputs (both internally and as reported by aquery); and both their path and contents factor into the identity of an action. None of these properties are true for "non-hermetic" inputs, which are a third category.

I think "hermetic" vs "non-hermetic" (or "hermetic" vs. "ambient") are better terms than "host" vs "remote", because "hermetic" inputs don't necessarily have to be fetched from a remote system (consider e.g. local_jdk_repository, which just points to files present on the host system, but is still hermetic with respect to those files, unlike local_config_cc).

Thanks for the feedback; I'll think about making the "Tool inputs" section clearer, by mentioning this third category of "ambient inputs" explicitly.

2. Recently I raised in rules_go a similar issue about the current Go toolchain (aka. Go SDK) setup. Provide a macro to register multiple go toolchains rules_go#3540
   Would something that combines multiple toolchains(multiple go SDK archives) for different execution platforms be a "universal" toolchain as referred to in your doc?

The defining characteristic of a "universal toolchain" (as described in my doc) is that the tools for every possible execution platform are always present in the inputs to an action, regardless of where the action executes. Note that I'm not (yet) proposing that Bazel should be able automatically combine preexisting per-platform toolchains into a universal toolchain (primarily because I'm not sure that's even possible to automate generally). I'm proposing that toolchain owners should be responsible for providing a prepackaged "universal" toolchain. I'm sure this could be done for Go if we wanted to.

3. There are many references about "keeping tools outside of workspace" and I think both "Ambient toolchain" and "Versioned toolchain" are solutions for my concept of "host toolchain"?

Yes, both "ambient toolchain" and "versioned toolchain" make use of ambient host tools, and are therefore non-hermetic.

4. Cache key rewriting is essentially what we are having today if we have a "remote toolchain" correct? The tools will be included as action inputs and thus, their hashes will be part of the action cache key calculation. When the action is uploaded to the action cache, we would want to programmatically generate different versions of the same action, one for each platform... or to rewrite the input action into a "universal" action that's gonna match similar action on all platforms (?).

The idea is for all platforms to rewrite the key in the same way and upload the result under that single key. I don't think we'd want to generate N different versions of the same action and store them separately; that would require Bazel to have knowledge of every possible toolchain implementation, which is one of the things this approach is trying to avoid.

An important point is that the rewritten key lacks complete information to execute the action, so it's not usable as a remote execution request; if we wanted this approach to also support remote builds, we'd have to introduce separate notions of "execution request" and "cache key" in the protocol.

5. Universal toolchain, to me, is an effort to shift the platform-specific logic into the toolchain itself. First of all, we gonna need a bigger toolchain that gona support all platforms. At first, I was thinking of the effort of rewriting everything with https://github.com/jart/cosmopolitan, but from your experimental branch master...experimental-universal-android-toolchain, it seems like we simply need an if statement to switch the logic based on platform detection. This increases the existing toolchain complexity, which may be acceptable for a smaller toolchain such as Go SDK, but could be a nightmare for Java/CC(?).

I'm convinced that all of the proposed approaches increase toolchain complexity, because they'd have to simultaneously support both the "multi-platform" and "single-platform" way of doing things. The only way we could avoid this is if we made "multi-platform" the default and deleted the "single-platform" implementations, but I'm not sure that's viable (we'd be asking single-platform projects to pay a performance penalty, lose some correctness guarantees, or both).

The "wrapper script with a switch statement" approach could definitely be improved upon; for example, my experimental branch doesn't currently attempt to deduplicate toolchain files that are actually the same on every platform, so the resulting toolchain is larger than it needs to be. Cosmopolitan is an interesting idea to further reduce the toolchain size, but I'd consider it a speculative project; I'm not confident that it can successfully build any C++ code we throw at it, and in the end we'd still pay a performance penalty on ARM.

6. This comment

   If Bazel had a way to lazily fetch the contents source artifacts, and to direct remote executors to fetch them, it would alleviate the issue since only toolchain files used by local actions would have to be fetched.

   Reminds me of how buck2 is currently doing http_archive as part of the build graph instead of in a different loading phase like Bazel.
   There is a concept of sub_target and dynamic outputs that would let you run an analyzer after the download and declare targets inside the archive. (Similar to providing a prepared BUILD file).
   Either Bazel would need something like this, or we need to revisit the FUSE discussion in Mark "Remote Output Service..." as dropped. proposals#311

Thanks for the buck2 pointer, I'll look into it.

To make a precise technical point, what we'd need is a "remote source service" (for source artifacts), not a "remote output service" (for derived artifacts). FUSE is definitely a possible implementation strategy (which, incidentally, closely mirrors how Blaze works at Google), but I hear that it will soon cease to be a viable option on MacOS. I also suspect that the scope of this service would be larger than just Bazel itself; for remote builds, you'd want to tell the remote executor to fetch inputs directly from the source service and avoid proxying them through Bazel.

This is obviously a very complex addition, so my preference would be to pursue it as an optimization layer on top of a simpler idea (e.g., a universal toolchain) instead of a core requirement.

[sluongng]

If you are looking for a pointer to Buck2:

• Anon target https://buck2.build/docs/rule_authors/anon_targets/
• Dynamic outputs https://buck2.build/docs/rule_authors/dynamic_dependencies/#implementation

Example of using anon target to run http_archive:

• protobuf toolchain https://github.com/facebook/buck2/blob/0f0c109c74c822d1562eb28c5ca4ed17e61e06c1/shim/third-party/proto/defs.bzl#L52-L66
• Go SDK https://github.com/sluongng/buck2-go/blob/9f938ed8b7bcb6f1580feb1fa14355d168726677/toolchains/go.bzl#L145-L165

[cjhopman]

Since buck2 was mentioned, I'll note that we've pretty successfully used the universal toolchain approach with buck2. The approach fits naturally into the build model (which, while buck2 and bazel are similar, how toolchains are handled is quite different, but I suspect it similarly fits nicely into the bazel model). We make it really simple to define tools that work that just package multiple underlying tools and select them at runtime, the work to support it on the tool/toolchain side is quite small as long as it's okay that the tools/toolchains are much larger. The main challenge then is around dealing with the cost of doubling the size of larger toolchains, and we have needed to do things that avoid downloading parts of the toolchain that aren't needed on a host.

[gregestren]

I want to simultaneously acknowledge community concern about universal toolchains while also acknowledging their strong benefits in simplicity and power.

I agree we don't want to overburden toolchain maintainers. But I like that it requires (?) them to audit more carefully that the bundled implementations are, in fact, equivalent, and nicely contained in a single manageable place. Versioning sounds harder to stay on top of and more correctness-prone.

I wonder how far Bazel can go to simplify that burden. I'm sure 100% automation is impossible. But maybe Bazel can reduce the pain of that responsibility, and even offer automatic validation checks?

Then the other big concern is file size, including users who are downloading everything to their laptop for the first time. Happy you're looking into how to optimize that. I have no idea personally what threshold makes that manageable.

[tjgq]

Recent developments in this space:

• Bazel 7 will ship with support for remote cache key scrubbing, which is a simplified version of the "cache key rewriting" idea described in the brainstorming doc.

• I've put together a small guide on how to construct your own universal toolchain.

[sluongng]

I found myself referencing this proposal today.

Could we please add this to https://github.com/bazelbuild/proposals with Draft status?

[lberki]

@sluongng bazelbuild/proposals#370 it is!