Multiple top-level platforms?

[lberki]

The command line option one uses to tell Bazel what platform to build for is called --platforms, in plural.

This was done this way to keep the option open to have multiple top-level platforms; this way, that would not be an incompatible change, it would only make a use case that is currently an error (specifying multiple platforms) not an error anymore.

Is this a good direction to explore? On one hand, it's clearly useful to be able to build things for multiple platforms at the same time and to run tests on multiple platforms at the same time, but it does not come for free. The main difficulty I foresee is that the Build Event Protocol currently does not support multiple top-level targets with the same configuration and if it did, we'd have to teach not only Bazel to produce such BEP events, but also every consumer of the BEP to understand such data.

A very pedestrian way to work around the lack of multiple top level platforms is a custom Starlark rule with a split transition. This has a number of disadvantages:

• Tests cannot be run for multiple platforms this way
• Every possible top-level target would need to be wrapped with such a split transition pseudo-alias

But it's still a workaround.

Please chime in with your use case (if you have one) or with why do you think this is difficult, if not listed above!

[lberki]

cc @katre (his opinion counts for a lot) and @jin (because he is working on things that could benefit from something like this)

[brentleyjones]

I've already sent this to @katre, and he listed a blurb in one of his docs (https://docs.google.com/document/d/1LD4uj1LJZa98C9ix3LhHntSENZe5KE854FNL5pxGNB4/edit?usp=sharing), but here is my take on it: https://docs.google.com/document/d/1OVagRkIArFv8N5fspUzhHrJ20od8zqWbOG51DTmuaD0/edit?usp=sharing

[lberki]

There was indeed such a document! I totally forget about that, I just opened this thread in my sudden enthusiasm.

[katre]

Yes, @brentleyjones 's document is a great read. I'm writing up the results of the big Platform API requirements gathering process, but I'll go ahead and spoil the results now: the suggestion to change how we handle multiple values for --platforms was not identified as a major blocker by users, and will not be prioritized by the Configurability team, although if anyone in the community (cough cough @brentleyjones) wants to write up a concrete proposal and implementation I'll happily push from this end to support it.

[brentleyjones]

Even if the survey says that we don't need the changes I proposed, we need something like them to unblock rules_apple migrating to the platforms API... which I think is quite important.

[lberki]

Ack. I'm all for choosing a few projects but doing them well and I can understand why you are reluctant to do it. If anything, I'd be more careful: the ramifications about BEP mean that even contributions would need to be carefully evaluated to make sure that they don't break BEP.

@katre do you think it makes sense to keep this thread open or is the "Multiplatform builds proposal" a better canonical place for the discussion?

[katre]

Dealing with questions like "how does BEP handle this type of build?" is absolutely part of the design process.

Let's go ahead and close this, and have any further discussion in the doc.

[gregestren]

This doesn't solve the BEP problem, but a quick technical observation: top-level targets today build in multiple configurations (because of incoming transitions). It's already the case that $ bazel build //:foo //:bar doesn't imply //:foo and //:bar have the same configuration.

I have no idea offhand how or if BEP represents that.

[lberki]

@gregestren that's true, but you can't have a case when there are multiple top-level targets with the same label that way because incoming transitions cannot be split ones.

[lberki]

Ack, I'll hereby close this thread then. If any further discussion is needed, let's do it here:

https://docs.google.com/document/d/1OVagRkIArFv8N5fspUzhHrJ20od8zqWbOG51DTmuaD0/edit

[illicitonion]

I have a different use-case than the doc is proposing - the doc is trying to address "I want to build multiple things, each of which may have a single canonical platform". Maybe this discussion is the right place for surfacing them?

Please chime in with your use case (if you have one)

My main use-case is wanting exhaustive tests on multiple platforms. If I've written some code which is platform-dependent but supports multiple platforms, I want to be able to bazel test --platforms=linux,mac,windows ... and to run all three versions of the tests, rather than needing to specifically make a macro which expands tests out to multiple versions of them and making sure it's used everywhere.

[lberki]

@brentleyjones what's the connection to the Apple rules? I thought that the way Apple rules work is that the top-level target is e.g. an ios_application which then does a split transition into multiple platforms, which is something already possible today.

[brentleyjones]

It does a split onto legacy flags, like cpu. It does not handle platforms (requires a platforms mapping file). I believe the doc discusses all the ways rules_apple needs the things mentioned, as I used it as the primary example throughout.

[lberki]

Is there an intrinsic limitation in Bazel that prevents Apple rules from splitting on --platforms instead of --cpu?

Multiple platforms behind a configuration transition is a much different (and easier) use case than multiple top-level platforms.

[brentleyjones]

Yes, which platform do we split on? We need a list of them to choose from. We can't use predefined ones because then we would lose customizations that users have set, like remote execution properties.

[lberki]

I don't quite understand the "which platform do we split on" question; you mean which platforms are going to be on the output of the split transition?

[brentleyjones]

Yes. We can chose cpus because they are all defined by bazel. But what platform do we use for "iOS simulator" or "iOS device"? Which platform do we use for the watch_application that might be specified on the ios_application target?

My quasi-proposal doc goes over how using --platforms with multiple values, along with filtering, allows the rules to know which platforms are available to it.