Syncing rules_{cc, java} and bazel_tools

[fmeum]

At the moment, the toolchain definitions for Java and C++ are duplicated between the Bazel repo and the rules_java and rules_cc ruleset repos. Since there is no automated process that keeps these repos in sync, they have diverged, with the ruleset repo versions missing a number of features and bug fixes (e.g., rules_java misses 7556e11).

This is problematic for multiple reasons:

• Bzlmod uses the toolchain definitions from rules_java rather than those in bazel_tools. As a result, a simple flip of the --enable_bzlmod flag causes subtle changes to Java toolchains.
• Bazel CI does seem to use the toolchain definitions from rules_java, whereas locally executed tests use bazel_tools. This led to very surprising failures on Verify bootclasspath version is at most compilation runtime version #18343 that took a few hours to debug.
• Buildifier lints have pushed users towards using the ruleset repos, but if they also get the toolchains from them, that results in breakages. This is even more surprising since the rules themselves are noop wrappers whereas the toolchains are very much different from what's shipped with Bazel.

I see at least the following options to improve the situation:

1. Set up a (semi-)automated process to keep rules_java and rules_cc updated with internal and external changes to Bazel.
2. Leave the ruleset repos as is for now, but ensure that both Bzlmod and Bazel CI don't use them.
3. Actually make the ruleset repos the sources of truth for toolchains.

@comius @Wyverald @meteorcloudy @oquenchil @hvadehra

[comius]

The direction I'd prefer to see is #3: making ruleset repos the source of truth and removing things from @bazel_tools where it's already possible. That's because bzlmod already takes ruleset repos as sot.

We could implement some automation internally to warn us if the changes are not in sync. That could come in handy when things are not removable from bazel_tools.

[meteorcloudy]

Once we have conclusion on what we should do, we can file additional issues for the action items.

[meteorcloudy]

Option 3 will make it slightly harder to ship changes to default toolchains Bazel uses, IIRC, it'll be

• Make changes to rules_cc/rules_java repo
• Make a new release
• Update the WORKSPACE suffix and MODULE.tools to point to the new version.

Kind of like the current java_tools release, is this fine with the Build API team?

Additionally, I'm not sure if there are more blockers of moving out the toolchains to rules_{cc, java}, but I definitely like this direction if you think we should give it a try.

[comius]

Isn't this very close to what we already need to be doing? Because of bzlmod we always need to update rules repos, do a release and update MODULE.tools.

And we need to sync bazel_tools and rules_cc/rules_java.

The diff is:

-1. Make changes to bazel_tools
2. Make changes to rules_cc/rules_java repo
3. Make a new release
4. Update MODULE.tools
+5. Update the WORKSPACE suffix

That is ideally, if bazel_tools is close to empty (or aliased out)

[meteorcloudy]

I was comparing to option, basically not using rules_java and rules_cc at all until the rules are fully starlarkified and moved there, but I guess we don't have to wait for that.

SG, then let's go ahead with option 3

[martis42]

As a Bazel user I would prefer option #3. The repos for the language specific rules are the location where I would also expect the language specific toolchains to live.