Expose previously defined repositories in repository_ctx

[aiuto]

The small goal is that a repository rule should be able to query for other repositories that are already loaded by the user and then take specific action based on that repository being present.

Motivating use case: license and package attestation for accurate SBOM creation.

• To produce accurate SBOMs we need to be able to query down to the packages that are used to build an artifact.
• To make this scalable, we need to unify the representation of license and package data across all remote repository types. For example, we can use the license and package_info rules from @rules_license to specify this data in the BUILD file for each module. That way we can write tools to gather and analyze the metadata without having to have each tool understand every remote repository type.
• In the long run, to produce good SBOMs, all modules must have package_info in their BUILD files. This can happen either explicitly, or implicitly, during module download and instantiation.

Today:

• Some upstream repositories to provide data via rules_license. That's an easy case. The upstream repository depends on rules_license via MODULE.bazel, and thus the user project transitively depends on it too.
• Some repository rule types, such as rules_jvm_external, do not depend on rules_license. I will be adding mechanism to create package_info targets in the generated BUILD files. The question is how to do that in the least invasive way.

I start with the assumptions that:

1. We can't simply force all users to depend on rules_license explicitly
2. We should not implicitly import it when using @bazel_tools//tools/build_defs/repo

To satisfy those constraints, downloaders like jvm.bzl and maven_rules.bzl will look for the presence of rules_license and alter behavior to include package_info targets if present, otherwise not. This is possible today with some repository rule trickery, and some repository rules already do these tricks. Overall, we could reduce complexity by being able to change behavior on what the user has already loaded.
Bonus if we can determine that the user explicitly included a repository in MODULE.bazel or WORKSPACE. That is a strong signal that they want to depend on those capabilities, so any other rule types should be able to make use of those.

cc: @shs96c

[aiuto]

A related concept to this is exposing a dictionary of workspace level features to repository rules. Something like

module[|workspace](
  name='my_project',
  features = {
     "prefer_locally_installed_tools": True,
     "enable_package_info": True,
  }
)

The dictionary would appear as repository_ctx.workspace_features. Only the top level workspace features would be used. There will, of course, be calls for feature creep. The obvious one is that we should be able to specify command line flags on the workspace. But we have .bazelrc for that. Thus, we have to ask the inverse question. Why can't we set these features in .bazelrc? My take on that is that we already have too many flags and it is too easy for people to override them with their local .rc file. The repository level features are ones which inform the correctness of the project. A user should not be able to override them for a single build.

@Wyverald Any thoughts? I don't think either of these concepts would impact module resolution.

[sluongng]

I made a comment in Gazelle just now related to this bazelbuild/bazel-gazelle#1314 (comment)

Having the ability to check whether rules_license does exist or not in the repository context would help signal other repository rules to generate relevant rules_license targets.
So the proposal is a 👍 from me.

[meteorcloudy]

Some repository rule types, such as rules_jvm_external, do not depend on rules_license

Why can't rules_jvm_external just depend on rules_license? What are the disadvantages?

[fmeum]

Strongly agree with this - even today, essentially every Bazel project depends on platforms and bazel-skylib, which is completely fine. As long as rules_license keeps a slim profile long-term by only providing the Starlark symbols required for interop, the cost would remain negligible.

[aiuto]

I'm trying to be a good citizen, but I can keep the declarative part of rules_license small. And by small, I mean 0 dependencies. The code which does analysis of the graph can be optional modules.

But then I need a way so that if a module just depends on rules_license you get the base subset, but if you (in MODULE.bazel) want the full rules_license you get a different archive. Just spitballing ideas here

• BCR has rules_license pointing to the small version
• BCR has rules_license_full pointing to the large version
• an extension in the large version binds "rules_license" to the code in rules_license_full.

That way it only brings in dependencies if you want to run the analysis.
There will still be some module trickery, and the distribution gets much harder to create.
I can probably get there.

Or.... we just separate the code very carefully so that @rules_license//rules:* can load with no dependencies and you just have to make sure you have the right compiler deps to use the rest of it.

[fmeum]

If rules_license just defines a bunch of providers and simple rules and there is a separate rules_license_audit module defining aspects, license checkers, etc., then rulesets would only have to depend on the former whereas corp projects can depend on the latter.

Is there additional complexity involved that would make this difficult? What kind of "binding" would be required other than cross-module load()?

[meteorcloudy]

I don't see rules_license currently has any non-dev dependencies at https://github.com/bazelbuild/rules_license/blob/27c9be72d2285755f90ffd0920ea9020939d682e/MODULE.bazel#L1

If it's just extra code for analysis of the graph in rules_pkg module, I think that's totally fine.

[aiuto]

I have been trying to keep this in one respository. I can publish two modules out of that.

[fmeum]

This setup should be much simpler now thanks to https://bazelbuild.slack.com/archives/C014RARENH0/p1684518734446419?thread_ts=1684518734.446419&cid=C014RARENH0

[aiuto]

I'm not making sense of that thread. IMO, cookbook examples in bazelbuild/examples, or it didn't happen.

…

On Wed, May 24, 2023 at 2:44 PM Fabian Meumertzheim < ***@***.***> wrote: This setup should be much simpler now thanks to https://bazelbuild.slack.com/archives/C014RARENH0/p1684518734446419?thread_ts=1684518734.446419&cid=C014RARENH0 — Reply to this email directly, view it on GitHub <#18473 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHBZPEOLYU53QIF75ZTXHZJJXANCNFSM6AAAAAAYLJGKPQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[meteorcloudy]

There is a link in that thread which points the documentation: https://github.com/bazel-contrib/publish-to-bcr/blob/main/templates/README.md#optional-configyml