Add bzl_library targets to Gazelle

[achew22]

What type of PR is this?

Other

What package or component does this PR mostly affect?

all

What does this PR do? Why is it needed?

This adds bzl_library targets for Gazelle. This is useful in generating skydocs.

[jayconrod]

Please open an issue explaining how this is intended to work. I can't really evaluate this without knowing more.

It's likely this needs testing. I don't know whether these bzl_library rules are sufficient, and I won't know if they break in the future.

[achew22]

It occurs to me that you already have skylib as a dependency in Gazelle, updated PR to reflect what it probably should have looked like in the first place.

[jayconrod]

Couple comments and questions.

I've opened #763 and bazelbuild/rules_go#2437 to track migrating documentation for those projects to Stardoc.

Sorry I don't have bandwidth to work on this now though.

[jayconrod]

Is there a naming convention for these rules? This doesn't seem descriptive, but I guess if it's just a set of all .bzl files in each package, then it's good to use the same name. Just want to make sure that's what other rulesets are doing.

[jayconrod]

Let's update this to match the convention from @bazel_skylib//gazelle/bzl.

[jayconrod]

Is it okay that this target isn't visible outside the workspace?

[jayconrod]

I'm okay with merging this PR, but I don't understand this well enough to commit to maintaining it yet. So this might change or be removed in the future.

If that's okay with you, please add a TODO:

# TODO(#762): investigate generating stardoc documentation from these files.
# bzl_library targets in this repository may change in the future, so use with caution.

[jayconrod]

Gentle ping.

[jayconrod]

Let's update this to match the convention from @bazel_skylib//gazelle/bzl.

[achew22]

@jayconrod, sorry this has become a huge yak shave. I don't know why I thought coordinating 6 CLs between 5 different repos would be easy. Clearly I'm an idiot 🤦 .

Here are my kafka-esque yaks:

• Handle .bzl files in @bazel_tools properly in the generator
• Handle internal and private directories for visibility
• Upgrade rules_go's dep on rules_proto past da714efa6a5128abe9542b7b7cb2d340c6bb1cad so there are bzl_library targets in rules_proto
• Which unblocks bzl_library targets in rules_go
• Add bzl_library targets to bazel-federation
• Fix the fakes in stardoc
• Expose the .bzl files in @bazel_tools so that these can actually be depended on by deps.bzl
• Add bzl_library targets to rules_cc
• This PR which adds bzl_library targets to gazelle
• Loop back over all these repos and update their deps chains to the latest versions of each.
• Uncomment all the places where I left a lingering please uncomment this reference when bazel has .bzl` files exposed to the outside world.

Which will then culminate in me trying to get stardoc to generate documentation for rules_go that is as nice as the hand written docs that you've got.

My kingdom for a nicely formatted build encyclopedia...

---

WRT this PR, I've regenerated so we can inspect the generated output. I didn't realize that there were more places in Gazelle that depend on @io_bazel_rules_go//go/private:common 😦 . Maybe I can get around this by explicitly adding visibility for gazelle in the rules.

It's possible that we could advance on this by just cutting deps off, but in my experimentation with skydoc it requires ALL your dependencies to be available. We could definitely get many of them and start iterating on getting the rest as soon as possible, but it will leave that last hop of skydocifying as a todo.

[jayconrod]

Expose the .bzl files in @bazel_tools so that these can actually be depended on by deps.bzl

Would this mean that Gazelle and rules_go wouldn't work with older versions of Bazel, since we'd reference bzl_library targets that aren't present? I'd like to maintain a decently long compatibility window.

WRT this PR, I've regenerated so we can inspect the generated output. I didn't realize that there were more places in Gazelle that depend on @io_bazel_rules_go//go/private:common 😦 . Maybe I can get around this by explicitly adding visibility for gazelle in the rules.

Probably better to just copy definitions from @io_bazel_rules_go//go/private instead of referencing them. I think that's just technical debt from when Gazelle was split out of rules_go.

It's possible that we could advance on this by just cutting deps off, but in my experimentation with skydoc it requires ALL your dependencies to be available. We could definitely get many of them and start iterating on getting the rest as soon as possible, but it will leave that last hop of skydocifying as a todo.

That seems like kind of unfortunate behavior. I don't know much about Skydoc, but is there any way around the all-or-nothing requirement?

[achew22]

Would this mean that Gazelle and rules_go wouldn't work with older versions of Bazel, since we'd reference bzl_library targets that aren't present? I'd like to maintain a decently long compatibility window.

I believe this would restrict your ability to generate the docs, but would not restrict your ability to use Gazelle.

Probably better to just copy definitions from @io_bazel_rules_go//go/private instead of referencing them. I think that's just technical debt from when Gazelle was split out of rules_go.

I've expanded this CL to do just that.

That seems like kind of unfortunate behavior. I don't know much about Skydoc, but is there any way around the all-or-nothing requirement?

Unfortunately Stardoc basically starts up the same engine that does .bzl parsing in Bazel and asks it to print out the starlark function. Without adding a mode to Bazel where you can ignore dependencies, I don't think that's possible. There is a significant rewrite on the horizon, but it isn't yet scheduled, has no headcount, and doesn't appear to be on any OKRs. I think going this route will end up being less painful unfortunately.

I've got the repos in a much better state now. This CL now passes when I --override_repository rules_go, but I think we should hold off on submitting this until we can get a rules_go release that includes bazelbuild/rules_go#2621 rather than try and shim things.

[achew22]

Last bit of progress before bed. I've gotten quite a bit farther on getting stardoc to crash. bazelbuild/stardoc#76 if you would like to see the latest error.

Bedtime now.

[achew22]

There is, in fact light at the end of this tunnel! With all the PRs linked above, a custom compiled bazel binary, about a hundred instances of --overrides_repository, exactly 1 line of code changed in rules_go to satisfy the parser, and many more hours of keyboard durability testing with my forehead I was able to generate some markdown for def.bzl in the rules_go package.

It's not perfect, but it's a damn good start. Also, it looks like doc inheritance is going to be a thing that needs to be fixed before the gazelle/rules_go docs can be migrated. I'll start putting something together on that front soon.

I've attached the .md file, renamed to .txt, that was generated in @io_bazel_rules_go//go/BUILD.bazel by:
def_doc.md.txt

load("@io_bazel_stardoc//stardoc:stardoc.bzl", "stardoc")

stardoc(
    name = "def-docs",
    out = "def_doc.md",
    input = "def.bzl",
    deps = [
        "//extras:embed_data",
        "//go/private:context",
        "//go/private:go_toolchain",
        "//go/private:providers",
        "//go/private/rules:library",
        "//go/private/rules:nogo",
        "//go/private/rules:sdk",
        "//go/private/rules:source",
        "//go/private/rules:wrappers",
        "//go/private/tools:path",
    ],
)

[jayconrod]

Cool! Looks like it's gotten pretty far.

Doesn't seem to be able to tunnel through macros like go_binary. I'm not sure how it could do that though without an explicit hint.

[achew22]

Yeah, I think that's [doc inheritance] (bazelbuild/stardoc#27). In that bug they have a proposed syntax that looks basically like a docstring @inherit(symbol_name) which seems fine to me.

[UebelAndre]

So with bazelbuild/rules_go#2712 closed, is there still a lot of outstanding work to be done on this PR?

[achew22]

If you look at #760 (comment) you can see a list of things to do. Unfortunately the changes that were merged into rules_cc are insufficient since they are not publicly visible so we need to fix that before Gazelle can get bzl_library targets.

[UebelAndre]

Hey, I have a hard time understanding what code is where exactly, but were the bzl_library targets added to rules_cc? And released? The release is the biggest mystery for me.

[achew22]

The bzl_library targets can be found here https://github.com/bazelbuild/rules_cc/search?q=bzl_library. Check the blame layer to find the commit associated with them and see what tags that's a part of.

[UebelAndre]

Does rules_cc have tags? I looked at the blame and don't know how to pair it back to a version of Bazel. It's been in for 2 months now so I'm assuming it's available?

[achew22]

I do not work on rules_cc so I have no information on the subject. Best of luck!

[UebelAndre]

I do not work on rules_cc so I have no information on the subject. Best of luck!

I'm thoroughly confused by your response. Did you not write #760 (comment) ? Is this PR forever blocked because you don't know how to tell if the next task is complete or not? Why even open bazelbuild/rules_cc#87 if you had no way to know whether or not the changes would be usable?

[UebelAndre]

Sorry, not tying to be obnoxious, this PR is blocking my ability to generate documentation through Stardoc. It seems like all the blockers are satisfied for completing this PR so I'd hugely appreciate seeing this go through 🙏 😅

[achew22]

Unfortunately, there are still blockers. It is not possible to depend on rules_cc's targets because they are marked as private. They can not be depended upon by this repo, nor any other repo.

[google-cla]

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

[UebelAndre]

Unfortunately, there are still blockers. It is not possible to depend on rules_cc's targets because they are marked as private. They can not be depended upon by this repo, nor any other repo.

The *.bzl sources of rules_cc now publicly visible as of bazelbuild/rules_cc@d66a13e

[ngeor]

@googlebot I consent