feat: add table of modules for a libraries-bom version to README

[alicejli]

Follow up from #6069

Publishing the table of included artifacts for a libraries-bom to the README

[meltsufin]

Would it make sense to consolidate all of the scripts into just one script, or at least a single entry-point that updates the readme so that it can be easily executed locally without having to think which scripts have to run in what order and with what arguments?

[meltsufin]

@alicejli Have you tried using the Maven flatten plugin?
Add the following configuration to libraries-bom/pom.xml:

      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>flatten-maven-plugin</artifactId>
        <configuration>
          <updatePomFile>true</updatePomFile>
          <pomElements>
            <dependencyManagement>expand</dependencyManagement>
          </pomElements>
        </configuration>
      </plugin>

Then run mvn flatten:flatten. You should get the .flattened-pom.xml that should list all of the artifacts and versions contained in the lbraries-bom.

[alicejli]

@alicejli Have you tried using the Maven flatten plugin? Add the following configuration to libraries-bom/pom.xml:

      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>flatten-maven-plugin</artifactId>
        <configuration>
          <updatePomFile>true</updatePomFile>
          <pomElements>
            <dependencyManagement>expand</dependencyManagement>
          </pomElements>
        </configuration>
      </plugin>

Then run mvn flatten:flatten. You should get the .flattened-pom.xml that should list all of the artifacts and versions contained in the lbraries-bom.

Ah that's very handy! Thanks for showing. That is much simpler.
Are there any concerns with adding this configuration to the libraries-bom/pom.xml? I don't think I need the <updatePomFile>true</updatePomFile> line for it to work.

It seems like if I have:

1. A github actions workflow that runs mvn flatten:flatten
2. a script that parses the .flattened-pom.xml and updates the README, and discards the created .flattened-pom.xml files.

that would be a neat way to create the table.

[alicejli]

@meltsufin I've updated the scripts to use the maven flatten plugin which reduces a lot of the complexity - thanks for the idea!

cc @burkedavison as we chatted about this a bit this morning

The main challenge that I'm curious to get your thoughts on how to solve is getting the latest/updated client documentation/product reference links.
Those links are contained within each module's .repo-metadata.json file, and the previous iteration of this PR was grabbing those links when checking out each repo.
The .repo-metadata.json files are not updated frequently (if ever), so it seems a bit much to re-introduce checking out the repos for just those links. The main thing that would get out of date is whenever a new client library is generated.

The most ongoing toilsome method (which I don't think would be too toilsome given that we don't generate new client libraries that frequently) would be to do nothing and have it as a documented step as part of new client library generation to manually update the links for the product/client reference in this repo.

Do you have any thoughts on the best approach to keeping those links updated?

[burkedavison]

have it as a documented step as part of new client library generation to manually update the links for the product/client reference in this repo

Despite updates to .repo-metadata.json being rare, they do happen; and solving this situation by manual-process would set up a second source of truth for that data with inevitable desyncing issues that must be discovered each time before we can fix them. (One example source of concern: We don't review changes made to handwritten repos, so we'd be trusting that every Java SDK developer knows about this table's implementation needs.)

This may be a departure from the current direction we've been discussing; but I'd rather see logic that fetches the source of truth, and regenerates the client-library portion of the table from scratch each time rather than in-place updating. In the end, I think the goal should be that this table doesn't need any special handling when adding, removing, or updating client libraries.

[meltsufin]

The main challenge that I'm curious to get your thoughts on how to solve is getting the latest/updated client documentation/product reference links. Those links are contained within each module's .repo-metadata.json file, and the previous iteration of this PR was grabbing those links when checking out each repo. The .repo-metadata.json files are not updated frequently (if ever), so it seems a bit much to re-introduce checking out the repos for just those links. The main thing that would get out of date is whenever a new client library is generated.

We certainly should not create another manually maintained source of truth for library metadata. However, I don't like getting this information from the .repo-metadata.json either. It's seems error-prone. I think the best place to get this information is from the pom.xml metadata. It already supports things like project name and URL. If we are currently not providing this information in the POM, let's update our generation scripts to do that. In the meantime the table can just be missing this information.

[alicejli]

The main challenge that I'm curious to get your thoughts on how to solve is getting the latest/updated client documentation/product reference links. Those links are contained within each module's .repo-metadata.json file, and the previous iteration of this PR was grabbing those links when checking out each repo. The .repo-metadata.json files are not updated frequently (if ever), so it seems a bit much to re-introduce checking out the repos for just those links. The main thing that would get out of date is whenever a new client library is generated.

We certainly should not create another manually maintained source of truth for library metadata. However, I don't like getting this information from the .repo-metadata.json either. It's seems error-prone. I think the best place to get this information is from the pom.xml metadata. It already supports things like project name and URL. If we are currently not providing this information in the POM, let's update our generation scripts to do that. In the meantime the table can just be missing this information.

Cool. We don't currently provide the links to the client/product documentation in the POM, but I can create an issue to do so (with a follow-up issue to update this table with the links from the POM once they exist).

I did some testing and I don't believe there's a way to use the maven flatten plugin to get the metadata for the dependencies as well as it only includes the info here: https://maven.apache.org/ref/2.2.1/maven-model/maven.html#class_dependency. So to get the links from the pom.xml metadata, it would involve a script to ping Maven Central. This is all doable, but just want to point out that I don't think we can create the full table from just using the flatten plugin.

One question before I update this PR - @meltsufin would it be okay to keep the current links? I feel like the table is much more useful with them and even if they get stale/new library links won't exist, I think it's better to have them than not. I can add a footnote to the table saying links may be out of date in the meantime.

[meltsufin]

@alicejli Correct, the flatten plugin is not going to give us that metadata from the POMs and we would have to fetch them from Maven Central, which I still think is more robust than the .repo-metadata.json.

Regarding the links, I assume you're talking about the product links, I'm OK with keeping them temporarily.

[alicejli]

@alicejli Correct, the flatten plugin is not going to give us that metadata from the POMs and we would have to fetch them from Maven Central, which I still think is more robust than the .repo-metadata.json.

Regarding the links, I assume you're talking about the product links, I'm OK with keeping them temporarily.

Yep, I agree on the point about getting the metadata from Maven Central. And yes, I'm talking about the links to the product documentation as well as the Cloud RAD client reference info. I will update the PR accordingly and then re-request review.

[meltsufin]

Yep, I agree on the point about getting the metadata from Maven Central. And yes, I'm talking about the links to the product documentation as well as the Cloud RAD client reference info. I will update the PR accordingly and then re-request review.

Can Cloud RAD links be automatically calculated on the fly?

[alicejli]

Yep, I agree on the point about getting the metadata from Maven Central. And yes, I'm talking about the links to the product documentation as well as the Cloud RAD client reference info. I will update the PR accordingly and then re-request review.

Can Cloud RAD links be automatically calculated on the fly?

Good point. Will auto-generate those.