feat: move synthtool templates to `library_generation/owlbot` #2443

diegomarquezp · 2024-02-06T16:32:35Z

This PR transfers the java-specific templates from synthtool to library_geneation/owlbot
There is a new branch in synthtool (googleapis/synthtool#1923) that has the removed templates.
The intention is to keep that synthtool branch as parallel until we fully roll out the hermetic build workflows to both HW libraries and the monorepo.

Approach

We add a list of template exclusions to the configuration yaml and call java.common_templates with a custom template path (pointing to our OwlBot Java Postprocessor implementation here in library_generation plus the template exclusions found in the yaml.

The list of exclusions were obtained from owlbot.py files. Since google-cloud-java has the same exclusions for all libraries, we use a repo-level configuration entry. An example of template excludes is:

sdk-platform-java/java-iam/owlbot.py

Lines 26 to 37 in b0a57b7

    
           ".github/*", 
        
           ".kokoro/*", 
        
           "CODE_OF_CONDUCT.md", 
        
           "CONTRIBUTING.md", 
        
           "LICENSE", 
        
           "SECURITY.md", 
        
           "java.header", 
        
           "license-checks.xml", 
        
           "README.md", 
        
           "samples/*", 
        
           "renovate.json", 
        
           ".gitignore"

Different approach possible?

We could modify all owlbot.py files to use something other than java.common_templates. For example

from custom_owlbot import common_templates
...
common_templates(excludes=[/*preserved excludes*/])

With such approach, we would not have to parse owlbot.py files and take advantage of the fact it's an executable script.

Follow up?

We probably don't want to call common_templates twice, so it may be better to modify owlbot.py files to reference our own implementation instead of synthtool. (This is similar to "Different approach possible?" but it is more of a follow up once the scripts are live).

snippet-bot · 2024-02-06T18:21:26Z

Here is the summary of possible violations 😱

There is a possible violation for not having product prefix.

library_generation/owlbot/templates/java_library/samples/install-without-bom/pom.xml:30, tag {{metadata['repo'

There is a format violation for a region tag.

library_generation/owlbot/templates/java_library/samples/install-without-bom/pom.xml:30, tag {{metadata['repo'

The end of the violation section. All the stuff below is FYI purposes only.

Here is the summary of changes.

You are about to add 1 region tag.

library_generation/owlbot/templates/java_library/samples/install-without-bom/pom.xml:30, tag {{metadata['repo'

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

Refresh this comment

JoeWang1127 · 2024-02-06T18:53:41Z

library_generation/owlbot/src/apply_repo_templates.py

+  )
+
+
+def parse_template_excludes(owlbot_py_contents: str) -> str:


I think parsing generated owlbot.py is not intuitive.
Can we change the implementation of java.common_templates?

java.common_templates lives in synthtool

If we want to change the implementation of that function we would need to transfer synthtool to sdk-platform-java (which is an actual option).

Also from the PR description, we could modify all owlbot.py files (and its template) to call our_own_owlbot.common_templates instead of java.common_templates

cc: @suztomo

As discussed with @suztomo, I'll use the configuration yaml to have a list of exclusions

I modified the implementation to use the configuration yaml

library_generation/owlbot/bin/entrypoint.sh

JoeWang1127 · 2024-02-06T18:58:39Z

library_generation/owlbot/src/requirements.in

+pyyaml
+absl-py


Why do we need these two packages?

Co-authored-by: Joe Wang <106995533+JoeWang1127@users.noreply.github.com>

…ove-synthtool-templates

diegomarquezp · 2024-02-08T19:53:24Z

https://github.com/googleapis/sdk-platform-java/pull/2443/checks?check_run_id=21380060834
The snippet-bot exclusions are not getting picked up due to a (wrongly stated) format problem. I'm reaching out to repo team to see what's going on

diegomarquezp · 2024-02-08T19:54:44Z

https://github.com/googleapis/sdk-platform-java/pull/2443/checks?check_run_id=21378993952

+ mv owl-bot-staging/java-apigee-connect temp
+ rm -rd owl-bot-staging/
+ mv temp owl-bot-staging
+ monorepo=/home/runner/work/sdk-platform-java/sdk-platform-java/library_generation
+ echo 'Rendering templates'
Rendering templates
+ python3 /home/runner/work/sdk-platform-java/sdk-platform-java/library_generation/owlbot/src/apply_repo_templates.py /home/runner/work/sdk-platform-java/sdk-platform-java/library_generation/test/resources/integration/google-cloud-java/generation_config.yaml /home/runner/work/sdk-platform-java/sdk-platform-java/library_generation
2024-02-08 19:26:53,119 synthtool [CRITICAL] > You are running the synthesis script directly, this will be disabled in a future release of Synthtool. Please use python3 -m synthtool instead.
Traceback (most recent call last):
  File "/home/runner/work/sdk-platform-java/sdk-platform-java/library_generation/owlbot/src/apply_repo_templates.py", line 13, in <module>
    from library_generation.model.generation_config import from_yaml
ModuleNotFoundError: No module named 'library_generation'
Library postprocessing failed
Error: Process completed with exit code 1.

Adding a step to install via setup.py so the library_generation package can be available between python-shell-python calls

JoeWang1127 · 2024-02-08T20:03:15Z

.github/snippet-bot.yml

@@ -4,3 +4,4 @@ ignoreFiles:
  - src/test/**
  - test/**
  - showcase/**
+  - library_generation/owlbot/templates/java_library/samples/install-without-bom/pom.xml


Why do we need this line?

#2443 (comment) has the context. Basically the region tag check thinks that template has a region tag (wrong). I'm yet to get a response from the automation team

JoeWang1127 · 2024-02-08T20:08:28Z

Since you format Python code in owlbot, could you change the workflow file to check against that directory as well?

https://github.com/googleapis/sdk-platform-java/blob/main/.github/workflows/verify_library_generation.yaml#L111

sonarcloud · 2024-02-08T20:27:13Z

Quality Gate passed for 'gapic-generator-java-root'

Issues
0 New issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

sonarcloud · 2024-02-08T20:31:07Z

Quality Gate passed for 'java_showcase_integration_tests'

Issues
0 New issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

🤖 I have created a release *beep* *boop* --- <details><summary>2.35.0</summary> ## [2.35.0](v2.34.0...v2.35.0) (2024-02-13) ### Features * add `generate_repo.py` ([#2431](#2431)) ([47b632a](47b632a)) * move synthtool templates to `library_generation/owlbot` ([#2443](#2443)) ([5c95472](5c95472)) ### Bug Fixes * Apiary Host returns user set host if set ([#2455](#2455)) ([5f17e62](5f17e62)) * Cancel the Timeout Task for HttpJson ([#2360](#2360)) ([b177d9e](b177d9e)) ### Dependencies * update dependency commons-codec:commons-codec to v1.16.1 ([#2473](#2473)) ([8c6e91d](8c6e91d)) * update google api dependencies ([#2469](#2469)) ([ad4d4e6](ad4d4e6)) * update google auth library dependencies to v1.23.0 ([#2466](#2466)) ([349a5d3](349a5d3)) * update google auth library dependencies to v1.23.0 ([#2476](#2476)) ([6c9127c](6c9127c)) * update google http client dependencies to v1.44.1 ([#2467](#2467)) ([87d1435](87d1435)) * update googleapis/java-cloud-bom digest to ac9893c ([#2472](#2472)) ([7fff34a](7fff34a)) * update grpc dependencies to v1.61.1 ([#2463](#2463)) ([9ec575b](9ec575b)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

This PR transfers the java-specific templates from synthtool to `library_geneation/owlbot` There is a new branch in synthtool (googleapis/synthtool#1923) that has the removed templates. The intention is to keep that synthtool branch as parallel until we fully roll out the hermetic build workflows to both HW libraries and the monorepo. ### Approach We add a list of template exclusions to the configuration yaml and call `java.common_templates` with a custom template path (pointing to our OwlBot Java Postprocessor implementation here in `library_generation` plus the template exclusions found in the yaml. The list of exclusions were obtained from owlbot.py files. Since google-cloud-java has the same exclusions for all libraries, we use a repo-level configuration entry. An example of template excludes is: https://github.com/googleapis/sdk-platform-java/blob/b0a57b70d589e2bdc6e5fcb8cf64d08c3496bc57/java-iam/owlbot.py#L26-L37 ### Different approach possible? We could modify all owlbot.py files to use something other than `java.common_templates`. For example ``` from custom_owlbot import common_templates ... common_templates(excludes=[/*preserved excludes*/]) ``` With such approach, we would not have to parse owlbot.py files and take advantage of the fact it's an executable script. ## Follow up? We probably don't want to call `common_templates` twice, so it may be better to modify owlbot.py files to reference our own implementation instead of synthtool. (This is similar to "Different approach possible?" but it is more of a follow up once the scripts are live). --------- Co-authored-by: Joe Wang <106995533+JoeWang1127@users.noreply.github.com>

🤖 I have created a release *beep* *boop* --- <details><summary>2.35.0</summary> ## [2.35.0](v2.34.0...v2.35.0) (2024-02-13) ### Features * add `generate_repo.py` ([#2431](#2431)) ([47b632a](47b632a)) * move synthtool templates to `library_generation/owlbot` ([#2443](#2443)) ([5c95472](5c95472)) ### Bug Fixes * Apiary Host returns user set host if set ([#2455](#2455)) ([5f17e62](5f17e62)) * Cancel the Timeout Task for HttpJson ([#2360](#2360)) ([b177d9e](b177d9e)) ### Dependencies * update dependency commons-codec:commons-codec to v1.16.1 ([#2473](#2473)) ([8c6e91d](8c6e91d)) * update google api dependencies ([#2469](#2469)) ([ad4d4e6](ad4d4e6)) * update google auth library dependencies to v1.23.0 ([#2466](#2466)) ([349a5d3](349a5d3)) * update google auth library dependencies to v1.23.0 ([#2476](#2476)) ([6c9127c](6c9127c)) * update google http client dependencies to v1.44.1 ([#2467](#2467)) ([87d1435](87d1435)) * update googleapis/java-cloud-bom digest to ac9893c ([#2472](#2472)) ([7fff34a](7fff34a)) * update grpc dependencies to v1.61.1 ([#2463](#2463)) ([9ec575b](9ec575b)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

diegomarquezp added 4 commits February 2, 2024 20:34

transfer templates from synthtool to owlbot

ba38e9c

use locally hosted templates

37af910

fix regex parsing

9e9e743

update protobuf and generator versions in IT

d2a8ca4

product-auto-label bot added the size: xl Pull request size is extra large. label Feb 6, 2024

diegomarquezp added 4 commits February 6, 2024 16:45

minor fixes

2cb0a31

install synthtool for unit tests

e1c262d

use repo level googleapis commitish for kms

79b41bc

comment fixes

7d6dabc

diegomarquezp marked this pull request as ready for review February 6, 2024 18:20

diegomarquezp requested a review from a team as a code owner February 6, 2024 18:20

diegomarquezp requested a review from JoeWang1127 February 6, 2024 18:20

JoeWang1127 reviewed Feb 6, 2024

View reviewed changes

library_generation/owlbot/bin/entrypoint.sh Outdated Show resolved Hide resolved

JoeWang1127 reviewed Feb 6, 2024

View reviewed changes

Update library_generation/owlbot/bin/entrypoint.sh

7c503c9

Co-authored-by: Joe Wang <106995533+JoeWang1127@users.noreply.github.com>

diegomarquezp marked this pull request as draft February 6, 2024 22:10

diegomarquezp added 7 commits February 7, 2024 18:35

use config yaml to compute exclusions

5587d0f

remove unused __init__.py files

65d8ef9

Merge remote-tracking branch 'origin/move-synthtool-templates' into m…

a0b0cfd

…ove-synthtool-templates

remove owlbot.py check in template rendering postprocessing step

b980b88

add comment for setup.py

f2cd5bf

remove unit test

f416b4c

simplify module processing script

8d348a4

diegomarquezp marked this pull request as ready for review February 7, 2024 18:56

diegomarquezp added 2 commits February 7, 2024 19:08

update googleapis commitish

9c778f6

update googleapis commitish ii

dbecc0e

diegomarquezp requested a review from suztomo February 7, 2024 20:48

suztomo approved these changes Feb 7, 2024

View reviewed changes

diegomarquezp added 11 commits February 8, 2024 18:31

Merge remote-tracking branch 'origin/main' into move-synthtool-templates

07bc35f

post-merge fixes

2539e1e

fix unit tests

560644c

delete generated goldens

e0aade6

restore necessary golden files

1663a0f

snippet bot to ignore templates

0fb4b7e

fix snippet bot ii

ccfee71

fix snippet bot iii

07ea2a6

use full path to conflicting pomxml

b38f755

use full path to actual conflicting pomxml

0e8b19e

lint

f4e7867

install lirbary_generation python package via setup.py

f86635d

JoeWang1127 reviewed Feb 8, 2024

View reviewed changes

JoeWang1127 approved these changes Feb 8, 2024

View reviewed changes

include owlbot python sources in python lint check

02bac22

diegomarquezp enabled auto-merge (squash) February 8, 2024 20:13

lint owlbot

e540631

diegomarquezp merged commit 5c95472 into main Feb 8, 2024
43 of 47 checks passed

diegomarquezp deleted the move-synthtool-templates branch February 8, 2024 20:31

release-please bot mentioned this pull request Feb 8, 2024

chore(main): release 2.35.0 #2448

Merged

JoeWang1127 mentioned this pull request Apr 12, 2024

fix: exclude path_to_yaml when compares configs #2633

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: move synthtool templates to `library_generation/owlbot` #2443

feat: move synthtool templates to `library_generation/owlbot` #2443

diegomarquezp commented Feb 6, 2024 •

edited

snippet-bot bot commented Feb 6, 2024 •

edited

JoeWang1127 Feb 6, 2024

diegomarquezp Feb 6, 2024

diegomarquezp Feb 6, 2024

diegomarquezp Feb 7, 2024

JoeWang1127 Feb 6, 2024

diegomarquezp Feb 7, 2024

diegomarquezp commented Feb 8, 2024 •

edited

diegomarquezp commented Feb 8, 2024

JoeWang1127 Feb 8, 2024

diegomarquezp Feb 8, 2024

JoeWang1127 commented Feb 8, 2024

sonarcloud bot commented Feb 8, 2024

sonarcloud bot commented Feb 8, 2024

	".github/*",
	".kokoro/*",
	"CODE_OF_CONDUCT.md",
	"CONTRIBUTING.md",
	"LICENSE",
	"SECURITY.md",
	"java.header",
	"license-checks.xml",
	"README.md",
	"samples/*",
	"renovate.json",
	".gitignore"

		)


		def parse_template_excludes(owlbot_py_contents: str) -> str:

		pyyaml
		absl-py

feat: move synthtool templates to library_generation/owlbot #2443

feat: move synthtool templates to library_generation/owlbot #2443

Conversation

diegomarquezp commented Feb 6, 2024 • edited

Approach

Different approach possible?

Follow up?

snippet-bot bot commented Feb 6, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

diegomarquezp commented Feb 8, 2024 • edited

diegomarquezp commented Feb 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JoeWang1127 commented Feb 8, 2024

sonarcloud bot commented Feb 8, 2024

Quality Gate passed for 'gapic-generator-java-root'

sonarcloud bot commented Feb 8, 2024

Quality Gate passed for 'java_showcase_integration_tests'

feat: move synthtool templates to `library_generation/owlbot` #2443

feat: move synthtool templates to `library_generation/owlbot` #2443

diegomarquezp commented Feb 6, 2024 •

edited

snippet-bot bot commented Feb 6, 2024 •

edited

diegomarquezp commented Feb 8, 2024 •

edited