feat: add `generate_repo.py` #2431

JoeWang1127 · 2024-01-30T00:11:38Z

In this PR:

Add generate_repo.py to generate libraries defined in a configuration file.
- Regenerate .repo-metadata.json
- Generate .OwlBot.yaml and owlbot.py for new clients
- Invoke generate_composed_library.py to generate multiple versions of a GAPIC library.
- Apply repo-level post processing using python
  - Regenerate root pom.xml
  - Regenerate gapic-libraries-bom/pom.xml
Format python script using black.
Add unit tests for utility functions.
Add an integration test using python, replacing shell integration tests.
Add a python lint check in CI.

This is step 2 of milestone 2 of hermetic build project.

library_generation/utilities.py

diegomarquezp · 2024-02-06T15:46:35Z

.github/workflows/verify_library_generation.yaml

What tool did you format the python files with? It'd be nice if that tool had a linter mode that fails on non-compliant files so we can add it as part of our linter step

What tool did you format the python files with?

I use black to format python code and it also has a validate function.

It'd be nice if that tool had a linter mode that fails on non-compliant files so we can add it as part of our linter step

Sure, I'll add a lint step in CI.

This tool is included when I install all requirements.

I added a python lint job in CI.

library_generation/generate_composed_library.py

JoeWang1127 · 2024-02-06T21:20:01Z

library_generation/utilities.py

+    if library.rpc_documentation:
+        repo_metadata["rpc_documentation"] = library.rpc_documentation
+
+    # generate .repo-meta.json


Add more comments here.

diegomarquezp · 2024-02-06T21:34:10Z

library_generation/test/resources/integration/java-bigtable/generation_config.yaml

As discussed, we can either enable the IT for this split repo or leave it as a follow up, whichever is most convenient

I think I'll remove the split repo integration test for now, since we mainly focus on generating google-cloud-java.

I think it should be fine with leaving the yaml file, although not touched by our scripts

Sure, I'll modify the integration test to reflect this.

Restored the config file for java-bigtable.

suztomo

It would be nice if I can see how the real main entry point (library_generation/generate_repo.py?) is invoked in verify_library_generation.yaml. No need to replace the Python-based integraiton tests.

diegomarquezp · 2024-02-07T20:52:27Z

library_generation/generate_repo.py

Maybe we can leave a follow up after this PR to update the README to reflect that the main entrypoint is this one and not the now-lower-level generate_library.sh

Sure, I'll follow up with this.

sonarcloud · 2024-02-07T22:18:08Z

Quality Gate passed for 'gapic-generator-java-root'

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

sonarcloud · 2024-02-07T22:22:07Z

Quality Gate passed for 'java_showcase_integration_tests'

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

🤖 I have created a release *beep* *boop* --- <details><summary>2.35.0</summary> ## [2.35.0](v2.34.0...v2.35.0) (2024-02-13) ### Features * add `generate_repo.py` ([#2431](#2431)) ([47b632a](47b632a)) * move synthtool templates to `library_generation/owlbot` ([#2443](#2443)) ([5c95472](5c95472)) ### Bug Fixes * Apiary Host returns user set host if set ([#2455](#2455)) ([5f17e62](5f17e62)) * Cancel the Timeout Task for HttpJson ([#2360](#2360)) ([b177d9e](b177d9e)) ### Dependencies * update dependency commons-codec:commons-codec to v1.16.1 ([#2473](#2473)) ([8c6e91d](8c6e91d)) * update google api dependencies ([#2469](#2469)) ([ad4d4e6](ad4d4e6)) * update google auth library dependencies to v1.23.0 ([#2466](#2466)) ([349a5d3](349a5d3)) * update google auth library dependencies to v1.23.0 ([#2476](#2476)) ([6c9127c](6c9127c)) * update google http client dependencies to v1.44.1 ([#2467](#2467)) ([87d1435](87d1435)) * update googleapis/java-cloud-bom digest to ac9893c ([#2472](#2472)) ([7fff34a](7fff34a)) * update grpc dependencies to v1.61.1 ([#2463](#2463)) ([9ec575b](9ec575b)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

In this PR: - Add `generate_repo.py` to generate libraries defined in a configuration file. - Regenerate `.repo-metadata.json` - Generate `.OwlBot.yaml` and `owlbot.py` for new clients - Invoke `generate_composed_library.py` to generate multiple versions of a GAPIC library. - Apply repo-level post processing using python - Regenerate root `pom.xml` - Regenerate `gapic-libraries-bom/pom.xml` - Format python script using `black`. - Add unit tests for utility functions. - Add an integration test using python, replacing shell integration tests. - Add a python lint check in CI. This is [step 2](https://docs.google.com/document/d/1JiCcG3X7lnxaJErKe0ES_JkyU7ECb40nf2Xez3gWvuo/edit?pli=1&tab=t.0#bookmark=id.h3n0hcp3ch2m) of milestone 2 of hermetic build project.

🤖 I have created a release *beep* *boop* --- <details><summary>2.35.0</summary> ## [2.35.0](v2.34.0...v2.35.0) (2024-02-13) ### Features * add `generate_repo.py` ([#2431](#2431)) ([47b632a](47b632a)) * move synthtool templates to `library_generation/owlbot` ([#2443](#2443)) ([5c95472](5c95472)) ### Bug Fixes * Apiary Host returns user set host if set ([#2455](#2455)) ([5f17e62](5f17e62)) * Cancel the Timeout Task for HttpJson ([#2360](#2360)) ([b177d9e](b177d9e)) ### Dependencies * update dependency commons-codec:commons-codec to v1.16.1 ([#2473](#2473)) ([8c6e91d](8c6e91d)) * update google api dependencies ([#2469](#2469)) ([ad4d4e6](ad4d4e6)) * update google auth library dependencies to v1.23.0 ([#2466](#2466)) ([349a5d3](349a5d3)) * update google auth library dependencies to v1.23.0 ([#2476](#2476)) ([6c9127c](6c9127c)) * update google http client dependencies to v1.44.1 ([#2467](#2467)) ([87d1435](87d1435)) * update googleapis/java-cloud-bom digest to ac9893c ([#2472](#2472)) ([7fff34a](7fff34a)) * update grpc dependencies to v1.61.1 ([#2463](#2463)) ([9ec575b](9ec575b)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

JoeWang1127 added 2 commits January 30, 2024 00:04

feat: add generate_repo.py

8c4bd4a

refactor model

f4bccaa

product-auto-label bot added the size: l Pull request size is large. label Jan 30, 2024

JoeWang1127 added 2 commits January 30, 2024 01:52

refactor

b2b3fe4

add private methods

e804991

product-auto-label bot added size: xl Pull request size is extra large. and removed size: l Pull request size is large. labels Jan 30, 2024

JoeWang1127 added 14 commits January 30, 2024 21:31

change config file

737dbbd

refactor to download repo only once

362fc27

use absolute path

a8cd899

do not remove files as owlbot cli will take care of it

449f3d5

move pre generation to utilities

ae4726c

remove new-client scripts

c134ebf

move api download function to utilities

23e4732

add helper function

f853130

fix unit tests

1478e4e

add copyright

cd90e04

refactor

5b60d25

regenerate root pom.xml

997cff5

generate gapic libraries bom using python

6894a55

Merge branch 'main' into feat/generate_repo

99afa38

JoeWang1127 mentioned this pull request Feb 1, 2024

chore: refactor new-client.py #2422

Closed

JoeWang1127 added 8 commits February 1, 2024 18:24

add unit tests

a7b210a

add unit tests

d3bc0f6

do not pull git repository

1edbb21

add unit test

0d14629

refactor

b7c6728

remove irrelevant test data

e76647c

do not use enum

ec8d51e

use apigeeconnect in yaml

818e3a9

JoeWang1127 added 2 commits February 5, 2024 15:54

covert transport

5adf0ee

enable diff_files

b9a717c

diegomarquezp reviewed Feb 6, 2024

View reviewed changes

library_generation/utilities.py Outdated Show resolved Hide resolved

diegomarquezp reviewed Feb 6, 2024

View reviewed changes

JoeWang1127 commented Feb 6, 2024

View reviewed changes

library_generation/generate_composed_library.py Outdated Show resolved Hide resolved

JoeWang1127 commented Feb 6, 2024

View reviewed changes

diegomarquezp reviewed Feb 6, 2024

View reviewed changes

JoeWang1127 added 6 commits February 6, 2024 21:45

refactor according to code review

2e0f3a9

code format

2c0f9b2

code format

e592f71

add python linter

369ca21

change job name

c17d321

Merge branch 'main' into feat/generate_repo

a0dc4cb

JoeWang1127 mentioned this pull request Feb 7, 2024

chore: Update showcase protos #2445

Merged

JoeWang1127 marked this pull request as ready for review February 7, 2024 17:45

JoeWang1127 requested a review from a team as a code owner February 7, 2024 17:45

Merge branch 'main' into feat/generate_repo

0e2f22b

suztomo approved these changes Feb 7, 2024

View reviewed changes

diegomarquezp reviewed Feb 7, 2024

View reviewed changes

bring back config for java-bigtable

c1ef755

diegomarquezp approved these changes Feb 7, 2024

View reviewed changes

JoeWang1127 merged commit 47b632a into main Feb 7, 2024
44 checks passed

JoeWang1127 deleted the feat/generate_repo branch February 7, 2024 22:51

release-please bot mentioned this pull request Feb 7, 2024

chore(main): release 2.35.0 #2448

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add `generate_repo.py` #2431

feat: add `generate_repo.py` #2431

JoeWang1127 commented Jan 30, 2024 •

edited

diegomarquezp Feb 6, 2024

JoeWang1127 Feb 6, 2024

JoeWang1127 Feb 6, 2024

JoeWang1127 Feb 6, 2024

JoeWang1127 Feb 6, 2024

diegomarquezp Feb 6, 2024

JoeWang1127 Feb 6, 2024

diegomarquezp Feb 7, 2024

JoeWang1127 Feb 7, 2024

JoeWang1127 Feb 7, 2024

suztomo left a comment

diegomarquezp Feb 7, 2024

JoeWang1127 Feb 7, 2024

sonarcloud bot commented Feb 7, 2024

sonarcloud bot commented Feb 7, 2024

feat: add generate_repo.py #2431

feat: add generate_repo.py #2431

Conversation

JoeWang1127 commented Jan 30, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

suztomo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarcloud bot commented Feb 7, 2024

Quality Gate passed for 'gapic-generator-java-root'

sonarcloud bot commented Feb 7, 2024

Quality Gate passed for 'java_showcase_integration_tests'

feat: add `generate_repo.py` #2431

feat: add `generate_repo.py` #2431

JoeWang1127 commented Jan 30, 2024 •

edited