feat: move Java Owlbot into this repository for postprocessing

[diegomarquezp]

This is stage 1 of the Owlbot replacement project.
This PR moves the implementation of Synthtool's Java Owlbot to this repository.

Changes

• Synthtool will be used as an external tool by cloning it and running pip install
• The Java Owlbot implementation has been transferred to sdk-platform-java
• The non-java-specific Owlbot CLI (copy-code) remains referenced as a docker image
• The interface of postprocess_library.sh will be modified (see next section). It will still, however, be intended to be called by generate_library.sh. It will still be expected to be run on pre-existing fully functional GAPIC libraries.

New interface of postprocess_library.sh

We will define the interface of postprocess_library.sh as follows:

Input parameters

Argument	Required	Description
postprocessing_target	yes	This is the main parameter to the script. Java Owlbot runs its scripts in this target directory and modifies it (hence this being also an output). Some files are expected to be here owlbot.py used by java-owlbot .OwlBot.yaml (optional) used to transfer code from preprocessed_sources_path .repo-metadata.json - essential metadata file used by java owlbot In the context of generate_library.sh, destination_path should be pointing to a folder with such files, since it is destination_path that will be called as postprocessing_target
versions_file	yes	Versions to be applied to README.md and pom files. This is done by java owlbot
preprocessed_sources_path	yes	This folder is the input preprocessed libraries, which should look like a googleapis-gen folder (example). The script will infer the necessary `proto_path` variable from the proto library. . In the perspective of generate_library.sh, the script will generate a temporary folder with such structure and will pass it to postprocess_library.sh

Outputs

Artifact	Description
postprocessing_target	Also passed as input to the script. The post-processor will modify the files contained here. It will also use copy-code to transfer preprocessed java sources into this folder if preprocessed_sources_path is provided..

Remarks

• Since synthtool and java owlbot run on python, a python version will be stored in library_generation/configuration. This will be managed via pyenv
• Since synthtool is designed to run on Linux only, the integration test with post-processing for macos were disabled. We will not follow up in a later stage by creating a synthtool-only image (POC)
• google-java-format was originally added in the image definition. The scripts will now download it at runtime. The formatter requires java 11, so the github integration test was updated to run on java 11.
• Since destination_path was previously being used as a pre-existing, fully functional GAPIC library when postprocessing is enabled, it will now clash with the java owlbot postprocessor by having unexpected folders in the GAPIC folder (i.e. preprocessed libraries produced by generate_library.sh will be sent to, for example, google-cloud-java/java-.../ which will produce a library with pre-processed folders). This is dealt with by having a temp_destination_path in generate_library.sh that we will either (1) use it as preprocessed_sources_path when postprocessing is enabled or (2) simply transfer its contents to destination_path when it's not enabled.
• Due to Hermetic Build integration tests fail when googleapis have proto changes not yet reflected in google-cloud-java or googleapis-gen #2303, I temporarily disabled the tests for google/cloud/asset/v1p2beta1 and google/cloud/compute

[suztomo]

Memo: As per the design doc, we won't use this until the end of Q1 2024.

[suztomo]

Until we start using this postprocessor (around Feb/March 2024), people, including me, continue make changes to the Owlbot Java postprocessor via the synthtool repository. Would you write some README section what steps are needed to copy these changes in future (when?)?

[blakeli0]

Until we start using this postprocessor (around Feb/March 2024), people, including me, continue make changes to the Owlbot Java postprocessor via the synthtool repository. Would you write some README section what steps are needed to copy these changes in future (when?)?

That's valid concern. How often do we usually update owlbot post-processor? We thought it does not change often so expected it to not change in another two months. But if it does change, I think we can either

1. Update the post-processing logic here as well when we make changes to synthtool.
2. Don't do copy the changes over until the final cut over, we can copy all the changes in the past few months at that time in one shot.

[blakeli0]

LGTM. As we discussed offline, please look into the possibility of moving the Java specific Github templates into this repo, but it can be a separate PR.

[meltsufin]

Nice work @diegomarquezp!

[diegomarquezp]

Until we start using this postprocessor (around Feb/March 2024), people, including me, continue make changes to the Owlbot Java postprocessor via the synthtool repository. Would you write some README section what steps are needed to copy these changes in future (when?)?

@suztomo Done

[diegomarquezp]

Until we start using this postprocessor (around Feb/March 2024), people, including me, continue make changes to the Owlbot Java postprocessor via the synthtool repository. Would you write some README section what steps are needed to copy these changes in future (when?)?

That's valid concern. How often do we usually update owlbot post-processor? We thought it does not change often so expected it to not change in another two months. But if it does change, I think we can either

1. Update the post-processing logic here as well when we make changes to synthtool.
2. Don't do copy the changes over until the final cut over, we can copy all the changes in the past few months at that time in one shot.

@blakeli0 I missed point 2 when modifying the README. I mentioned in the README that there were slight tweaks here and there in the transferred owlbot implementation. Whether we modify them within or after the two coming months, the instructions apply to both cases equally.

edit: I would lean for avoiding modifications until end up using this implementation, then we can transfer all the changes since this PR and then.

[sonarcloud]

Quality Gate passed for 'gapic-generator-java-root'

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

[diegomarquezp]

IT for google/cloud/asset/v1 failing as recent changes in googleapis have not been reflected in google-cloud-java (pending PR: googleapis/google-cloud-java#10138)

[sonarcloud]

Quality Gate passed for 'java_showcase_integration_tests'

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud