Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: create pr after library generation #10503

Merged
merged 47 commits into from
Mar 12, 2024
Merged
Show file tree
Hide file tree
Changes from 38 commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
a18841c
feat: create pr after library generation
JoeWang1127 Mar 8, 2024
949c779
add comment
JoeWang1127 Mar 8, 2024
fbdf767
setup branch
JoeWang1127 Mar 8, 2024
5c3b52f
create or update pr
JoeWang1127 Mar 8, 2024
f1fb22f
enable workflow
JoeWang1127 Mar 8, 2024
87d9c3a
setup github token
JoeWang1127 Mar 8, 2024
03af469
select one library to verify the workflow
JoeWang1127 Mar 8, 2024
86f3b9b
change format
JoeWang1127 Mar 8, 2024
e859513
remove head
JoeWang1127 Mar 8, 2024
03193a3
add quote
JoeWang1127 Mar 8, 2024
f5e36ac
shorten config for testsing
JoeWang1127 Mar 8, 2024
84b5543
add a library
JoeWang1127 Mar 9, 2024
668f6af
add head
JoeWang1127 Mar 9, 2024
087982f
set remote repo
JoeWang1127 Mar 9, 2024
8f2f6cf
commit change in configuration
JoeWang1127 Mar 9, 2024
2702a5c
update googleapis commit
JoeWang1127 Mar 9, 2024
adbeb2a
test body
JoeWang1127 Mar 9, 2024
9f9a7b4
set x
JoeWang1127 Mar 9, 2024
b65be6f
set x for docker
JoeWang1127 Mar 9, 2024
eabbc07
set env
JoeWang1127 Mar 9, 2024
ebffdbd
change env
JoeWang1127 Mar 9, 2024
8f955d7
restore generation
JoeWang1127 Mar 9, 2024
41cc413
remove gke v1alpha2
JoeWang1127 Mar 9, 2024
d0d0cdb
add apphub
JoeWang1127 Mar 9, 2024
8acbc80
add chat
JoeWang1127 Mar 9, 2024
0862a24
add cloudcontrolspartner (commented out)
JoeWang1127 Mar 9, 2024
746676f
remove compute small
JoeWang1127 Mar 9, 2024
7f7c38d
update committish
JoeWang1127 Mar 9, 2024
8b12617
restore cloudcontrolspartner
JoeWang1127 Mar 9, 2024
8d698ab
update policytroubleshooter
JoeWang1127 Mar 9, 2024
df44e18
add v2 for securitycenter
JoeWang1127 Mar 9, 2024
f9d7e7e
add securityposture
JoeWang1127 Mar 9, 2024
4e78e89
add workspaceevents
JoeWang1127 Mar 9, 2024
6acc593
change step order
JoeWang1127 Mar 9, 2024
d58bdc2
add variable
JoeWang1127 Mar 11, 2024
1752882
edit title
JoeWang1127 Mar 11, 2024
0211cfe
use partial clone
JoeWang1127 Mar 11, 2024
d336503
use job level env
JoeWang1127 Mar 11, 2024
23190c4
change commit message
JoeWang1127 Mar 11, 2024
6815a40
Merge branch 'main' into feat/hermetic-pr-creation
JoeWang1127 Mar 11, 2024
400025a
change workflow name
JoeWang1127 Mar 11, 2024
fd27bab
allow empty commit
JoeWang1127 Mar 11, 2024
8698923
extract image tag
JoeWang1127 Mar 11, 2024
554938f
remove owlbot:run tag
JoeWang1127 Mar 11, 2024
c964025
change workflow name
JoeWang1127 Mar 11, 2024
a1dc0ce
define bash branch
JoeWang1127 Mar 11, 2024
b103130
remove branch dispatcher
JoeWang1127 Mar 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
81 changes: 80 additions & 1 deletion .github/workflows/verify-generation-config.yaml
blakeli0 marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,43 @@ name: Verify Generation Configuration
on:
schedule:
- cron: '0 2 * * *' # nightly at 2 am UTC
push:
branches:
- feat/hermetic-pr-creation
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll remove this trigger after pr is approved.

workflow_dispatch:


jobs:
generate-from-configuration:
runs-on: ubuntu-22.04
env:
branch_name: generate-libraries
repo_volumes: "-v repo-google-cloud-java:/workspace/google-cloud-java"
steps:
- uses: actions/checkout@v4
- name: get baseline commit
id: get-baseline
shell: bash
run: |
echo "baseline_commit=$(grep googleapis_commitish generation_config.yaml | cut -d " " -f2 | xargs)" >> "$GITHUB_ENV"
- name: setup branch for pull request
id: setup-branch
shell: bash
run: |
[ -z "`git config user.email`" ] && git config --global user.email "cloud-java-bot@google.com"
[ -z "`git config user.name`" ] && git config --global user.name "cloud-java-bot"
# try to find a open pull request associated with the branch
pr_num=$(gh pr list -s open -H "${branch_name}" -q . --json number | jq ".[] | .number")
# create a branch if there's no open pull request associated with the
# branch; otherwise checkout the pull request.
if [ -z "${pr_num}" ]; then
git checkout -b "${branch_name}"
else
gh pr checkout "${pr_num}"
fi
echo "pr_num=${pr_num}" >> "$GITHUB_ENV"
env:
GH_TOKEN: ${{ secrets.CLOUD_JAVA_BOT_TOKEN }}
- name: setup docker environment
shell: bash
run: |
Expand All @@ -18,10 +48,22 @@ jobs:
docker volume rm repo-google-cloud-java
fi
docker volume create --name "repo-google-cloud-java" --opt "type=none" --opt "device=$(pwd)" --opt "o=bind"
- name: update googleapis commit to latest
id: update-commit
shell: bash
run: |
mkdir tmp-googleapis
# use partial clone because only commit history is needed.
git clone --filter=blob:none https://github.com/googleapis/googleapis.git tmp-googleapis
pushd tmp-googleapis
git pull
latest_commit=$(git rev-parse HEAD)
popd
rm -rf tmp-googleapis
sed -i -e "s/^googleapis_commitish.*$/googleapis_commitish: ${latest_commit}/" generation_config.yaml
- name: generate from configuration
shell: bash
run: |
repo_volumes="-v repo-google-cloud-java:/workspace/google-cloud-java"
docker run --rm \
${repo_volumes} \
-v /tmp:/tmp \
Expand All @@ -30,3 +72,40 @@ jobs:
-e "REPO_BINDING_VOLUMES=${repo_volumes}" \
gcr.io/cloud-devrel-public-resources/java-library-generation:latest \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally we should not use latest, but rather a versioned docker image, otherwise it is not hermetic. But since this is the first time we use the scripts and there might be bug fixes in this cycle, let's keep it for now. Once the scripts are stable, we should read the gapic-generator-java version from the build config file and use that version for the docker image.
For now, can we extract latest as a parameter and share it between this step and the step below?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just FYI, we have a release job that releases gcr.io/cloud-devrel-public-resources/java-library-generation:${GGJ_VERSION}, so if we are to remove it, we can use that tag (and configure renovate bot)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once the scripts are stable, we should read the gapic-generator-java version from the build config file and use that version for the docker image.

I think we should group the docker image tag with generator update so that renovate bot can update them together.

For now, can we extract latest as a parameter and share it between this step and the step below?

Sounds good.

python /src/generate_repo.py generate --generation-config-yaml=/workspace/google-cloud-java/generation_config.yaml --repository-path=/workspace/google-cloud-java
- name: generate pull request description
id: generate-description
shell: bash
run: |
# the pr description (body) will be available in pr-description.txt after
# running this command.
set -x
docker run --rm \
${repo_volumes} \
-v /tmp:/tmp \
-v /var/run/docker.sock:/var/run/docker.sock \
-e "RUNNING_IN_DOCKER=true" \
-e "REPO_BINDING_VOLUMES=${repo_volumes}" \
gcr.io/cloud-devrel-public-resources/java-library-generation:latest \
python /src/generate_pr_description.py generate --generation-config-yaml=/workspace/google-cloud-java/generation_config.yaml --baseline-commit="${baseline_commit}"
env:
baseline_commit: ${{ env.baseline_commit }}
- name: create or update the pull request
shell: bash
run: |
title="chore: generate libraries at $(date)"
Copy link
Member

@suztomo suztomo Mar 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My instinct says "chore:" does not goes to Release Please pull request and the pull request description is not read by Release Please. Let's see.

(If this is really the case, we can fix it by "BEGIN_COMMIT_OVERRIDE".)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel the same way. Can we make sure we understand the behavior before merging? e.g. If it is a chore, is release please going to include the NESTE_COMMIT in the description? If it is a feat, is the title going to be included? The behavior of release-please will affect how we generate the PR descriptions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to @chingor13, release-please won't skip a commit if it's a chore, it just ignores chore commit messages.

git add java-* pom.xml gapic-libraries-bom/pom.xml versions.txt generation_config.yaml
git commit -m "chore: ${title}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"chore" appears twice.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the commit message.

if [ -z "${pr_num}" ]; then
git remote add monorepo https://cloud-java-bot:${GH_TOKEN}@github.com/${{ github.repository }}.git
git fetch -q --unshallow monorepo
git push -f monorepo "${branch_name}"
set -x
gh pr create --title "${title}" --label "owlbot:run" --head "${branch_name}" --body "$(cat pr_description.txt)"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we still need owlbot:run here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The check is not removed yet.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should remove the label here and disable the check for this branch, otherwise owlbot postprocessing would still be run against this PR right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to removing the required check for owlbot postprocessor and the label

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

else
git push
gh pr edit "${pr_num}" --title "${title}"
gh pr edit "${pr_num}" --body "$(cat pr_description.txt)"
fi
env:
GH_TOKEN: ${{ secrets.CLOUD_JAVA_BOT_TOKEN }}
pr_num: ${{ env.pr_num }}
47 changes: 43 additions & 4 deletions generation_config.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
gapic_generator_version: 2.37.0
protobuf_version: '25.2'
googleapis_commitish: 20c357f6f5c7b995b617b15cab61999e6574bdab
googleapis_commitish: 6500290663163ba7dc6e0a35231772f5f78c3b62
owlbot_cli_image: sha256:623647ee79ac605858d09e60c1382a716c125fb776f69301b72de1cd35d49409
synthtool_commitish: 6612ab8f3afcd5e292aecd647f0fa68812c9f5b5
template_excludes:
Expand Down Expand Up @@ -167,6 +167,14 @@ libraries:
GAPICs:
- proto_path: google/appengine/v1

- api_shortname: apphub
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which script do you use to populate this yaml?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I manually added these libraries, the parameters are copied from new library generation PR.

name_pretty: App Hub API
product_documentation: https://cloud.google.com/app-hub/docs/overview
api_description: App Hub simplifies the process of building, running, and managing applications on Google Cloud.
rpc_documentation: https://cloud.google.com/app-hub/docs/reference/rpc
GAPICs:
- proto_path: google/cloud/apphub/v1

- api_shortname: area120tables
name_pretty: Area 120 Tables
product_documentation: https://area120.google.com/
Expand Down Expand Up @@ -434,6 +442,14 @@ libraries:
GAPICs:
- proto_path: google/cloud/channel/v1

- api_shortname: chat
name_pretty: Google Chat API
product_documentation: https://developers.google.com/chat/concepts
api_description: The Google Chat API lets you build Chat apps to integrate your services with Google Chat and manage Chat resources such as spaces, members, and messages.
rest_documentation: https://developers.google.com/chat/api/reference/rest
GAPICs:
- proto_path: google/chat/v1

- api_shortname: cloudbuild
name_pretty: Cloud Build
product_documentation: https://cloud.google.com/cloud-build/
Expand All @@ -459,6 +475,14 @@ libraries:
- proto_path: google/cloud/commerce/consumer/procurement/v1
- proto_path: google/cloud/commerce/consumer/procurement/v1alpha1

- api_shortname: cloudcontrolspartner
name_pretty: Cloud Controls Partner API
product_documentation: https://cloud.google.com/sovereign-controls-by-partners/docs/sovereign-partners
api_description: Provides insights about your customers and their Assured Workloads based on your Sovereign Controls by Partners offering.
GAPICs:
- proto_path: google/cloud/cloudcontrolspartner/v1
- proto_path: google/cloud/cloudcontrolspartner/v1beta

- api_shortname: cloudquotas
name_pretty: Cloud Quotas API
product_documentation: https://cloud.google.com/cloudquotas/docs/
Expand Down Expand Up @@ -490,7 +514,6 @@ libraries:
excluded_dependencies: grpc-google-cloud-compute-v1
GAPICs:
- proto_path: google/cloud/compute/v1
- proto_path: google/cloud/compute/v1small

- api_shortname: confidentialcomputing
name_pretty: Confidential Computing API
Expand Down Expand Up @@ -922,7 +945,6 @@ libraries:
GAPICs:
- proto_path: google/cloud/gkehub/v1
- proto_path: google/cloud/gkehub/v1alpha
- proto_path: google/cloud/gkehub/v1alpha2
- proto_path: google/cloud/gkehub/v1beta
- proto_path: google/cloud/gkehub/v1beta1

Expand Down Expand Up @@ -1336,6 +1358,7 @@ libraries:
api_id: policy-troubleshooter.googleapis.com
GAPICs:
- proto_path: google/cloud/policytroubleshooter/v1
- proto_path: google/cloud/policytroubleshooter/iam/v3

- api_shortname: policysimulator
name_pretty: Policy Simulator API
Expand Down Expand Up @@ -1530,7 +1553,7 @@ libraries:
- proto_path: google/cloud/security/privateca/v1
- proto_path: google/cloud/security/privateca/v1beta1

# duplicated api_shortname
# duplicated api_shortname
- api_shortname: securitycenter
name_pretty: Security Command Center
product_documentation: https://cloud.google.com/security-command-center
Expand All @@ -1549,6 +1572,7 @@ libraries:
- proto_path: google/cloud/securitycenter/v1
- proto_path: google/cloud/securitycenter/v1beta1
- proto_path: google/cloud/securitycenter/v1p1beta1
- proto_path: google/cloud/securitycenter/v2

- api_shortname: securitycenter
name_pretty: Security Command Center Settings API
Expand All @@ -1570,6 +1594,13 @@ libraries:
GAPICs:
- proto_path: google/cloud/securitycentermanagement/v1

- api_shortname: securityposture
name_pretty: Security Posture API
product_documentation: https://cloud.google.com/security-command-center/docs/security-posture-overview
api_description: Security Posture is a comprehensive framework of policy sets that empowers organizations to define, assess early, deploy, and monitor their security measures in a unified way and helps simplify governance and reduces administrative toil.
GAPICs:
- proto_path: google/cloud/securityposture/v1

- api_shortname: servicecontrol
name_pretty: Service Control API
product_documentation: https://cloud.google.com/service-infrastructure/docs/overview/
Expand Down Expand Up @@ -1964,6 +1995,14 @@ libraries:
- proto_path: google/cloud/workflows/v1
- proto_path: google/cloud/workflows/v1beta

- api_shortname: workspaceevents
name_pretty: Google Workspace Events API
product_documentation: https://developers.google.com/workspace/events
api_description: The Google Workspace Events API lets you subscribe to events and manage change notifications across Google Workspace applications.
rest_documentation: https://developers.google.com/workspace/events/reference/rest
GAPICs:
- proto_path: google/apps/events/subscriptions/v1

- api_shortname: workstations
name_pretty: Cloud Workstations
product_documentation: https://cloud.google.com/workstations
Expand Down