Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Download node binary in GH action #22683

Merged
merged 1 commit into from May 14, 2024

Conversation

yhwang
Copy link
Member

@yhwang yhwang commented May 7, 2024

Description

Add a script to download node/yarn binary in GH action with retry feature to prevent the unstable connection when downloading the node/yarn binary in the Maven build.

fixed: #22676

Motivation and Context

Use the retry mechanism to avoid the unstable connection when building the presto-main

Impact

In the GH action workflows, run the script with the maven cache population.

Test Plan

Verify the CI runs for this PR

Contributor checklist

  • Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
  • PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
  • Documented new properties (with its default value), SQL syntax, functions, or other functionality.
  • If release notes are required, they follow the release notes guidelines.
  • Adequate tests were added if applicable.
  • CI passed.

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== NO RELEASE NOTE ==

@yhwang yhwang requested a review from a team as a code owner May 7, 2024 05:20
@yhwang yhwang requested a review from presto-oss May 7, 2024 05:20
@yhwang yhwang force-pushed the ghaction-download-node branch 2 times, most recently from 0012c7c to 762777d Compare May 7, 2024 05:38
@yhwang
Copy link
Member Author

yhwang commented May 7, 2024

this CI run: https://github.com/prestodb/presto/actions/runs/8980256103/job/24663568049?pr=22683 hit the cache.
let me comment out part of the script to run the node/yarn binary download logic. I will uncomment the code after verification.

@yhwang yhwang force-pushed the ghaction-download-node branch 2 times, most recently from 8154be5 to dc910aa Compare May 7, 2024 05:50
@yhwang
Copy link
Member Author

yhwang commented May 7, 2024

this CI job: https://github.com/prestodb/presto/actions/runs/8980341932/job/24663806501?pr=22683 verified the download_nodejs code. No maven cache and download_nodejs did retrieve node/yarn and store them into the maven cache. in the Maven Check job, the frontend plugin uses the binary files from the maven cache to install node/yarn.

ready to uncomment the code and revert the pom.xml change.

Add a script to download node/yarn binary in GH action with retry
feature to prevent the unstable connection when downloading
the node/yarn binary in the Maven build.

Signed-off-by: Yihong Wang <yh.wang@ibm.com>
@yhwang yhwang changed the title [WIP] Download node binary in GH action Download node binary in GH action May 7, 2024
Copy link
Contributor

@elharo elharo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any possibility we could load everything including node from the Maven repository system instead? This separate load mechanism has proved to be a large problem for those of us behind corporate firewalls.

@yhwang
Copy link
Member Author

yhwang commented May 7, 2024

@elharo unfortunately, node/yarn is not in the Maven repo. and I agree with you about the corporate firewall. The same issue also applied to Docker Hub or other services with rate limits based on IP address. As for node and yarn, they shouldn't be blocked by the corporate firewall since most of the developers need them. I believe corporates should have proxy/caching for those tools.

The solution here is for GH action and that's why the script is put under the .github/bin directory. Based on the current workflows, if the Populate maven cache job fails, the workflow will fail. Downloading the node/yarn and putting them into the Maven local cache with retry should avoid the unstable connection issue. Once the Populate maven cache succeeds, the cache could be reused by other workflows.

@tdcmeehan tdcmeehan merged commit 1863d80 into prestodb:master May 14, 2024
56 checks passed
@yhwang yhwang deleted the ghaction-download-node branch May 14, 2024 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Multiple modules failing in CI [Could not download Node.js]
3 participants