GH-325: [Website] Update website deployment script to use latest LTS version of Node.js and Webpack 5.75.0 #326

kevingurney · 2023-03-02T11:58:41Z

Overview

This pull request modifies the apache/arrow-site website deployment workflow (.github/workflows/deploy.yml) to use the latest LTS version of Node.js and Webpack 5.75.0 to work around the build issue described in #325.

Qualification

To qualify these changes, I:

Submitted these changes to the main branch of the mathworks/arrow-site fork in order to trigger the gh-pages deployment workflow. I then selected gh-pages as the GitHub Pages deployment branch and verified that the site was deployed as expected to https://mathworks.github.io/arrow-site/. For an example of a successful workflow run, see: https://github.com/mathworks/arrow-site/actions/runs/4313253336/jobs/7524824999.
I inspected the GitHub Actions workflow steps to ensure there are no errors.

Future Directions

While qualifying with the fork deployment workflow, I realized that I needed to manually change the GitHub Pages deployment branch from asf-site to gh-pages in the "Pages" settings of the mathworks/arrow-site fork. This wasn't immediately obvious, and it isn't listed explicitly as a required step in the README.md of apache/arrow-site. It would helpful to add an explicit note about this step. I've captured this as [Website] Add note about the need to set GitHub Pages deployment source branch to gh-pages when previewing website changes on a fork of apache/arrow-site #327 and addressed it with PR GH-327: [Website] Add note about the need to set GitHub Pages deployment source branch to gh-pages when previewing website changes on a fork of apache/arrow-site #328.
As described in the "Workarounds" section of the description of [Website] Website deployment workflow (deploy.yml) is failing due to Node.js 18 version bump in ubuntu-latest GitHub Actions runner image and Webpack usage of md4 hashing algorithm #325, there is still more we could choose to do to address the root cause of these build failures (the deprecation of the md4 hash algorithm in Node 18). This would include setting the output.hashFunction to xxhash64 for Webpack.
We could move the workflow into a container to make it easier to reproduce the website build process on a local machine (see the discussion in the comments on this pull request).

Notes

Thank you @sgilmore10 for your help with this pull request!
Thank you to @avantgardnerio for your suggestion to move the deployment workflow inside of an ubuntu:latest container!

Closes #325.

Co-authored-by: Sarah Gilmore <sgilmore@mathworks.com>

raulcd

Thanks @kevingurney for the PR!

.github/workflows/deploy.yml

avantgardnerio

LGTM!

.github/workflows/deploy.yml

kou

We use Node.js 12 on Ubuntu 22.04 with this change, right?

https://github.com/apache/arrow-site/actions/runs/4314961494/jobs/7528685953#step:13:17

npm WARN EBADENGINE   current: { node: 'v12.22.9', npm: '8.5.1' }

Node.js 12 is an EOL-ed Node.js. So it looks like that this is similar to 1. in GH-325.

It seems that 2. in GH-325 can be implemented with the following:

$ npm install webpack # this updates package.json and package-lock.json

If it works, it seems that it's a straitforword approach.

.github/workflows/deploy.yml

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

…e package list. Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

…de`.

…higher.

…sion.

…14.2 or higher.

kou · 2023-03-03T08:56:35Z

.github/workflows/deploy.yml

@@ -27,17 +27,31 @@ jobs:
  deploy:
    name: Deploy
    runs-on: ubuntu-latest
+    container:
+      image: ubuntu:22.04 # Ubuntu 22.04 is the latest LTS version as of April 21, 2022: https://wiki.ubuntu.com/Releases


Can we try reverting this and changes related to this? (e.g.: explicit apt-get, ImageOS: ubuntu22 and so on)

If we can use the ubuntu-latest image directly, we can use simpler deploy.yml and don't need to maintain Ubuntu version for this.

I'm OK with just using ubuntu-latest and no container. However, I know there were some concerns about this approach expressed by @avantgardnerio (e.g. the proprietary nature and lack of debuggability of ubuntu-latest).

However, I can try this approach and if no one else in the community has reservations about it, then I am OK with it.

I'm testing out the workflow with no container now on https://github.com/mathworks/arrow-site/tree/GH-325-no-container.

Ah, I can understand the debuggability concern. If we make this job local debuggable, we should not use actions/* in ubuntu:XXX container like we did in apache/arrow: https://github.com/apache/arrow/blob/main/.github/workflows/cpp.yml#L92-L99

If we need local debuggability, we may need to choose another approach.

Thanks.
It seems to work. I prefer the with container approach to the no container approach but I want to hear opinions from others.

Thanks for confirming this.

How about the following steps to improve the current situation?

We merge "no container" version https://github.com/mathworks/arrow-site/tree/GH-325-no-container because it's simpler than this version

We create a Dockerfile based on ubuntu:22.04 and use it in CI and https://github.com/apache/arrow-site#using-docker (This prevents breaking @alamb 's use case accidentally)

We extract command lines used in 2. as a shell script and fine tune the shell script to work on host Ubuntu 22.04 too (@avantgardnerio 's use case)

Thanks @avantgardnerio and @kou for your sharing your thoughts! I agree. This sounds like a reasonable path forward.

Just to clarify:

"We create a Dockerfile based on ubuntu:22.04 and use it in CI"

By "use it in CI" do you mean that you would like the GitHub Actions workflow to run inside of a Docker container based on this Dockerfile? If so, doesn't this conflict with 1. (which is to use the "no container" workflow)?

Sorry for the confusion. I just want to make sure we are all on the same page before moving forward.

My idea is that we just use docker build && docker run in GitHub Actions instead of container:.

(More specifically, we will use docker/build-push-action instead of raw docker build and push a built image to ghcr.io to reuse the built image on local for https://github.com/apache/arrow-site#using-docker . https://github.com/groonga/groonga-delta/blob/main/.github/workflows/docker.yml is a bit old but may help you to understand my idea.)

If my explanation isn't enough, please ask me more!

Thanks for clarifying @kou! This make sense!

I'll start working on these changes now.

alamb · 2023-03-07T14:35:46Z

Are we happy with this PR at the moment? Any concerns about merging it ?

kou · 2023-03-07T20:51:59Z

Ah, sorry. I forgot to left a comment. I'll left a comment soon.

avantgardnerio · 2023-03-07T21:09:28Z

rgot to left a comment. I'll

I'm happy... I'll wait a little bit for @kou

kou · 2023-03-07T21:18:02Z

Commented: #326 (comment)

kou

How about revert ci/ changes for now and merge this pull request to fix the current CI failure?
We can work on the step 2. with ci/ changes as a follow-up pull request.

kevingurney · 2023-03-09T10:56:40Z

Sounds good to me. I'll revert the ci/ changes.

I've created a follow up issue #331 to capture the desire to improve the container workflow.

kevingurney · 2023-03-09T10:58:23Z

I've reverted the changes and updated the PR title and description to more accurately reflect the changes.

kou

+1

Thanks!

Run website deployment workflow inside of an ubuntu:latest container.

a1dfe9b

Co-authored-by: Sarah Gilmore <sgilmore@mathworks.com>

raulcd reviewed Mar 2, 2023

View reviewed changes

.github/workflows/deploy.yml Outdated Show resolved Hide resolved

Use v3 of actions/checkout.

650e625

avantgardnerio approved these changes Mar 2, 2023

View reviewed changes

kou reviewed Mar 2, 2023

View reviewed changes

.github/workflows/deploy.yml Outdated Show resolved Hide resolved

.github/workflows/deploy.yml Outdated Show resolved Hide resolved

.github/workflows/deploy.yml Outdated Show resolved Hide resolved

.github/workflows/deploy.yml Outdated Show resolved Hide resolved

kou reviewed Mar 2, 2023

View reviewed changes

.github/workflows/deploy.yml Outdated Show resolved Hide resolved

kevingurney and others added 11 commits March 3, 2023 01:56

Use ubuntu:22.04, rather than ubuntu:latest as the container image.

1bd48be

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

Update name of workflow step that calls apt-get update to update th…

16c8a11

…e package list. Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

Use v3 of actions/setup-node.

a842667

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

Delete bundle install step.

e240ebe

Change .nvmrc to use lts/hydrogen as Node.js version.

60b85ad

Use .nvmrc file with node-version-file input to `actions/setup-no…

a526225

…de`.

Update Webpack version to 5.75.0.

6f9568f

Increase Node.js 'engines' requirement in package.json to 18.14.2 or …

22361aa

…higher.

Use 2 spaces for indentation.

6d30aab

Add comment about Ubuntu 22.04 being the the latest available LTS ver…

998bb83

…sion.

package-lock.json update Node.js 'engines' requirement to version 18.…

80a5274

…14.2 or higher.

kou reviewed Mar 3, 2023

View reviewed changes

kevingurney added 2 commits March 3, 2023 04:03

Replace lts/hydrogen with lts/* in .nvmrc.

067009b

Remove container from website deployment workflow.

41a0080

kou reviewed Mar 8, 2023

View reviewed changes

kevingurney changed the title ~~GH-325: [Website] Use the official Ubuntu container for deployment workflow to use old Node.js~~ GH-325: [Website] Upgrade website deployment script to use latest Node.js LTS and Webpack 5.75.0 Mar 9, 2023

kevingurney force-pushed the GH-325 branch from bcb73ea to 41a0080 Compare March 9, 2023 10:57

kevingurney changed the title ~~GH-325: [Website] Upgrade website deployment script to use latest Node.js LTS and Webpack 5.75.0~~ GH-325: [Website] Update website deployment script to use latest Node.js LTS and Webpack 5.75.0 Mar 9, 2023

kevingurney changed the title ~~GH-325: [Website] Update website deployment script to use latest Node.js LTS and Webpack 5.75.0~~ GH-325: [Website] Update website deployment script to use latest LTS version of Node.js and Webpack 5.75.0 Mar 9, 2023

kevingurney mentioned this pull request Mar 9, 2023

[Website] Create a custom Docker container for running the website build and deployment steps #331

Open

kou approved these changes Mar 10, 2023

View reviewed changes

kou merged commit b955d79 into apache:main Mar 10, 2023
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-325: [Website] Update website deployment script to use latest LTS version of Node.js and Webpack 5.75.0 #326

GH-325: [Website] Update website deployment script to use latest LTS version of Node.js and Webpack 5.75.0 #326

kevingurney commented Mar 2, 2023 •

edited

raulcd left a comment

avantgardnerio left a comment

kou left a comment

kou Mar 3, 2023 •

edited

kevingurney Mar 3, 2023 •

edited

kevingurney Mar 3, 2023 •

edited

kou Mar 3, 2023

kou Mar 3, 2023 •

edited

kou Mar 7, 2023

kevingurney Mar 8, 2023

kou Mar 8, 2023

kou Mar 8, 2023

kevingurney Mar 8, 2023

alamb commented Mar 7, 2023

kou commented Mar 7, 2023

avantgardnerio commented Mar 7, 2023

kou commented Mar 7, 2023

kou left a comment

kevingurney commented Mar 9, 2023 •

edited

kevingurney commented Mar 9, 2023 •

edited

kou left a comment

GH-325: [Website] Update website deployment script to use latest LTS version of Node.js and Webpack 5.75.0 #326

GH-325: [Website] Update website deployment script to use latest LTS version of Node.js and Webpack 5.75.0 #326

Conversation

kevingurney commented Mar 2, 2023 • edited

Overview

Qualification

Future Directions

Notes

raulcd left a comment

Choose a reason for hiding this comment

avantgardnerio left a comment

Choose a reason for hiding this comment

kou left a comment

Choose a reason for hiding this comment

kou Mar 3, 2023 • edited

Choose a reason for hiding this comment

kevingurney Mar 3, 2023 • edited

Choose a reason for hiding this comment

kevingurney Mar 3, 2023 • edited

Choose a reason for hiding this comment

kou Mar 3, 2023

Choose a reason for hiding this comment

kou Mar 3, 2023 • edited

Choose a reason for hiding this comment

kou Mar 7, 2023

Choose a reason for hiding this comment

kevingurney Mar 8, 2023

Choose a reason for hiding this comment

kou Mar 8, 2023

Choose a reason for hiding this comment

kou Mar 8, 2023

Choose a reason for hiding this comment

kevingurney Mar 8, 2023

Choose a reason for hiding this comment

alamb commented Mar 7, 2023

kou commented Mar 7, 2023

avantgardnerio commented Mar 7, 2023

kou commented Mar 7, 2023

kou left a comment

Choose a reason for hiding this comment

kevingurney commented Mar 9, 2023 • edited

kevingurney commented Mar 9, 2023 • edited

kou left a comment

Choose a reason for hiding this comment

kevingurney commented Mar 2, 2023 •

edited

kou Mar 3, 2023 •

edited

kevingurney Mar 3, 2023 •

edited

kevingurney Mar 3, 2023 •

edited

kou Mar 3, 2023 •

edited

kevingurney commented Mar 9, 2023 •

edited

kevingurney commented Mar 9, 2023 •

edited