Migrate the CI workflows to the new runners on EKS cluster

[hussein-awala]

This PR updates the labels used in runs-on to migrate the workflows to the new GHA runners running on Airflow EKS cluster, and managed by an Actions Runner Controller (ARC).

In the new cluster, we can choose between 3 different sizes for the self hosted runners (the instance type could be change):

• small -> t4g-medium (2 cpus, 4Gb of ram) -> runs-on: ${{ fromJson(needs.build-info.outputs.runs-on-small) }}
• medium -> t4g-xlarge (4 cpus, 16Gb of ram) -> runs-on: ${{ fromJson(needs.build-info.outputs.runs-on-medium) }}
• large -> t4g-2xlarge (8 cpus, 32 Gb of ram) -> runs-on: ${{ fromJson(needs.build-info.outputs.runs-on-large) }}

And we can add new runner sets with different characteristics in the future, for ex create a specific docker image which contains some pre-installed packages and use it in specific jobs, or maybe create a new runner image from Airflow CI image and update it via the CI after updating the breeze or Airflow dependencies. (to be discussed later).

Also, this PR removes all the steps which setup python using actions/setup-python@v4, because of a bug in the action (actions/setup-python#705). As a workaround, I pre-installed python in the runner docker image, we can revert this change after fixing the bug, or keep it based on the performance of the runner.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

I have bad experiences with conditions embedded using the expreession language. It's prone to errors. Would not it be better to generate runs-on-small/medium/large as direct output of selective checks?

That has the potential that we can also base it on labels later when we connect ARC - then we could for example apply an "ARC" label to PR to test it on the ARC setup.

This has the nice property that selective checks are also very well covered by unit tests so we can easily test any logic we will apply there.

[potiuk]

Suggestion: Maybe the output should be named "selective-checks.outputs.small-runner", "selective-checks.outputs.medium-runner" etc.

That will be a nice "abstraction layer" (selective checks based on labels will choose what the "medium" runner really is.

That will also read better:

runs-on: "${{ fromJson(needs.build-info.outputs.medium-runner) }}"

[potiuk]

I am afraid this will not work if we want to merge it and would like to freely switch between the "public", "AWS" and "ARC" runners.

I would prefer to have the case where we have a way to easily switch to choose which runners we are using on a PR-by PR basis (i.e. for example apply label to make specific PR to run on ARC - similalry as we do with "use public runners") - at least for a few week transition period.

I think what can be done is that this action can be conditional based on (yes, you guessed it) selective checks output. I think when you are calling a composite action, you are able to pass it a parameter from the calling action, so we could pass "use-arc" or simlar selective check output and based on that we could either run the setup python action or not.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[potiuk]

We might want to come back to this change. It was stale for a while but probably worth keeping it until we get to changing our CI env

[hussein-awala]

We might want to come back to this change. It was stale for a while but probably worth keeping it until we get to changing our CI env

Recently I did a similar migration in a personal project, I will come back to this one in two weeks with a much better version.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.