Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GitHub action to test PR on coredev. #3540

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

mauritsvanrees
Copy link
Sponsor Member

@mauritsvanrees mauritsvanrees commented May 23, 2022

This is a gitHub Action for testing a package PR on buildout.coredev.
For now (May 2023) it is an experiment, but I think we need it.
Reasons:

  1. The robot tests on Jenkins are flaky/unstable, which most likely is due to several robot jobs running parallel on one node.
  2. Python 3.10 is not working with the 'Python Shining Pandas' plugin we use on Jenkins. We had it running with a different script for a while, but these changes got lost. Should be restorable, but let's try GHA.

One thing to watch out for, is that robot tests are not always reported as failures.
On the one hand there is config for that on Jenkins:
plone/jenkins.plone.org#297
On the other hand I see the same problem locally:
#3537 [update: fixed]

TODO: put this in a reusable workflow in plone/.github:
https://docs.github.com/en/actions/using-workflows/reusing-workflows
But first we should see if it actually works.

Aftwerwards, we could install it in all relevant Plone repos using https://github.com/asottile/all-repos
But nothing too hasty. :-)

Note: I regularly force push this branch.

@mauritsvanrees mauritsvanrees marked this pull request as ready for review May 23, 2022 09:47
@mauritsvanrees mauritsvanrees marked this pull request as draft May 23, 2022 09:48
@mauritsvanrees mauritsvanrees force-pushed the test-coredev branch 20 times, most recently from 6dda00b to 4978076 Compare May 23, 2022 21:02
@mauritsvanrees
Copy link
Sponsor Member Author

@fredvd @gforcada What do you think of this approach as (partial) alternative to Jenkins? For starters we could move the robot tests and the 3.10 tests to GHA.
We should definitely look at reusable workflows. With a bit of luck, the yaml file per repository only needs about five lines. I have not tried this yet though.

Note that we suffer from some unstable folder contents tests, just like Jenkins.

mauritsvanrees added a commit to plone/.github that referenced this pull request May 26, 2022
- Run robot tests on all supported Python versions.
- Run all Python 3.10 tests.

Copied from my draft PR plone/Products.CMFPlone#3540
I will change that one to use this new reusable workflow.
@mauritsvanrees
Copy link
Sponsor Member Author

I have moved to a reusable workflow, which is easy to do, once you realise the workflow must be in a specific directory. Now the yaml file in this repo is basically just five lines. We can make a starter workflow to make that even easier.
Tests are currently running.

@mauritsvanrees
Copy link
Sponsor Member Author

If we need something to replace parts of mr.roboto, the GitHub CLI is available in GitHub Actions. You can for example use it to add a comment, like we do with "To start Jenkins jobs, copy-paste the next comment.".
I use it locally for simple things like getting a list of open issues and PRs before I make a release of a package.

@gforcada
Copy link
Sponsor Contributor

I still did not have time to look at it properly, but my fear, of having to duplicate lots and lots of configuration is down to only 5 lines, that sounds quite good to me ✨ 💯

The idea then is to get every repository to run its tests on its own as well as the whole test suite? 🤔

For simple PR that sounds enough, if there are several PRs working together, that might be a bit more problematic, but I guess we will find a solution 👍🏾

As for 3.10, sorry, indeed we had it working, but somehow it got lost, it should be easy, as in directly calling python3.10 and creating a venv manually, so it should be fairly quick to get it back to work, unfortunately I'm rather busy for the next few weeks still 😖

@mauritsvanrees
Copy link
Sponsor Member Author

For multiple related PRs we could create a plip config in coredev.

This is a gitHub Action for testing a package PR on buildout.coredev.
For now (May 2022) it is an experiment, but I think we need it.
Reasons:

1. The robot tests on Jenkins are flaky/unstable, which most likely is due to several robot jobs running parallel on one node.
2. Python 3.10 is not working with the 'Python Shining Pandas' plugin we use on Jenkins.  We had it running with a different script for a while, but these changes got lost.  Should be restorable, but let's try GHA.

One thing to watch out for, is that robot tests are not always reported as failures.
On the one hand there is config for that on Jenkins:
plone/jenkins.plone.org#297
On the other hand I see the same problem locally:
#3537
But those two problems seem solved by now (June 2022).

We use a reusable workflow in [`plone/.github`](https://github.com/plone/.github/blob/main/reusable-workflows/coredev.yml).

Aftwerwards, we could install it in all relevant Plone repos using https://github.com/asottile/all-repos
But nothing too hasty. :-)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants