Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
CI: switch to GitHub Actions - step 6: PHP integration tests
This commit:
* Adds a GH Actions workflow for the PHPUnit Integration test CI check.
* Removes all references to that check from the `.travis.yml` configuration.
* Replaces the "Build Status" badge in the Readme with a badge using the results from the GH `IntegrationTest` Action runs.
All other workflows have their own status badges, so the Travis badge can now be safely removed.
Notes:
1. Builds will run on:
- Select pushes using branch filtering similar to before.
- All pull requests.
- When manually triggered.
Note: manual triggering of builds has to be [explicitly allowed](https://github.blog/changelog/2020-07-06-github-actions-manual-triggers-with-workflow_dispatch/). This is not a feature which is enabled by default.
2. If a previous GH actions run for the same branch hadn't finished yet when the same branch is pushed again, the previous run will be cancelled.
In Travis, this was an option on the "Settings" page - "Auto cancellation" -, which was turned on for most, if not all, Yoast repos. The `concurrency` configuration in the GHA script emulates the same behaviour.
3. In contrast to various other scripts, the matrix for these integration tests is explicitly set with all variables explicitly defined for each build, as we want full control over the three variables (`php_version`, `wp_version` and `multisite`) used for each run.
4. As a MySQL database is needed, we need to explicitly add that service.
The MySQL version used varies based on the PHP version against which the tests are run for two reasons:
1. It's good practice to test against multiple different MySQL versions to verify that any queries run are MySQL cross-version compatible.
2. While WP is _supposed to_ support MySQL 8.0 since WP 5.4, in actual fact, WordPress doesn't fully support MySQL 8.0 on PHP < 7.4.
Also see: https://core.trac.wordpress.org/ticket/52496
Note: in the Travis script, the PHP 5.6 build was explicitly set to use the `trusty` dist to force the use of MySQL 5.6. In GH Actions, this is not directly linked to the Ubuntu image used. All the same, the script has been set up to explicitly use MySQL 5.6 for PHP 5.6, same as before.
5. The default ini settings used by the `setup-php` action are based on the `php.ini-production` configuration.
This means that `error_reporting=E_ALL & ~E_DEPRECATED & ~E_STRICT`, `display_errors` is set to `Off` and `zend.assertions` is set to `-1` (= do not compile).
For the purposes of CI, especially for linting and testing code, I'd recommend running with `zend.assertions=-1`, `error_reporting=-1` and `display_errors=On` to ensure **all** PHP notices are shown.
Note: the defaults will be changed in the next major of `setup-php` to be based on the `php.ini-develop` configuration, but that may still be a while off.
Refs:
* shivammathur/setup-php#450
* shivammathur/setup-php#469
6. While the `setup-php` action offers the ability to [install the PHAR file for PHPUnit](https://github.com/shivammathur/setup-php#wrench-tools-support), I've elected not to use that option as we need to do a `composer install` anyway to get the other dependencies, like WP Test Utils.
7. The PHPUnit version(s) supported and needed to run the tests depend on **both** the WP version used as well as the PHP version used.
As an `if` condition on a step would get pretty complicated to handle this, I've introduced a separate - fully documented - step to determine the "type" of install we need.
The output of that step is subsequently used in the `if` condition for follow-on steps to make sure the correct PHPUnit version will be installed by Composer.
8. Composer dependency downloads will be cached for faster builds using a [predefined GH action](https://github.com/marketplace/actions/install-composer-dependencies) specifically created for this purpose.
The alternative would be to handle the caching manually, which would add three extra steps to the script.
Note: Caching works differently between Travis and GH Actions.
On GH Actions, once a cache has been created, it can't be updated. It can only be replaced by a new cache with a different key.
As the PHP version, the `composer.json` and a potential `composer.lock` hash are all part of the key used by the above mentioned action, this difference should not have a significant impact.
Ref: https://docs.github.com/en/actions/advanced-guides/caching-dependencies-to-speed-up-workflows
9. As there is a committed `composer.lock` file and the PHPUnit version is locked at PHPUnit 5.x, we have to do an on-the-fly update of the PHPUnit version depending on the WP and PHP version used for the current build.
As we still also want to benefit from the Composer caching which the `ramsey/composer-install` action sets up, I've done some nifty trickery with the options passed to the `composer-install` action to get that working.
- The `dependency-versions: "highest"` basically changes the command used by the action from `composer install` to `composer update`, however we don't want to update _all_ dependencies as run-time dependencies should remain locked at their original version.
- To that end, I'm passing additional `composer-options` which will limit the update to just and only the test dependencies.
- These type of updates are only run when needed based on the "install type" determination as described in (7).
10. Installing WordPress was previously done via inline commands in the Travis script. This has now been replaced by the more comprehensive `tests/bin/install-wp-tests.sh` script as made available by the WP CLI `scaffold` command.
As WPCLI is not used anywhere else in the workflow, a copy of the script, with source annotations, is being introduced into the plugin codebase.
Note: this script needs to have the `x` flag set to allow it to be executed.
Ref: https://github.com/wp-cli/scaffold-command/blob/master/templates/install-wp-tests.sh
Some differences between the scripts:
- The plugin will no longer be copied into the WordPress install, as, to be honest, that's just not necessary with the current test setup/test bootstrap.
- The location to which WP is downloaded and installed is different between the scripts, however, this makes no difference for our purposes as the WP Test Utils library already takes the default install locations from the `install-wp-tests.sh` script into account.
- To test against the "nightly" of WordPress, the matrix needs to set the `wp_version` to `trunk` or `nightly` instead of `master` (as was previously used in some scripts).
- WordPress will be set up to always use `mysqli` for database commands, even on PHP 5.6.
Differences with the Travis implementation:
* The most appropriate version of the test dependencies will now be used for _all_ builds, not just for builds against PHP 8.0/`nightly`.
* Except for the test dependencies, no other dependencies will be updated during the test run.
* In Travis, there was a reference to a `atanamo/php-codeshift` package, which was removed after the prefixing was finished.
This package is not a (root) dependency of the WPSEO Free plugin, so removing it doesn't actually have any effect.
Instead, the `league/oauth2-client` package is removed, as that package should not be used during the test run as it will be prefixed and the prefixed version should be used instead.
* The "mix" of PHP/WP versions + Multisite used in the matrix has been adjusted slightly compared to Travis.
The current mix in GHA ensure there is a run against high + low WP on low PHP, a run against high + low WP on high PHP and a run against mulisite for all supported WP versions.
* In addition to the PHP versions against which the tests were previously run on Travis, the tests will now also be run against PHP 8.1.
As PHP 8.1 has been released, the (remaining) test failures on this build need to be fixed, though as these are only _deprecation notices_, the fact that these tests are currently failing is nothing to be too concerned about for the time being.
To prevent new failures from being introduced in the mean time, I'm not allowing the build to fail, but will instead add expectations for the deprecations or skip requirements to the few tests which still have them for the time being.
Builds against WP `trunk`, will be "allowed to fail".
Note: if a "allowed to fail" job actually fails, the workflow will show as successful, but there will still be a red `x` next to a PR.
This is a known issue in GHA: https://github.com/actions/toolkit/issues/399
There are work-arounds possible for this, however, the work-arounds would hide failures even more, meaning that chances are they won't be seen until they actually become a problem (no longer allowed to fail), which is too late.
* Instead of using Git to clone WP, the WP-CLI script is now used to retrieve the relevant parts of WP and initiate the database and the `wp-config` settings.
* There are a number of files which are usually created by Grunt and are needed during the test run.
In Travis, these files were created on the fly from within the Travis script to ensure tests would not error out on these files missing..
As this was only done in Travis, PHP focussed plugin developers who rarely use Grunt locally, would still end up with these errors when they would run the tests locally.
As the need for these files is inherent to the integration tests, creating these files from within the integration test bootstrap file will make this more stable and will allow this to work both in CI as for dev-local test runs.
* In Travis, test runs would run either as single site or as multisite. In GH Actions, each build will always run the tests against WP in single site mode and for those builds with the `multisite` flag set to `true` in the matrix, the test will **_also_** be run in multisite mode.
* In Travis, for select test runs, code coverage would be calculated and uploaded to CodeClimate.
As CodeClimate did not signal back to PRs if a PR changed the code coverage and by how much, this was not very effective, other than to keep track over time, though in practice, nobody seemed to even check on this or check on the reports at all.
With this in mind, code coverage will no longer be run, nor uploaded to CodeClimate.
The intention is to eventually run code coverage on the Jenkins server.- Loading branch information
