Testing infrastructure – backlog

[borekb]

With #1389, the most pressing issues with testing have been fixes. This is a backlog of other items that we should do in the near future:

• Only enable Xdebug for ...:debug scripts. Right now, it's always enabled and just not configured with the right idekey in non-debug scripts which makes things slower. Here and here, also mentioned here: Dockerized dev setup #1227 (comment).
• Update Xdebug to 2.7.0.
• Rename images to something like versionpress/wordpress and versionpress/cli, i.e., make "CLI" part of the image name so that it's clear when digests are used.
• Review WP-CLI installation into the WP image, it looks a bit weird.
• Update PHPUnit. Depends on minimum required version bump, see Bump required PHP version to 7.2 #1408.
• Don't base our CLI image on wordpress:cli, it does a lot more than we need.
• Explore re-running only failed tests: Implement "previous failures first" mode sebastianbergmann/phpunit#46
• Close Fix remaining tests under Docker, post-beta #1284 (Fix remaining tests under Docker, post-beta) and port those to a new issue
• Review backlog from the "Noted for future work" section in Dev setup updates – spring 2018 #1329
• Another big one is WP-CLI / Selenium worker switching – revised approach #1258 (WP-CLI / Selenium worker switching). That issue is a bit old but overall, we should still be able to run all tests with both workers, or pick a worker with a flag like npm run tests --worker=selenium. Things like a worker or a specific test site to use shouldn't be hardcoded in test-config.yml.
  • Related: How to parametrize PHPUnit tests.