Aggregate test results for the E2E Tests and Tests Workflows

[munderseth]

Hi @adimoldovan, @kraftbj, @jeherve, @brbrr, this is Mark from Testspace - https://automattic.testspace.com/.

I had an idea about aggregating your test results I thought I would post in your Discussions. I hope this is the right vehicle to share my idea vs email.

Testspace now supports aggregating results spanning GitHub workflows, based on the same commit reference. You could consider aggregating all of your test results contained in E2E Tests and Tests workflows into a single results record. I believe this would provide an interesting perspective on the effect of changes on your repo.

Could be organized something like the following:

[E2E Tests] 
  [pre-connection]
  [connection]
  [post-connection]
  [gutenberg]
[PHP Tests]
  [5.6 WP latest]
  [7.0 WP latest]
   ..
[JS Tests]
  ..

To aggregate just add the action setup and push to the jobs and use folders to organize. Similar to what exists in the current E2E Tests workflow.

Soon Testspace will be offering new sorting and filtering options (i.e. slow tests, flaky groups, etc.) that could be useful when triaging.

[kraftbj]

@munderseth Thanks for reaching out! The slow and flaky could be really interesting and I'm looking forward to that.

[munderseth]

Hi @kraftbj I wanted to let you know that have recently added a few new filtering / sorting features. One example that "might" be helpful is using the Failure Cause. It groups failures based on the same failure error message.

Take a look at one of your recent results - https://automattic.testspace.com/spaces/136795/result_sets/170379/failures. Select Failure cause in the Group by.

[kraftbj]

@munderseth Thanks again for your insight and work here. We ended up implementing a dashboard on our end. See #21485

[munderseth]

Thanks for letting me know.

FYI if you are ever interested in running "optional" manual tests checkout our approach - https://help.testspace.com/manual/overview.