Implement PR workflow for running test262 on babel PRs

[jbhoosreddy]

This PR introduces testing of babel transforms with test262 in every PR on each commit. It is a followup to #10556 which runs test262 tests on every commit to master. This PR utilizes changes to babel-test262-runner, here: (babel/babel-test262-runner@399abd...b5eaa57)

A major aspect of the PR workflow is that it compares the test results on a PR against the test results on master. This is because there are frankly too many tests and many esoteric failing tests that to present them to a new contributor seems like an excessive barrier to entry for new contributions. Therefore, we have pivoted to compare the results from the previous master job and show only the results that have changed.

Due to @hzoo's suggestion to focus on testing only /language tests, and @JLHwung's babel config caching PR babel/babel-test262-runner#12, we're able to run the tests in about 30m. This. while being a lot, is comparable to the existing job on TravisCI.

I also want to highlight authors' of some really cool and helpful packages that I was able to utilize which made this task much easier than implementing from scratch!

• tap-mocha-reporter
• tap-merge
• make-tap-output
• highland
• @tap-format/parser

Also, also props to @nicolo-ribaudo for the quick turnaround on many PR reviews and help throughout this process!

Q	A
Fixed Issues?	Relates to #4987
Patch: Bug Fix?
Major: Breaking Change?
Minor: New Feature?
Tests Added + Pass?	Yes
Documentation PR Link
Any Dependency Changes?
License	MIT

[buildsize]

File name	Previous Size	New Size	Change
babel-preset-env.js	2.78 MB	2.78 MB	9 bytes (0%)
babel-preset-env.min.js	1.67 MB	1.67 MB	9 bytes (0%)
babel.js	2.96 MB	2.96 MB	9 bytes (0%)
babel.min.js	1.63 MB	1.63 MB	9 bytes (0%)
test262.tap	9.86 MB	[deleted]

[jbhoosreddy]

There is an existing travis CI job which takes 21m to finish. Running all test262 language tests average 30 minutes. Given that, I'm inclining towards running them on every commit instead of requiring maintainers to trigger it manually. Thoughts @hzoo @nicolo-ribaudo ?

[jbhoosreddy]

The test262 failure with 2 failure tests is a bit concerning as a) my PR shouldn't cause any existing tests to fail and b) it wasn't failing in my previous approach of syncing with master @JLHwung. I think we might need to explore this space a little bit more, or potentially I'm not using it correctly.

So I'm going to experiment by leaving the changeset as is, and manually rebase against master.

Edit: I should have merged instead of rebase. I lost the status check in the PR but here's the link to the job I am mentioning here: https://app.circleci.com/jobs/github/babel/babel/12410

[jbhoosreddy]

While this is running, I do want to note, that I don't think that I've completely mapped out all the error scenarios that can happen (while I've tried to tackle a lot of them in my various babel-test262-runner PRs), and when you factor in the mechanics of running about 40,000 tests on two different branches, and situations like test case timeouts, and other weirdness, and then comparing them together, I can't guarantee that this MVP will not have false negatives. Certainly this can be improved over time.

Given the seemingly daunting difficulty in that, I've pivoted my approach to sort of how can this be a useful source of information for the maintainers. While it's a bit disappointing not to always get all green ticks (although I think it'll be green most of the time), I don't think failing the test262 job (in its current form) should block a PR, but rather be a signal for maintainers and contributors to inspect the results and see if they are related or not and make a judgement based on that.

We could circle back to running the test262 job only on maintainer approval. Or we can explore other options. I'll defer to the maintainers' judgement on what they'll find most helpful here. @nicolo-ribaudo @JLHwung @hzoo

[nicolo-ribaudo]

@jbhoosreddy Would it be possible to run this workflow automatically on master, and with approval on PRs?

Running on master should always be safe, and it won't run on PRs by default since it's not guaranteed that it should pass.

[jbhoosreddy]

@jbhoosreddy Would it be possible to run this workflow automatically on master, and with approval on PRs?

Running on master should always be safe, and it won't run on PRs by default since it's not guaranteed that it should pass.

Yes, absolutely! If that's the consensus, I'll update the PR to do that then.

[JLHwung]

Great work!

[nicolo-ribaudo]

Thanks 🎉