Feature: Using test262 CI data

[hzoo]

Feature Request: Using our newly added test262 support!

test262 is the Official JS conformance tests
Mike just wrote a great post about the work behind bringing test262 support to JS parsers (we added it back in 2017).
"If JavaScript parsers were also using the project, not only would they become more correct, but they would be better-positioned to contribute tests of their own, so everyone else would benefit, too."

Well many of the JS tools don't run test262 either, and this is just the first step for us (it would be a great community effort to bring this to other tools like TypeScript or bundlers like Webpack)

We can close out #4987 (3 years later) since via #10579 (thanks to amazing work of @jbhoosreddy and many others) we are able to run the /language syntax tests of the official JavaScript test suite (called test262) on CI per PR (takes 30min). We had landed

We are using https://github.com/babel/babel-test262-runner and CircleCI artifacts to do it (do a diff on the results)!

This is more of a tracking issue to think about some ideas towards what we want to do with this moving forward (if there is something interesting maybe someone can writeup a more in-depth issue to track and link it here):

• The results can be difficult to sort through (especially since there are so many), so a way to view them ideally on a website (or even queryable as an API type service) would be great! Similar to test262.report or a https://isfiberreadyyet.com/ type thing. I don't think 100% spec compliancy is a realistic goal (browsers aren't either) but we definitely want to know what we are missing.
• A realistic goal in terms of project roadmap vision is to at least cover all the tests that do pass in the the majority of browsers (example: so if a test fails in Babel but not in say v8/spidermonkey/webkit we should look into fixing it for compat as priority vs. a test that others fail at as well)
• Long term: maybe create a more in-depth version of compat-table through test262 data that would be used for preset-env? This would also help inform the kind of bugs we want to fix in preset-env and the new preset-modules that @developit started
• Similarly: use the data to be more clear/specific about what loose/spec mode means (accurately depicting the examples/tests that fail/pass due to options?)

[jbhoosreddy]

To add more technical information here, for future readers' sake:

We're storing the test262 result artifacts in tap format in the form of test262.tap file on master or PR jobs' artifact.

If you want to get the latest master branch results, you can use https://github.com/babel/babel-test262-runner command

npm i
node lib/download-master-artifact <optional file path>

The flat tap format is convenient if in the future we might want to compress the data for transport. But in either case, there are various parsers which can give you a simple parsed JSON structure to work with. Examples include @tap-format/parser and tap-parser.. In fact, babel-test262-runner uses these under the hood.

Another note, it may help to additionally save only the errors as that significantly reduces the size of data to deal with. We could potentially start saving them in CI and working off of that.