Run tc39/test262 against Babel

[xtuc]

Context

The idea is to run tc39/test262 tests suite against Babel.

Running the official spec tests against Babel can make sure we comply to the spec or at least know when we don’t. We’ll want to figure out how to do filtering on things we can’t compile (proxy, TCO, etc) and set up an easy way to check failing tests and file issues and PRs for them.

This issue will be used to track this.

Previous discussions:

• test262 conformance as an line item the compat table compat-table/compat-table#830
• https://github.com/bakkot/test262-web-runner/
• Steps to run test262 with Babel bterlson/test262-harness#58
• https://github.com/motiz88/babel-smoke-tests

Possible Solution

• https://github.com/bterlson/test262-harness (it seems pretty straightforward, replace node with babel-node ?)

I think we should use our own test runner. We might want to skip some tests.

There are some restriction in order to run the tests. They are well explained in https://github.com/tc39/test262/blob/master/INTERPRETING.md#test-execution.

The tests could be run automaticly on the CI (they are taking some time).

/cc @kangax

---

Edit: 11/12

My test is available here: https://github.com/xtuc/babel-test262.

Some observations I made:

• Running all tests takes too long (build log)
• Mocha reporter and only the language built-ins tests (build log) and this time the ouput was too long.
• Mocha dot reporter and only the language built-ins tests seems to work pretty well (build log)

[kangax]

I did this couple months ago when testing Babili. Here's my script to run from within chakra-core repo — https://gist.github.com/kangax/d4e96384bdce3528d41808ae1843df14

Here's one of the issues I filed back then — #4756

[xtuc]

The project now allows to run tests with given Babel transformations (for convienience I choose to use babel-preset-env).

I wanted it to be as simple and flexible as possible. See the README to see its usage.

I used an environement variable to configure the target browser.

TARGET_BROWSERS="last 200 versions"

Please tell me if this make sense to you and/or way to improve it.

Example build job: https://travis-ci.org/xtuc/babel-test262/builds/184788391.

TODO

• @kangax, would this suffice to run your tests on Babili?
• Define tests
  • Target browsers? Polyfill? Plugins? Presets?
  • Which test suite in test262?
• Could it be integrated within Babel's CI?
• There are still some bugs with the runner

[kangax]

@xtuc looking good! For our use case, one simple thing we could try as Babili smoke test is to try running jQuery unit tests after transpiling jQuery with Babili. It's a small/simple repo with easy to run unit tests so should be a good starting point.

[gsathya]

@littledan can our test262 test infra be used by babel or is it tightly coupled with our CI?

[hzoo]

I know there's https://github.com/bakkot/test262-web-runner and https://github.com/bterlson/test262-harness too but haven't looked into it that much

[littledan]

@gsathya It would take some infrastructure work to hook everything up. I'm not sure what you'd get out of V8's infrastructure. I'd recommend starting with either the web runner or the official Node-based command-line runner.

[chicoxyzzy]

I didn't see this issue earlier. I ❤️ the idea and will be happy to help if I can.

Also we can add babel/babel-preset-env#117 to discussions in first message

[gsathya]

I'm not sure what you'd get out of V8's infrastructure.

Mostly the harness -- ability to skip tests + cli

[littledan]

@gsathya I think V8's infrastructure for this is a bit heavy for this purpose. It would probably be simpler to rebuild something different than use that.

[hzoo]

Babylon has this now courtesy of @jugglinmike babel/babylon#654, will need to do something else for Babel transforms

[hzoo]

Update: working via babel/babel-test262-runner and we also have #10500 as a per to test in circle. We can ask about being adding to http://test262.report too

[hzoo]

Via #10579 we are able to run the language tests in 30m, so I think we can close this and make a new issue for next steps (some ideas):

• Visualize the diff/failures via a website, maybe link those as potential bugfix/issues for contributors (depending on the test, as some of them we may want to keep as wontfix), or something like test262.report
• Use this data for preset-env/compat-table?
• Use the data to figure out how to better explain what a loose/spec mode really means with the descriptions and actual tests that pass/fail

[jbhoosreddy]

Yeah.. I think this issue is good to be closed.

I do have a couple minor PRs outstanding:

1. local testing workflow: Add local babel workflow babel-test262-runner#25
2. better comparision strategy: Compare different lengths of test results babel-test262-runner#24
3. docs for maintainers: Add Debugging test262 errors in CI document babel-test262-runner#22

But they don't require this issue to remain open.

[hzoo]

Cool closing in favor of #10734 and we can add our new ideas there! Really happy this landed and hope everyone else is too 🙂!