Requirements for scoring results from Test262

[foolip]

I promised in #110 that I would explain what we'd need to score results from Test262 as part of Interop 2023, so that @bkardell can investigate.

My knowledge of Test262 is very cursory, but I think that only engines (V8, SpiderMoney, JSC, etc.) are tested and not those engines as integrated into browsers (Chrome, Firefox, Safari, etc.). I don't know what the results format is, but https://test262.report/ shows results and it's probably possible to work backwards from there to find sample report files.

To score results from Tests262, we'd ideally want test results from the same browser releases that the rest of tests (in WPT) are run again. And we'd want the results format to be compatible with WPT's wptreport.json format, with either a test or test/subtest structure that can be scored in the same fashion.

The most straightforward way of achieving this might be some kind of adapter to run all of Test262 as part of ./wpt run and get results in the same format. I think that @Ms2ger has been involved in discussions around something like that a few years ago.

[foolip]

I found the prior work/discussions that I vaguely remembered:

• Testing test262 in different (web platform) agents wpt#8308
• Script that creates web-platform-test wrappers for test262's tests wpt#8980
• Experiment: run Gecko's mochitests with some adapter code wpt#19438

@annevk and @jgraham were somewhat involved with that.

[foolip]

Having looked at web-platform-tests/wpt#8980, I think a path forward here could be:

• Revive Script that creates web-platform-test wrappers for test262's tests wpt#8980. This will require an RFC, and I expect discussion about how to make these tests optional, probably not run by default.
• Configure the runner infrastructure for wpt.fyi to run Test262.
• Done?

[Ms2ger]

Hmm, would we check in the results of the script somewhere? That seems suboptimal. I also wonder if we could use .any.js to reduce the amount of work done in the script.

[foolip]

Given how many tests there are I'd also prefer to not check them in. That also avoids license questions. It would make more sense to me as either a build step, or a feature of wptrunner to generate the right responses on the fly. Some change to the manifest is also needed.

[foolip]

I've tried reviving the previous attempt in web-platform-tests/wpt#35621. It doesn't work yet, tests fail because assert isn't defined.

[foolip]

Interop 2023 launched yesterday, and we ended up not needing any integration with Test262.

[jgraham]

Note that this isn't quite true, we ended up with a one-off port of the backreference tests. If we want to accept features that are only tested in test262 going forward then we should figure out a better plan for integrating the suites.

[gsnedders]

Let's keep this open as something we should probably try to solve over the following year (or at least decide whether or not to solve!), so we're not hastily trying to do this in the run up to Interop 2024.

[foolip]

We didn't have any proposals in Interop 2024 that needed Test262, and we didn't start an investigation effort for it. We did however start an investigation effort for WebAssembly testing, which might pave the way for something here.