Externalize tests

[p-bakker]

This PR splits the tests and their results into 2 separate files, with the results being stored in JSON format, allowing automated modification, as to open up the possibility to eliminate the manual updating of test results

Additionally, this split makes it possible to create custom Compat Tables, for example with just all the different versions of one environment, including for example the nightly version. Note that are more enhancements possible for custom Compat Tables, even going as far as splitting this repo in a compat-table library and a Compat Table website (github pages) repo, but this are next steps imho

Is does mean that data-common.json is obsolete (and thus removed) as all the test results are not stored in JSON

Closes #1879
Closes #1493
Checks off on of the actions in #965
Building block for #614
Building block for #1790

Did the conversion of data-.js files > tests..js & results-*.json using the following script:

const fs = require('fs');

const extractResults = (entry, result) => {
	if (Array.isArray(entry)) {
		const mapping = {};

		entry.forEach(entry => {
			mapping[entry.name] = extractResults(entry);
		});

		return mapping;
	}

	return entry.res || extractResults(entry.subtests);
}

[
	'es5',
	'es6',
	'es2016plus',
	'esnext',
	'esintl',
	'non-standard',
].forEach(fileName => {
	console.log(`Processing 'data-${fileName}'`);

	const data = require(`./data-${fileName}`);

	// Extract just the results from the data-*.js modules, convert their format and export to result-*.json
	fs.writeFileSync(
		`results-${fileName}.json`, 
		JSON.stringify(
			extractResults(JSON.parse(JSON.stringify(data.tests))),
			null,
			'\t'
		),
		'utf8');

	// Read data-*.js, remove test results and write to tests-*.js
	fs.writeFileSync(
		`./test-${fileName}.js`,
		fs
			.readFileSync(`./data-${fileName}.js`, 'utf8')
			.replace(/,$(\s*)res *: \{.*?$\1}/gms, ''),
		'utf8'
	);

	// Check the conversion
	const tests = require(`./test-${fileName}`).tests;
	const results = require(`./results-${fileName}`);

	if (tests.length !== Object.keys(results).length) console.error(`Test mismatch: tests.length !== Object.keys(results).length (${tests.length} vs. ${Object.keys(results).length})`);

	if (!tests.every((entry, idx) => {
		return !entry.subtests || entry.subtests.length === Object.keys(results[entry.name]).length
	})) console.error(`Subtest mismatch`);
});

[p-bakker]

Would it be ok to update the min nodejs version to 14 or 18? Want to use parseArgs either natively (Node 18) or via @pkgjs/parseArgs (Node 14)

[p-bakker]

(re)marked this as Draft again, as I'm contemplating some additional changes, but would already appreciate some feedback on the general direction of extracting the results into their own json files.

Has that any change of getting accepted?

[p-bakker]

Any feedback anyone?

Don't want to spend more time on this if the general direction of this PR, namely splitting the tests and results into separate files, is gonna be a no-go to begin with

[p-bakker]

I guess the change in where the test results are stored would break the mechanism in Babel that downloads and converts the compat data, but imho it would be easy to adapt Babel accordingly

[p-bakker]

@ljharb @chicoxyzzy any chance to get some feedback on the general idea to extract the results out from the data-*.js files and into .json files?

Is it worth my time to finish this PR and get it out of draft status so it can be considered for merge? Or is this general idea dead in the water?

[ljharb]

It's a pretty major change so it's hard to develop an opinion about it, I think :-)

[p-bakker]

Ok, but what does that mean exactly? 🙂

Does it mean you and/or others are actively thinking about whether or not this change is something you're willing to accept?

Or do you need something additional from me to be able to make that decision?

Or ....?

[ljharb]

It means I'm actively thinking about it. I can't speak for the other maintainers.