Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Externalize results from tests #1879

Open
p-bakker opened this issue Oct 6, 2023 · 0 comments · May be fixed by #1881
Open

Externalize results from tests #1879

p-bakker opened this issue Oct 6, 2023 · 0 comments · May be fixed by #1881

Comments

@p-bakker
Copy link

p-bakker commented Oct 6, 2023

ATM each test and its results on all the different environments are co-located together in the data-*-js modules

This poses 2 problems:

  • prevents automated updates of the results when tests start passing/failing
  • prevents generating compat tables for a completely different set of environments

The later of these 2 points I've already have a (draft) PR for (#1791), but the flaw in that PR to allow passing in custom results is that the approach breaks down in linking the custom results to a test, if the name of a test would ever change. I'm not sure how often this happens, but with the current setup that would not be an issue, but once we'd allow providing external results, it would become problematic as it would cause the externally recorded/stored results to get disconnected from the tests in data-*.js.

As for the first point: as the results are stored in an object inside a module, unless you somehow modify the sourcecode of the module through code, there's no way afaics to automate the updating of the results, making it manual labour

So, I would propose the following:

  • rename the data-.js files to test-.js
  • externalize the results from these test-.js files to plain json files, using the results-.json naming convention
  • use the name of (sub)test to link the test (from test-.js) to the results in results-.js
  • optional: allow overriding the displayed testname in the generated HTMl by providing the optional displayName in addition to the name on a (sub)test in test-*.js

This achieves the following goals:

  • a testrunner could now be made to automatically modify the results-*.json file, as to allow to make updating the Compat Table less manual
  • forces the name of a (sub)test to not get changed without thinking about it
  • prepares the codebase for the next step, which is to allow providing alternative environments and results from the command line, which in turn allows creating custom compat tables

Eventually this repo could then even be spilt in a package to create Compat Tables in general and a repo for the kangas.github.io gh pages (as suggested by @ljharb in #1791)

Thoughts?

@p-bakker p-bakker linked a pull request Oct 9, 2023 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant