-
Notifications
You must be signed in to change notification settings - Fork 444
Test262 Technical Rationale Report, October 2017
Since mid 2014, Test262 has received consistent management and maintenance work, resulting in significant improvements towards the larger goal of improved relevance through increased coverage and broader adoption.
Test262 is a project that belongs to Ecma and its maintenance is governed by TC39 member company representatives. It was revived back in late 2014, early 2015, to be a repository of conformance tests for ECMA-262, ECMA-402 (Intl), and ECMA-404 (JSON) specification, to be used by a broader spectrum of implementers—breaking out of its former browser-only limitation. Every implementation of JavaScript, be it engine, parser, or runtime—with no preference given by Ecma or the project maintainers—can use the tests to verify conformance and interoperability across different platforms. Just like the ECMA-262 specification, Test262 is implementation agnostic, and therefore must provide all test materials in an unbiased format.
In the course of this effort, many errors have been found and fixed, or eliminated, whenever possible. Test262 proved itself reliable in exposing web reality issues and sometimes specification issues. A shortlist of examples include the removal of the Proxy's enumerable trap, and consistency across TypedArrays, ArrayBuffer and DataView constructors. Even smaller details have been found and fixed as a result of work on Test262.
One of the most pressing issues the project faces is organization and structure. Ensuring that Test262 can be consumed by varying clients is paramount to its relevance—and has been the source of significant feedback which has lead to many issues filed and subsequent improvements made. Examples:
- https://github.com/tc39/test262/issues/1202
- https://github.com/tc39/test262/issues/1175
- https://github.com/tc39/test262/issues/1062
- https://github.com/tc39/test262/issues/957
Test262 is a test suite used by many different platforms, this is a unique phenomenon, but also a challenge. To maintain the project, and its interoperability, Test262 needs to offer a useful file structure, as well as relevant per-test metadata. This creates two primary challenges: human and machine readability. To address the latter, each test file includes a block of YAML formatted metadata (called “frontmatter”), which provides information that can be used by a test runner to understand how to interpret, execute, and validate the test. Frontmatter documentation is available in the project within Interpreting and Contributing. Even these files, like all documentation, require maintenance. Test262 recognizes that out of date documentation is a problem and the project is working to fix that. As a step towards improving the documentation, the following will hopefully serve to provide technical clarity.
The project content is already protected by a root license file, a TC39 org CLA, and the Ecma membership agreement. Maintainers are working to eliminate this convention, which will mean new test files will not require the copyright header. This required legal assistance and the work is already in progress to fix it.
When work on Test262 restarted in late 2014, respective ids for ES5.1 (es5id) and ES6 (es6id) were added as a reference to the relevant specification sections. The es5id values were actually a mix of ES3 and ES5.1, derived from the old file naming convention. The es6id tag was created to ensure that newly added test files and changed test files could be differentiated from previously existing material. It was quickly made clear that this would not scale, as section numbers themselves are subject to change between every new revision and each test would have to be updated to include a new “es*id” for each new specification—including all extant test files affected by the section change. Regardless, a minimum reference was still necessary to uniformly communicate which specification sections are being tested in the given file. The solution was to create esid by eliminating the version number from the metadata tag name, and use the fragment id of the newly HTML-ified specification as the value. This also has the benefit of allowing Stage 3 feature tests to be added to Test262 with an esid, as their proposal documents will have matured enough to provide a reliable fragment id for new content being added to the main specification. The burden of updating a newly created esid through the staging process is minimal.
Descriptions were initially added to the frontmatter to provide human readable (and machine usable) context to each test file. Over time, some descriptions have been found to be too verbose, while failing to provide the general frame of reference of the test goals for each file—causing confusion instead of clarity. Since identifying this issue, the project has been making an effort to keep descriptions short, mostly in a single line usually with less than a total of 100 characters.
While the test file and directory structure promotes easy navigability for humans, it does not provide enough information for machines running the tests. Test files inevitably include one or more feature to be tested, either of its own behavior or its behavior when interacting with other parts of the language. Test262 has been steadily working to improve categorization of test files with the inclusion of a “features” tag for each test file. This provides control to consumers of Test262 that wish to include, or exclude, specific features from test runs. This is necessary for implementers of Stage 2 and 3 features, subset engines, and greenfield runtimes.
Tests which contain code that is written to intentionally exercise grammar productions and static semantics that are explicitly specified to throw SyntaxError exceptions are expected Test262 has worked to define, and refine, this information to clearly indicate those negative outcome expectations, as well as providing information indicating when they occur in an “early” or “runtime” phase. This is also being improved to address further details. https://github.com/tc39/test262/issues/1254
Test262 tests also allow their author to provide an optional “info” tag, which may be used to communicate long-form details about relevant specification steps being exercised in the given test file. It is primarily expected to contain direct quotes from the specification text, as the information is relevant for the test review process, and for additional context anytime the tests need revisiting, which may occur as a result of specification changes. This information is also valuable for understanding test coverage of a given feature.
The flags tag is fundamental to running tests correctly, as it communicates to a test runner when a test should run in only strict mode or only non-strict mode, if it's an async test and the runner should wait for completion, if it should be interpreted in a raw mode, etc.
Some optional metadata tags exists that are entirely irrelevant to the test itself, but are allowed, e.g. “authors”, which contains information that could alternatively be derived from the git history, if such information were ever to become necessary.
Since early 2015, Test262 has been working to eliminate gibberish test file names, e.g. those that reference a specification section number. File names corresponding to section numbers have the same problem "version" reference ids have -- specifically, the names will become meaningless as soon as a new edition of the specification is are edited or published.
Human readable names with long standing meaning have been used in recently added tests. There isn't a strict convention for the new test file naming because a strict convention would be impossible complex and is unnecessary. The purpose of the human readable names is to make the tests easier to browse, understand and contribute to. Any name which aims to meet these goals is sufficient, keeping naming patterns consistency within any given directory is preferable for the same reasons.
Test262 just wants file names that clear to a human and valid to a machine.
Test262 is a large project which now covers the ES specifications really well. Despite that, there are old legacy tests that are simply not updated because they are technically correct. A long term effort to update these tests is in progress.
The project is large because it attempts to verify the conformance of every observable detail in the specifications that it represents, and this demands granularity of test files to promote an easy separation for runners and reporting. Raising abstraction might become harmful to interoperable consumption, and may inadvertently hide details about the test or relevant test context. The trade off for maintenance burden is the use of test generation tools. For that, the project holds files for test generation with consistent checks to avoid conflicts of manual updates on the generated files.
The efforts put into Test262 over the last 3 years has allowed the project to promote CI tooling for automation, including metadata linting and general verifications to assert the consistency of the project.
Test262’s task of providing a detailed specification conformance test suite is supported by efforts to improve and evolve on two equally important fronts: maintenance and consumption. Both of these aspects require thoughtful and well considered efforts to successfully serve their broader purpose: to allow for the testing and verification of all JavaScript engine, parser, and runtime implementations—which includes, but is not limited to, those used in browsers, servers, and embedded environments.
Although structural conventions and maintenance policies may look complex, the result is consistent testing material which may be used across myriad JavaScript implementation targets. The minor additional efforts required of contributions does not exceed that which is considered reasonable for most large open source projects.
It's also safe to say the communication on where and how Test262 is used is not clear to everyone. This project does not hold much data for reporting results and consumption use. Providing this data could help everyone with a better perspective.
Test262 is committed to making the following improvements:
- Improve documentation for contributing to the project that improves access for everyone.
- Provide a technical rationale document, similar to this report, in the main repository.
- Review and refine the test metadata to better serve consumer clients and tooling.