Tests: New .testhtml format to replace .js language tests

[RunDevelopment]

I added a new test file format that replaces the .js language tests (all 5 of them).

Motivation

.js language tests are inconvenient because commands like npm run test:languages -- --update don't affect them. They also require more escaping because code is stored in JS strings.

A new test suite came up here. Inspired by this, I came up with .testhtml. .testhtml files are very similar to our current .test files. The only difference is that they store the HTML output instead of the simplified JSON stream. This allows .testhtml files to test the same code as current .js files while being as convenient as .test files.

Changes

• Added .testhtml format (implementation + doc),
• Removed support for .js test files can converted all current .js test files into .testhtml files.
• Language tests can now load plugins. This means that languages can now test their compatibility/interaction with plugins.

---

/cc @JaKXz

[github-actions]

No JS Changes

Generated by 🚫 dangerJS against 61739ed

[joshgoebel]

What makes these special and requiring a different format than the traditional .test format?

Or: Why were the JS tests different in the first place?

[RunDevelopment]

Two reasons:

1. .test files only included limited information about the token stream. E.g. they don't include aliases. .js files include the complete HTML produced by Prism.
2. The token streams stored in .test files are based on Prism.tokenize + a few hooks. This means that they only test this section of the highlighting process. On the other hand, .js files test almost the entire highlighting process.

Basically, .test files test everything before a token stream gets converted into HTML while .js (and now .testhtml) files test (almost) everything. This "almost everything" is sometimes necessary to test a few languages that contain small plugins (e.g. Markup and Markdown).

[joshgoebel]

Thanks for the clarification. I was curious about this because I've never loved how we do our markup testing... though the idea of just merging the input and output files has long been an idea I've had (ie, what you're doing with testhtml). Currently we have them as entirely separate files in our repo.

[JaKXz]

I think this is a step in a good direction! This is exactly a problem I was thinking about and glad that we're on the same page about the confidence in the system -- however, with my limited context it seems like we're re-inventing a wheel for a solved problem...

I wrote a full response here: JaKXz#1 (comment)

[RunDevelopment]

Closed in favor of #3148.