Simplified ESM build #3704

trusktr · 2023-07-05T23:48:41Z

Issue:

[v2] type checking and intellisense support #3702 (read this first)

Continues from PR

Add prepare script, making it easy to install Prism from outside of NPM. #3699

To test this in an existing app, install Prism v2 like so:

npm install prismjs@github:trusktr/prism#simple-esm

To try an existing Prism v2 example along with a Node ESM test try this repo:

https://github.com/trusktr/prism-v2-test

npm install # this installs Prism from git, from this branch
npm test # this verifies that importing Prism in Node ESM works (we should perhaps move that test into this repo)

The gold standard today is that if a project is both as close to vanilla ESM as possible and importable in Node ESM, it is likely to work with today's build tools.

In particular, this change tries to stay as close to vanilla ESM as possible, by avoiding the use of package.json exports field (browsers do not read package.json, let along an exports field).

The example app above shows importing Prism without any configuration or build. In the worst case, if they wish to avoid putting ./node_modules directly in their import statement, they can create a <script type=importmap> element to map prismjs to the node_modules location.

All Mocha unit tests pass except two of them. I also removed the components.json build, which I think needs to be restored for when new languages are added (? I'm not sure yet), but now we'll ensure that we restore what is needed on this simplified foundation.

The mantras here are:

simple ES Module output that requires the least amount of tooling or configuration to import in vanilla ES Modules (namely, in a browser where this is most likely to be imported, where browser ESM is a subset of Node ESM).
we start with most vanilla ESM setup before starting to add any build steps, so that we're staying on the most compatible ESM path.

This is as simple as it can possibly get for browser ES Modules (apart from us also including a sample importmap that they can paste into their HTML, which we can do.

This is still WIP, we need to look at the components.json build requirement.

ensure the components.json build is handled if needed
fix last type definition issues (described in [v2] type checking and intellisense support #3702 (comment))

Comments in the diff have some more details.

The `prepare` script is the NPM-standard way in which outside-of-NPM packages can be build, for example with installed from a git location. After merging this into the main branch (currently `master`), people will be able to do this: ```sh npm install prismjs@github:PrismJS/prism ``` If they want to try it right now before v2 is merged (if this is merged into v2), they can do this: ```sh npm install prismjs@github:PrismJS/prism#v2 ``` Without this, such installations are not possible. What NPM does during the `install` process is - it clones the specified repo, - checks out the specified git ref (f.e. `v2`), - installs dependencies including dev dependencies, - runs the `prepare` script if any (the `prepare` script will *always* have dev dependencies available to it), - and finally packages the result in the same way as `npm pack` such that build outputs are included in the final package that gets installed into the user's node_modules This would be a way for people to easily test v2 right now and provide feedback.

… structure, with type definitions and source maps, in a way that is as compatible with vanilla ES Modules as possible

github-actions · 2023-07-05T23:49:17Z

No JS Changes

Generated by 🚫 dangerJS against 3486a8e

trusktr · 2023-07-06T00:09:50Z

package.json

@@ -2,14 +2,16 @@
 	"name": "prismjs",
 	"version": "1.29.0",
 	"description": "Lightweight, robust, elegant syntax highlighting. A spin-off project from Dabblet.",
-	"main": "prism.js",
+  "type": "module",
+	"main": "dist/core/prism.js",


Opt into the modern Node ESM standard, and point to the main file to get Prism from (there was no more ./prism.js).

trusktr · 2023-07-06T00:10:18Z

package.json

 	"style": "themes/prism.css",
 	"engines": {
 		"node": ">=14"
 	},
 	"scripts": {
 		"benchmark": "ts-node benchmark/benchmark.ts",
-		"build": "ts-node scripts/build.ts",
+		"build": "ts-node-esm scripts/build.ts && npm run tsc",


Running with ts-node-esm runs files as Node ESM, rather than CommonJS.

trusktr · 2023-07-06T00:10:41Z

package.json

 	"style": "themes/prism.css",
 	"engines": {
 		"node": ">=14"
 	},
 	"scripts": {
 		"benchmark": "ts-node benchmark/benchmark.ts",
-		"build": "ts-node scripts/build.ts",
+		"build": "ts-node-esm scripts/build.ts && npm run tsc",
+		"prepare": "npm run build",


This is from #3699

trusktr · 2023-07-06T00:11:46Z

package.json

@@ -23,7 +25,8 @@
 		"test:plugins": "ts-mocha tests/plugins/**/*.ts",
 		"test:runner": "ts-mocha tests/testrunner-tests.ts",
 		"test": "npm-run-all test:*",
-		"tsc": "tsc && tsc -p tests/tsconfig.json"
+    "typecheck": "tsc --noEmit && tsc -p tests/tsconfig.json --noEmit",
+		"tsc": "tsc"


We use a plain and simple tsc step to build dist/ output. The new typecheck script replaces the old tsc script for type checking without emitting output.

We should probably cover the other TS files as well, currently this covers only src/ and tests/.

trusktr · 2023-07-06T00:14:08Z

.eslintrc.cjs

@@ -15,7 +15,7 @@ module.exports = {
 		'eqeqeq': ['error', 'always', { 'null': 'ignore' }],

 		// imports
-		'import/extensions': ['warn', 'never'],
+		'import/extensions': ['warn', 'always'], // "always" is most compatible with default vanilla ES Modules


Having .js extensions, even if not to taste, is needed for the simplest ESM setup, and TypeScript officially recommends writing .js directly in TypeScript import statements for this purpose. So, we force it here so we don't mess up.

If we remove the .js extensions, the vanilla ESM example over at https://github.com/trusktr/prism-v2-test will break.

trusktr · 2023-07-06T00:15:50Z

.eslintrc.cjs

@@ -15,7 +15,7 @@ module.exports = {
 		'eqeqeq': ['error', 'always', { 'null': 'ignore' }],


This file is the only remnant of CommonJS in the repo (apart from the dependencies which I haven't updated yet).

In the upcoming ESLint v9, the new "flat config" format will also be in Node ESM format, and we'll have to move ESLint config to ESM at that point.

trusktr · 2023-07-06T00:16:19Z

.mocharc.json

@@ -0,0 +1,7 @@
+{
+	"Xrequire": "@babel/register",
+		"Xexperimental-specifier-resolution=node": "",


trusktr · 2023-07-06T00:16:44Z

.mocharc.json

+	"Xrequire": "@babel/register",
+		"Xexperimental-specifier-resolution=node": "",
+	"node-option": [
+		"loader=ts-node/esm"


This makes ts-mocha run our files as Node ESM files.

trusktr · 2023-07-06T00:17:32Z

scripts/build.ts

-import rollupTerser from '@rollup/plugin-terser';
-import rollupTypescript from '@rollup/plugin-typescript';
+// import rollupTerser from '@rollup/plugin-terser';
+// import rollupTypescript from '@rollup/plugin-typescript';


cleanup: delete unused code

Note how much less build code we need to maintain with this setup. But see the comment about components.json.

trusktr · 2023-07-06T00:18:56Z

scripts/build.ts

+// 	all_languages_placeholder: () => Promise.resolve(languageIds),
+// 	title_placeholder: async () => {
+// 		const rawTitles = new Map<string, string>();
+// 		for (const [id, entry] of Object.entries(components.languages)) {


todo: figure if/how to handle components.json. For now I simply disabled buildJS.

trusktr · 2023-07-06T00:19:51Z

scripts/build.ts

@@ -42,7 +44,7 @@ async function minifyCSS() {
 	for (const id of pluginIds) {
 		const file = path.join(SRC_DIR, `plugins/${id}/prism-${id}.css`);
 		if (fs.existsSync(file)) {
-			input[`plugins/prism-${id}.css`] = file;
+			input[`plugins/${id}/prism-${id}.css`] = file;


Here we output the .css files so they are in the same location in dist/ as they are in src/.

Less confusion: For anything in src/**/*, just replace src/ with dist/ to arrive at the output path. Easier mental model.

trusktr · 2023-07-06T00:22:40Z

tests/components-test.ts


+const __dirname = path.dirname(fileURLToPath(import.meta.url));


__dirname is not available in Node ESM files. Instead we have import.meta.url from which to base path handling.

trusktr · 2023-07-06T00:31:08Z

tsconfig.json

@@ -25,35 +25,31 @@
    // "moduleDetection": "auto",                        /* Control what method is used to detect module-format JS files. */

    /* Modules */
-    "module": "commonjs",                                /* Specify what module code is generated. */
+    "module": "ESNext",                                  /* Specify what module code is generated. */


Switch to ESM output format. All JS engines support top-level await (esnext).

trusktr · 2023-07-06T00:36:18Z

tsconfig.json

    // "rootDir": "./",                                  /* Specify the root folder within your source files. */
-    // "moduleResolution": "node",                       /* Specify how TypeScript looks up a file from a given module specifier. */
+    "moduleResolution": "node",                          /* Specify how TypeScript looks up a file from a given module specifier. */


This is not part of vanilla ESM, but it is very common to have this set to node even for vanilla web projects, using import maps as needed, because this is the default and simplest way in which type definitions flow into code in the ESM format (TS automatically grabs types from node_modules, which is the most common way people currently install libraries).

This covers non-relative bare module specifiers in our tests (f.e. import ... from 'some-lib'), while code inside of src/ does not actually have any such library imports. If code in src/ did have library imports, I would have needed to add an importmap to my example at https://github.com/trusktr/prism-v2-test.

trusktr · 2023-07-06T00:37:22Z

tsconfig.json

    // "baseUrl": "./",                                  /* Specify the base directory to resolve non-relative module names. */
    // "paths": {},                                      /* Specify a set of entries that re-map imports to additional lookup locations. */
    // "rootDirs": [],                                   /* Allow multiple folders to be treated as one when resolving modules. */
    // "typeRoots": [],                                  /* Specify multiple folders that act like './node_modules/@types'. */
    // "types": [],                                      /* Specify type package names to be included without being referenced in a source file. */
-    // "allowUmdGlobalAccess": true,                     /* Allow accessing UMD globals from modules. */
+    "allowUmdGlobalAccess": false,                        /* Allow accessing UMD globals from modules. */


Globals are typically considered bad. Disable type defs for globals by default, so that they don't sneak into our scope unintentionally from node_modules. Add them explicitly when/if needed.

trusktr · 2023-07-06T00:37:46Z

tsconfig.json

    // "moduleSuffixes": [],                             /* List of file name suffixes to search when resolving a module. */
    // "resolveJsonModule": true,                        /* Enable importing .json files. */
    // "noResolve": true,                                /* Disallow 'import's, 'require's or '<reference>'s from expanding the number of files TypeScript should add to a project. */

-    /* JavaScript Support */
-    "allowJs": true,                                  /* Allow JavaScript files to be a part of your program. Use the 'checkJS' option to get errors from these files. */
-    "checkJs": true,                                  /* Enable error reporting in type-checked JavaScript files. */


There's no JS, not needed.

trusktr · 2023-07-06T00:40:16Z

tsconfig.json

    // "emitDeclarationOnly": true,                      /* Only output d.ts files and not JavaScript files. */
-    // "sourceMap": true,                                /* Create source map files for emitted JavaScript files. */
+    "sourceMap": true,                                   /* Create source map files for emitted JavaScript files. */


declaration plus declarationMap enables the feature where Go-To-Definition leads back to source code instead of declaration files. Really nice!

sourceMap enables source mapping for anyone who will further process the files. For example if they ever import from 'prismjs' and then compile their app with Webpack or Rollup, it is nice to be able to see errors/etc relative to the original source positions. If someone ever reports a bug, this will be handy because they can show the original location and source in the bug report, etc.

trusktr · 2023-07-06T00:42:23Z

tsconfig.json

    // "importHelpers": true,                            /* Allow importing helper functions from tslib once per project, instead of including them per-file. */
-    // "importsNotUsedAsValues": "remove",               /* Specify emit/checking behavior for imports that are only used for types. */
+    // "importsNotUsedAsValues": "error",                /* Specify emit/checking behavior for imports that are only used for types. */
+	"verbatimModuleSyntax": true,


importsNotUsedAsValues is deprecated. verbatimModuleSyntax is the new option. The rule is simple:

Any import with a type label gets removed. The rest is left behind. This is the least confusing compared to before: whatever imports you write are what you get (minus type imports). No room for error. It is now our responsibility to write or not write the imports we want, without TypeScript trying to guess.

Someone can use a tool like Webpack or Rollup to apply module tree shaking at their own risk of dealing with breakage, if they want to.

trusktr · 2023-07-06T00:43:10Z

tsconfig.json

@@ -70,7 +66,7 @@

    /* Interop Constraints */
    // "isolatedModules": true,                          /* Ensure that each file can be safely transpiled without relying on other imports. */
-    // "allowSyntheticDefaultImports": true,             /* Allow 'import x from y' when a module doesn't have a default export. */
+    "allowSyntheticDefaultImports": true,                /* Allow 'import x from y' when a module doesn't have a default export. */


Required for compatibility with Node ESM. In Node ESM, an imported CommonJS module's module object is always the default export inside of a Node ESM file, and that is what this enforces.

Prism code (in src/) does not import any libraries, let alone any CommonJS modules. However, it is possible that our infrastructure code (build, tests, benchmark) may import a library in CommonJS format, and because those files now run as Node ESM, this is required just in case those files do import a native Node CommonJS library.

trusktr · 2023-07-06T00:47:36Z

Alrighty, I commented in all the relevant files to explain the changes. The rest of the vast majority of uncommented files are all just format updates (adding .js extensions, tweaking one or two imports for Node ESM compat, fixing __dirname).

trusktr and others added 2 commits June 29, 2023 18:32

simplify the TS build: output a simpler structure based on the source…

7052543

… structure, with type definitions and source maps, in a way that is as compatible with vanilla ES Modules as possible

trusktr mentioned this pull request Jul 6, 2023

[v2] type checking and intellisense support #3702

Open

trusktr commented Jul 6, 2023

View reviewed changes

.mocharc.json

@@ -0,0 +1,7 @@

{

"Xrequire": "@babel/register",

"Xexperimental-specifier-resolution=node": "",

Copy link

Author

trusktr Jul 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cleanup

trusktr commented Jul 6, 2023

View reviewed changes

update tests tsconfig.json

3486a8e

trusktr commented Jul 6, 2023

View reviewed changes

This was referenced Jul 6, 2023

Add prepare script, making it easy to install Prism from outside of NPM. #3699

Open

Simplify and modernize Docsify docsifyjs/docsify#2104

Closed

trusktr marked this pull request as ready for review July 29, 2023 16:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Simplified ESM build #3704

Simplified ESM build #3704

trusktr commented Jul 5, 2023 •

edited

github-actions bot commented Jul 5, 2023 •

edited

trusktr Jul 6, 2023

trusktr Jul 6, 2023

trusktr Jul 6, 2023

trusktr Jul 6, 2023

trusktr Jul 6, 2023

trusktr Jul 6, 2023 •

edited

trusktr Jul 6, 2023

trusktr Jul 6, 2023

trusktr Jul 6, 2023 •

edited

trusktr Jul 6, 2023

trusktr Jul 6, 2023

trusktr Jul 6, 2023

trusktr Jul 6, 2023

trusktr Jul 6, 2023

trusktr Jul 6, 2023

trusktr Jul 6, 2023

trusktr Jul 6, 2023

trusktr Jul 6, 2023

trusktr Jul 6, 2023 •

edited

trusktr commented Jul 6, 2023

		@@ -15,7 +15,7 @@ module.exports = {
		'eqeqeq': ['error', 'always', { 'null': 'ignore' }],


		const __dirname = path.dirname(fileURLToPath(import.meta.url));

Simplified ESM build #3704

Are you sure you want to change the base?

Simplified ESM build #3704

Conversation

trusktr commented Jul 5, 2023 • edited

github-actions bot commented Jul 5, 2023 • edited

No JS Changes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trusktr Jul 6, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trusktr Jul 6, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trusktr Jul 6, 2023 • edited

Choose a reason for hiding this comment

trusktr commented Jul 6, 2023

trusktr commented Jul 5, 2023 •

edited

github-actions bot commented Jul 5, 2023 •

edited

trusktr Jul 6, 2023 •

edited

trusktr Jul 6, 2023 •

edited

trusktr Jul 6, 2023 •

edited