Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add path mapping support to ESM and CJS loaders #1585

Open
wants to merge 79 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
79 commits
Select commit Hold shift + click to select a range
3be1fde
Add path mapping support to ESM loader
geigerzaehler Dec 29, 2021
c542d98
fixup! Add path mapping support to ESM loader
geigerzaehler Dec 29, 2021
0cbfa6c
fixup! fixup! Add path mapping support to ESM loader
geigerzaehler Dec 30, 2021
e7082b1
fixup! fixup! fixup! Add path mapping support to ESM loader
geigerzaehler Dec 30, 2021
69a397b
Merge remote-tracking branch 'origin/main' into path-mapping
cspotcode Jan 24, 2022
2fcb982
address code review comments
geigerzaehler Jan 24, 2022
48f5262
map esm paths in all included files
geigerzaehler Jan 24, 2022
a19454b
improve error message when mapped module is not found
geigerzaehler Jan 24, 2022
32e26e8
Review changes; add CommonJS path mapping
cspotcode Jan 25, 2022
362935b
fix failing tests
cspotcode Jan 25, 2022
e573fd7
add path mapping to docs
cspotcode Jan 25, 2022
0012b22
add flag to enable/disable path mapping in the two loaders
cspotcode Jan 25, 2022
b6352e3
fix windows tests?
cspotcode Jan 25, 2022
85643fd
fix windows tests?
cspotcode Jan 25, 2022
9c46688
Merge remote-tracking branch 'origin/main' into path-mapping
cspotcode Jan 31, 2022
bede1b6
add path mapping option to docs
cspotcode Jan 31, 2022
80746c2
changes
cspotcode Jan 31, 2022
885b7b1
replace equal (deprecated) with strictEqual
charles-allen Feb 25, 2022
c3dbe73
extract shared tsconfig
charles-allen Feb 25, 2022
8c5fd23
move import targets 2-deep (to support combined baseUrl + * path)
charles-allen Feb 25, 2022
e66d236
test: baseUrl + no paths
charles-allen Feb 25, 2022
23f40b1
test: baseUrl + * path
charles-allen Feb 25, 2022
54fdbba
test: fallback to node_modules
charles-allen Feb 25, 2022
78652b2
clean up destructuring
charles-allen Feb 25, 2022
b8e6fb7
fix setting project
charles-allen Feb 25, 2022
2973399
test: fallback to built-in
charles-allen Feb 25, 2022
5ffd905
avoid space in command & add comment
charles-allen Feb 25, 2022
d200301
fix setting project and PATH
charles-allen Feb 27, 2022
c621af0
test: skip type-defs
charles-allen Feb 27, 2022
ec038c1
test: external imports ignore paths
charles-allen Feb 27, 2022
d471d1d
tests: relative/base-relative imports ignore paths
charles-allen Feb 27, 2022
d5728dc
Merge remote-tracking branch 'origin/main' into path-mapping
cspotcode Mar 1, 2022
5b686e1
Updates
cspotcode Mar 1, 2022
cb3706e
add missing package.json from node_modules
cspotcode Mar 1, 2022
d00bf73
rolling back a config change; I was wrong
cspotcode Mar 1, 2022
398db86
add helper execEsm(...)
charles-allen Mar 12, 2022
a91d5d7
test: decouple base-url-no-paths test (by extracting it)
charles-allen Mar 12, 2022
7829b54
test: decouple skip-type-definition test (by extracting it)
charles-allen Mar 12, 2022
9fd944e
test: refactor to apply tests across multiple module types & project …
charles-allen Mar 14, 2022
4e0d363
temporarily move old tests out of the way
charles-allen Mar 14, 2022
09cc514
test: refactor again to restore shared examples/node_modules (while m…
charles-allen Mar 14, 2022
70d9bf5
import `assert` async so we can assert pre-conditions (& so sut impor…
charles-allen Mar 14, 2022
e50abd4
sync tsconfigs
charles-allen Mar 14, 2022
73d6acc
fix error message in `import-node-built-in` (it's not a precondition)
charles-allen Mar 14, 2022
d02cbed
try to clean up cjs imports :/
charles-allen Mar 14, 2022
283309d
simplify tests
charles-allen Mar 14, 2022
f2e2f65
test: restore "ignore type definition" tests
charles-allen Mar 14, 2022
93ac2ac
delete obsolete examples
charles-allen Mar 14, 2022
bf7bfcd
fix star paths
charles-allen Mar 14, 2022
c26b9dd
move old tests a bit more out of the way
charles-allen Mar 14, 2022
09ebf63
Merge remote-tracking branch 'origin/main' into path-mapping
cspotcode Mar 15, 2022
e0af738
ensure ts-node is installed before running tests
cspotcode Mar 15, 2022
31acd50
disable typechecking in tests
cspotcode Mar 15, 2022
36377fc
test: add a case that imports an esm lib from node_modules
charles-allen Mar 28, 2022
56ef378
use require everywhere in cjs examples
charles-allen Mar 28, 2022
4dcff42
destructure import proxyLodash; align both depends-on-lodash deps as cjs
charles-allen Mar 28, 2022
ba176bf
fix baseUrl
charles-allen Mar 28, 2022
38ad3f5
revert assertions to use expect(err).toBeNull(); Clean up config cons…
charles-allen Mar 28, 2022
1687de9
prefer function declaration to arrow function
charles-allen Mar 28, 2022
8e9b117
move base 2 deep, so star path can be used in addition to base
charles-allen Mar 28, 2022
806ce74
tests: non-relative imports
charles-allen Mar 28, 2022
0fd8a5c
fix import extensions
charles-allen Mar 28, 2022
02818e3
tests: imports from js, jsx, tsx
charles-allen Mar 28, 2022
6faedd9
rename under-base to below-base (nicer lexical sort)
charles-allen Mar 28, 2022
64bb679
tests: relative imports
charles-allen Mar 28, 2022
058d2e2
tests: import invalid path
charles-allen Mar 28, 2022
4a3b6b4
test: should not use star-path to resolve relative import
charles-allen Mar 28, 2022
dc7e950
tests: basic path mapping
charles-allen Mar 28, 2022
c5ea9b0
fix import style
charles-allen Mar 28, 2022
7bfe31a
tests: map using first available candidate
charles-allen Mar 28, 2022
c8c35ca
tests: more specific path; static path
charles-allen Mar 28, 2022
99ca516
comment out file-system-base-relative import
charles-allen Mar 28, 2022
4fefc57
tests: mapping from js, jsx, tsx files
charles-allen Mar 28, 2022
3a0067e
clean up (delete all old tests)
charles-allen Mar 28, 2022
78e3eda
tweaks before pulling in the latest main branch
cspotcode May 18, 2022
4f5bc35
Merge remote-tracking branch 'origin/main' into path-mapping
cspotcode May 18, 2022
ef926b9
fix
cspotcode May 18, 2022
7552fc4
style tweak
cspotcode May 19, 2022
6632481
turn on experimental resolver; add required file extensions to esm tests
cspotcode May 19, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
45 changes: 22 additions & 23 deletions dist-raw/node-errors.js
@@ -1,30 +1,29 @@
exports.codes = {
ERR_INPUT_TYPE_NOT_ALLOWED: createErrorCtor(joinArgs('ERR_INPUT_TYPE_NOT_ALLOWED')),
ERR_INVALID_ARG_VALUE: createErrorCtor(joinArgs('ERR_INVALID_ARG_VALUE')),
ERR_INVALID_MODULE_SPECIFIER: createErrorCtor(joinArgs('ERR_INVALID_MODULE_SPECIFIER')),
ERR_INVALID_PACKAGE_CONFIG: createErrorCtor(joinArgs('ERR_INVALID_PACKAGE_CONFIG')),
ERR_INVALID_PACKAGE_TARGET: createErrorCtor(joinArgs('ERR_INVALID_PACKAGE_TARGET')),
ERR_MANIFEST_DEPENDENCY_MISSING: createErrorCtor(joinArgs('ERR_MANIFEST_DEPENDENCY_MISSING')),
ERR_MODULE_NOT_FOUND: createErrorCtor((path, base, type = 'package') => {
return `Cannot find ${type} '${path}' imported from ${base}`
}),
ERR_PACKAGE_IMPORT_NOT_DEFINED: createErrorCtor(joinArgs('ERR_PACKAGE_IMPORT_NOT_DEFINED')),
ERR_PACKAGE_PATH_NOT_EXPORTED: createErrorCtor(joinArgs('ERR_PACKAGE_PATH_NOT_EXPORTED')),
ERR_UNSUPPORTED_DIR_IMPORT: createErrorCtor(joinArgs('ERR_UNSUPPORTED_DIR_IMPORT')),
ERR_UNSUPPORTED_ESM_URL_SCHEME: createErrorCtor(joinArgs('ERR_UNSUPPORTED_ESM_URL_SCHEME')),
ERR_UNKNOWN_FILE_EXTENSION: createErrorCtor(joinArgs('ERR_UNKNOWN_FILE_EXTENSION')),
}
exports.codes = {}

function joinArgs(name) {
return (...args) => {
return [name, ...args].join(' ')
function defineError(code, buildMessage) {
if (!buildMessage) {
buildMessage = (...args) => args.join(' ')
}
}

function createErrorCtor(errorMessageCreator) {
return class CustomError extends Error {
exports.codes[code] = class CustomError extends Error {
constructor(...args) {
super(errorMessageCreator(...args))
super(`${code}: ${buildMessage(...args)}`)
this.code = code
}
}
}

defineError("ERR_INPUT_TYPE_NOT_ALLOWED")
defineError("ERR_INVALID_ARG_VALUE")
defineError("ERR_INVALID_MODULE_SPECIFIER")
defineError("ERR_INVALID_PACKAGE_CONFIG")
defineError("ERR_INVALID_PACKAGE_TARGET")
defineError("ERR_MANIFEST_DEPENDENCY_MISSING")
defineError("ERR_MODULE_NOT_FOUND", (path, base, type = 'package') => {
return `Cannot find ${type} '${path}' imported from ${base}`
})
defineError("ERR_PACKAGE_IMPORT_NOT_DEFINED")
defineError("ERR_PACKAGE_PATH_NOT_EXPORTED")
defineError("ERR_UNSUPPORTED_DIR_IMPORT")
defineError("ERR_UNSUPPORTED_ESM_URL_SCHEME")
defineError("ERR_UNKNOWN_FILE_EXTENSION")
40 changes: 35 additions & 5 deletions src/esm.ts
Expand Up @@ -15,6 +15,7 @@ import {
import { extname } from 'path';
import * as assert from 'assert';
import { normalizeSlashes } from './util';
import { createPathMapper } from './path-mapping';
const {
createResolve,
} = require('../dist-raw/node-esm-resolve-implementation');
Expand Down Expand Up @@ -93,6 +94,7 @@ export function registerAndCreateEsmHooks(opts?: RegisterOptions) {

export function createEsmHooks(tsNodeService: Service) {
tsNodeService.enableExperimentalEsmLoaderInterop();
const mapPath = createPathMapper(tsNodeService.config.options);
cspotcode marked this conversation as resolved.
Show resolved Hide resolved

// Custom implementation that considers additional file extensions and automatically adds file extensions
const nodeResolveImplementation = createResolve({
Expand Down Expand Up @@ -147,12 +149,40 @@ export function createEsmHooks(tsNodeService: Service) {
return defer();
}

// pathname is the path to be resolved
let candidateSpecifiers: string[] = [specifier];

if (context.parentURL) {
const parentUrl = new URL(context.parentURL);
if (parentUrl.pathname && extname(parentUrl.pathname) === '.ts') {
cspotcode marked this conversation as resolved.
Show resolved Hide resolved
const mappedSpecifiers = mapPath(specifier);
if (mappedSpecifiers) {
candidateSpecifiers = mappedSpecifiers.map((path) =>
pathToFileURL(path).toString()
);
}
}
}

let candidateSpecifier: string | undefined;
while ((candidateSpecifier = candidateSpecifiers.shift())) {
cspotcode marked this conversation as resolved.
Show resolved Hide resolved
try {
return await nodeResolveImplementation.defaultResolve(
candidateSpecifier,
context,
defaultResolve
);
} catch (err) {
const isNotFoundError = (<any>err).code === 'ERR_MODULE_NOT_FOUND';
cspotcode marked this conversation as resolved.
Show resolved Hide resolved
if (isNotFoundError && candidateSpecifiers.length > 0) {
continue;
} else {
throw err;
cspotcode marked this conversation as resolved.
Show resolved Hide resolved
}
}
}

return nodeResolveImplementation.defaultResolve(
specifier,
context,
defaultResolve
throw new Error(
`Empty mapped paths for ${specifier} imported from ${context.parentURL}`
geigerzaehler marked this conversation as resolved.
Show resolved Hide resolved
);
}

Expand Down
131 changes: 131 additions & 0 deletions src/path-mapping.ts
@@ -0,0 +1,131 @@
import type * as ts from 'typescript';
import { join as joinPath } from 'path';

// Path mapper returns a list of mapped specifiers or `null` if the
// given `specifier` was not mapped.
type PathMapper = (specifier: string) => string[] | null;

export function createPathMapper(
compilerOptions: ts.CompilerOptions
): PathMapper {
if (compilerOptions.paths) {
if (!compilerOptions.baseUrl) {
throw new Error(`Compiler option 'baseUrl' required when 'paths' is set`);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is compilerOptions.baseUrl guaranteed to be an absolute path, or do we need to resolve it relative to the tsconfig's location?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I assumed that we always get the configuration from ts.readConfigFile() in src/configuration.ts and the function resolves the base URL relative to the config file. But there are probably other ways to set the base URL like through the command line, API, or environment.

To address this I think it make sense to resolve any base URL that is not a URL but a relative path to the current directory.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, we almost always get the configuration from ts.readConfigFile(), so we should be good. I think the only way to set it otherwise is via the API? Since we don't support all of tsc's command-line flags.

Maybe the correct approach is to resolve baseUrl once within create() so that we have a reliable, absolute baseUrl to be used elsewhere.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe the correct approach is to resolve baseUrl once within create() so that we have a reliable, absolute baseUrl to be used elsewhere.

Alternatively, we could require baseUrl to be absolute and throw an error otherwise. This would allow us to avoid implicit, environment-dependent behavior for users of the API. This might prevent some confusion when path mapping doesn’t work as expected but only fails at the resolution stage instead of the construction stage. (For example a user may think the base URL they set on the API is considered relative to tsconfig.json and their scripts work whenever they are run from the project root. But then they run the script from a subdirectory and it fails.)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, let's do that. In create() we throw an error if baseUrl is not absolute.

Copy link
Collaborator

@cspotcode cspotcode Jan 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, scratch that, seems we always pass options through TypeScript's API to be normalized, even when provided via our API:

$ node
> require('ts-node').create({compilerOptions: {baseUrl: "foobar"}}).config.options.baseUrl
'/home/ubuntu/dev/ts-node/ts-node/path-mapping/foobar'

So I guess we're all good.

}

const mappings = Object.entries(compilerOptions.paths).map(
([patternString, outputs]) => ({
pattern: parsePattern(patternString),
outputs,
})
);
const mappingConfig = { mappings, baseUrl: compilerOptions.baseUrl };

return function map(specifier: string): string[] | null {
return mapPath(mappingConfig, specifier);
};
} else {
return () => null;
}
}

interface MappingConfig {
mappings: Mapping[];
baseUrl: string;
}

interface Mapping {
pattern: Pattern;
outputs: string[];
}

type Pattern =
| {
type: 'wildcard';
prefix: string;
suffix: string;
}
| { type: 'static'; value: string };

function mapPath(mappingConfig: MappingConfig, path: string): string[] | null {
let bestMatchWeight = -Infinity;
let bestMatch: [Mapping, string] | null = null;

for (const mapping of mappingConfig.mappings) {
if (patternWeight(mapping.pattern) > bestMatchWeight) {
const match = matchPattern(mapping.pattern, path);
if (match !== null) {
bestMatch = [mapping, match];
bestMatchWeight = patternWeight(mapping.pattern);
}
}
}

if (bestMatch) {
const [mapping, match] = bestMatch;
return mapping.outputs.map((output) =>
joinPath(mappingConfig.baseUrl, output.replace('*', match))
);
} else {
return null;
}
}

// Return the submatch when the pattern matches.
//
// For the wildcard pattern string `a*z` and candidate `afooz` this
// returns `foo`. For the static pattern `bar` and the candidate `bar`
// this returns `bar`.
function matchPattern(pattern: Pattern, candidate: string): string | null {
switch (pattern.type) {
case 'wildcard':
if (
candidate.length >= pattern.prefix.length + pattern.suffix.length &&
candidate.startsWith(pattern.prefix) &&
candidate.endsWith(pattern.suffix)
) {
return candidate.substring(
pattern.prefix.length,
candidate.length - pattern.suffix.length
);
} else {
return null;
}
case 'static':
if (pattern.value === candidate) {
return candidate;
} else {
return null;
}
}
}

// Pattern weight to sort best matches.
//
// Static patterns have the highest weight. For wildcard patterns the
// weight is determined by the length of the prefix before the glob
// `*`.
function patternWeight(pattern: Pattern): number {
if (pattern.type === 'wildcard') {
return pattern.prefix.length;
} else {
return Infinity;
}
}

function parsePattern(patternString: string): Pattern {
const indexOfStar = patternString.indexOf('*');
if (indexOfStar === -1) {
return { type: 'static', value: patternString };
}

if (patternString.indexOf('*', indexOfStar + 1) !== -1) {
throw new Error(`Path pattern ${patternString} contains two wildcards '*'`);
}

return {
type: 'wildcard',
prefix: patternString.substring(0, indexOfStar),
suffix: patternString.substring(indexOfStar + 1),
};
}
10 changes: 10 additions & 0 deletions src/test/index.spec.ts
Expand Up @@ -1224,6 +1224,16 @@ test.suite('ts-node', (test) => {
if (semver.gte(process.version, '14.13.1'))
await runModuleTypeTest('override-to-esm', 'mjs');
});

test('path mapping', async () => {
const { err } = await exec(
`${CMD_ESM_LOADER_WITHOUT_PROJECT} index.ts`,
{
cwd: join(TEST_DIR, './esm-path-mapping'),
}
);
expect(err).toBe(null);
});
}

if (semver.gte(process.version, '12.0.0')) {
Expand Down
32 changes: 32 additions & 0 deletions tests/esm-path-mapping/index.ts
@@ -0,0 +1,32 @@
import * as assert from 'assert';
cspotcode marked this conversation as resolved.
Show resolved Hide resolved

// Path is mapped
import map1foo from 'map1/foo.js';

// Path is mapped using `.jsx` extension
import map1jsx from 'map1/jsx.js';

// Path is mapped using the first candidate `mapped/2-foo` and not `mapped/2a-foo`
import map2foo from 'map2/foo.js';

// Path is mapped using the second candidate because the first `mapped/2-bar.ts`
// does not exist
import map2bar from 'map2/bar.js';

// Path is mapped using `.js` extension
import map2js from 'map2/js.js';

// Path is mapped using the more specific pattern instead of
// `mapped/2-specific/foo
import map2specific from 'map2/specific/foo.js';

// Path is mapped when using no wildcard
import mapStatic from 'static';

assert.equal(map1foo, 'mapped/1-foo');
assert.equal(map1jsx, 'mapped/1-jsx');
assert.equal(map2foo, 'mapped/2-foo');
assert.equal(map2bar, 'mapped/2a-bar');
assert.equal(map2js, 'mapped/2a-js');
assert.equal(map2specific, 'mapped/2-specific-foo');
assert.equal(mapStatic, 'mapped/static');
1 change: 1 addition & 0 deletions tests/esm-path-mapping/mapped/1-foo.ts
@@ -0,0 +1 @@
export default 'mapped/1-foo';
6 changes: 6 additions & 0 deletions tests/esm-path-mapping/mapped/1-jsx.jsx
@@ -0,0 +1,6 @@
export default 'mapped/1-jsx';

const React = {
createElement() {},
};
const div = <div></div>;
1 change: 1 addition & 0 deletions tests/esm-path-mapping/mapped/2-foo.ts
@@ -0,0 +1 @@
export default 'mapped/2-foo';
1 change: 1 addition & 0 deletions tests/esm-path-mapping/mapped/2-specific-foo.ts
@@ -0,0 +1 @@
export default 'mapped/2-specific-foo';
1 change: 1 addition & 0 deletions tests/esm-path-mapping/mapped/2a-bar.ts
@@ -0,0 +1 @@
export default 'mapped/2a-bar';
1 change: 1 addition & 0 deletions tests/esm-path-mapping/mapped/2a-foo.ts
@@ -0,0 +1 @@
export default 'mapped/2a/foo';
1 change: 1 addition & 0 deletions tests/esm-path-mapping/mapped/2a-js.js
@@ -0,0 +1 @@
export default 'mapped/2a-js';
1 change: 1 addition & 0 deletions tests/esm-path-mapping/mapped/static.ts
@@ -0,0 +1 @@
export default 'mapped/static';
3 changes: 3 additions & 0 deletions tests/esm-path-mapping/package.json
@@ -0,0 +1,3 @@
{
"type": "module"
}
15 changes: 15 additions & 0 deletions tests/esm-path-mapping/tsconfig.json
@@ -0,0 +1,15 @@
{
"compilerOptions": {
"module": "ESNext",
"allowJs": true,
"jsx": "react",
"baseUrl": "./mapped",
"paths": {
"map1/*": ["./1-*"],
"map2/*": ["./2-*", "./2a-*"],
"map2/specific/*": ["./2-specific-*"],
"static": ["./static.js"]
},
"moduleResolution": "node"
}
}