Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

working mock test #39240

Closed
wants to merge 19 commits into from
Closed
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
24 changes: 24 additions & 0 deletions doc/api/esm.md
Expand Up @@ -806,6 +806,30 @@ const require = createRequire(cwd() + '/<preload>');
}
```

In order to allow communication between the application and the loader another
bmeck marked this conversation as resolved.
Show resolved Hide resolved
argument is provided to the preload code `port`. This is available as a
bmeck marked this conversation as resolved.
Show resolved Hide resolved
parameter to the loader hook and inside of the source text returned by the hook.
Some care must be taken in order to properly `ref()` and `unref()` the
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does it mean to ref() and unref() something? How would I do so?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ref/unref determines if an event loop is kept alive to stuff. I could link to https://nodejs.org/dist/latest-v17.x/docs/api/worker_threads.html#portref ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, a link to the docs/example would help.

`MessagePort` to prevent a process from being in a state where it won't close
normally.

```js
/**
* This example causes
bmeck marked this conversation as resolved.
Show resolved Hide resolved
* @param {object} utilities
* @param {MessagePort} utilities.port
bmeck marked this conversation as resolved.
Show resolved Hide resolved
*/
export function globalPreloadCode({ port }) {
port.onmessage = (evt) => {
// ...
};
return `\
port.postMessage('I went to the Loader and back');
port.onmessage = eval;
`;
}
```

### Examples

The various loader hooks can be used together to accomplish wide-ranging
Expand Down
32 changes: 32 additions & 0 deletions lib/internal/modules/esm/initialize_import_meta.js
@@ -0,0 +1,32 @@
'use strict';

bmeck marked this conversation as resolved.
Show resolved Hide resolved
const { getOptionValue } = require('internal/options');
const experimentalImportMetaResolve =
getOptionValue('--experimental-import-meta-resolve');
const { PromisePrototypeThen, PromiseReject } = primordials;
const asyncESM = require('internal/process/esm_loader');

function createImportMetaResolve(defaultParentUrl) {
return async function resolve(specifier, parentUrl = defaultParentUrl) {
return PromisePrototypeThen(
asyncESM.esmLoader.resolve(specifier, parentUrl),
bmeck marked this conversation as resolved.
Show resolved Hide resolved
({ url }) => url,
(error) => (
error.code === 'ERR_UNSUPPORTED_DIR_IMPORT' ?
error.url : PromiseReject(error))
);
};
}

function initializeImportMeta(meta, context) {
const url = context.url;

// Alphabetical
if (experimentalImportMetaResolve)
meta.resolve = createImportMetaResolve(url);
bmeck marked this conversation as resolved.
Show resolved Hide resolved
meta.url = url;
}

module.exports = {
initializeImportMeta
};
52 changes: 44 additions & 8 deletions lib/internal/modules/esm/loader.js
Expand Up @@ -18,6 +18,7 @@ const {
SafeWeakMap,
globalThis,
} = primordials;
const { MessageChannel } = require('internal/worker/io');

const {
ERR_INVALID_ARG_TYPE,
Expand All @@ -39,6 +40,9 @@ const {
defaultResolve,
DEFAULT_CONDITIONS,
} = require('internal/modules/esm/resolve');
const {
initializeImportMeta
} = require('internal/modules/esm/initialize_import_meta');
const { defaultLoad } = require('internal/modules/esm/load');
const { translators } = require(
'internal/modules/esm/translators');
Expand Down Expand Up @@ -76,6 +80,8 @@ class ESMLoader {
defaultResolve,
];

#importMetaInitializer = initializeImportMeta;

/**
* Map of already-loaded CJS modules to use
*/
Expand Down Expand Up @@ -359,7 +365,15 @@ class ESMLoader {
if (!count) return;

for (let i = 0; i < count; i++) {
const preload = this.#globalPreloaders[i]();
const channel = new MessageChannel();
const insidePreload = channel.port1;
insidePreload.unref();
const insideLoader = channel.port2;
insideLoader.unref();
bmeck marked this conversation as resolved.
Show resolved Hide resolved

const preload = this.#globalPreloaders[i]({
port: insideLoader
});

if (preload == null) return;

Expand All @@ -373,22 +387,44 @@ class ESMLoader {
const { compileFunction } = require('vm');
const preloadInit = compileFunction(
preload,
['getBuiltin'],
['getBuiltin', 'port', 'setImportMetaCallback'],
{
filename: '<preload>',
}
);
const { NativeModule } = require('internal/bootstrap/loaders');

FunctionPrototypeCall(preloadInit, globalThis, (builtinName) => {
if (NativeModule.canBeRequiredByUsers(builtinName)) {
return require(builtinName);
let finished = false;
let replacedImportMetaInitializer = false;
let next = this.#importMetaInitializer;
try {
FunctionPrototypeCall(preloadInit, globalThis, (builtinName) => {
if (NativeModule.canBeRequiredByUsers(builtinName)) {
return require(builtinName);
}
throw new ERR_INVALID_ARG_VALUE('builtinName', builtinName);
}, insidePreload, (fn) => {
if (finished || typeof fn !== 'function') {
throw new ERR_INVALID_ARG_TYPE('fn', fn);
}
replacedImportMetaInitializer = true;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m struggling to follow this code. Are we in a callback here? Can this be split out?

I see that by now we’ve apparently replaced import.meta initializer, but where did that happen? Some comments might help.

const parent = next;
next = (meta, context) => {
return fn(meta, context, parent);
};
});
} finally {
finished = true;
if (replacedImportMetaInitializer) {
this.#importMetaInitializer = next;
GeoffreyBooth marked this conversation as resolved.
Show resolved Hide resolved
}
throw new ERR_INVALID_ARG_VALUE('builtinName', builtinName);
});
}
}
}

importMetaInitialize(meta, context) {
this.#importMetaInitializer(meta, context);
}
GeoffreyBooth marked this conversation as resolved.
Show resolved Hide resolved

/**
* Resolve the location of the module.
*
Expand Down
28 changes: 3 additions & 25 deletions lib/internal/modules/esm/translators.js
Expand Up @@ -8,8 +8,6 @@ const {
ObjectGetPrototypeOf,
ObjectPrototypeHasOwnProperty,
ObjectKeys,
PromisePrototypeThen,
PromiseReject,
SafeArrayIterator,
SafeMap,
SafeSet,
Expand Down Expand Up @@ -52,9 +50,6 @@ const {
const { maybeCacheSourceMap } = require('internal/source_map/source_map_cache');
const moduleWrap = internalBinding('module_wrap');
const { ModuleWrap } = moduleWrap;
const { getOptionValue } = require('internal/options');
const experimentalImportMetaResolve =
getOptionValue('--experimental-import-meta-resolve');
const asyncESM = require('internal/process/esm_loader');
const { emitWarningSync } = require('internal/process/warning');
const { TextDecoder } = require('internal/encoding');
Expand Down Expand Up @@ -111,25 +106,6 @@ async function importModuleDynamically(specifier, { url }) {
return asyncESM.esmLoader.import(specifier, url);
}

function createImportMetaResolve(defaultParentUrl) {
return async function resolve(specifier, parentUrl = defaultParentUrl) {
return PromisePrototypeThen(
asyncESM.esmLoader.resolve(specifier, parentUrl),
({ url }) => url,
(error) => (
error.code === 'ERR_UNSUPPORTED_DIR_IMPORT' ?
error.url : PromiseReject(error))
);
};
}

function initializeImportMeta(meta, { url }) {
// Alphabetical
if (experimentalImportMetaResolve)
meta.resolve = createImportMetaResolve(url);
meta.url = url;
}

// Strategy for loading a standard JavaScript module.
translators.set('module', async function moduleStrategy(url, source, isMain) {
assertBufferSource(source, true, 'load');
Expand All @@ -138,7 +114,9 @@ translators.set('module', async function moduleStrategy(url, source, isMain) {
debug(`Translating StandardModule ${url}`);
const module = new ModuleWrap(url, undefined, source, 0, 0);
moduleWrap.callbackMap.set(module, {
initializeImportMeta,
initializeImportMeta: (meta, wrap) => this.importMetaInitialize(meta, {
url: wrap.url
}),
importModuleDynamically,
});
return module;
Expand Down
39 changes: 39 additions & 0 deletions test/es-module/test-esm-loader-mock.mjs
@@ -0,0 +1,39 @@
// Flags: --loader ./test/fixtures/es-module-loaders/mock-loader.mjs
import '../common/index.mjs';
import assert from 'assert/strict';
import mock from 'node:mock';
bmeck marked this conversation as resolved.
Show resolved Hide resolved

mock('node:events', {
EventEmitter: 'This is mocked!'
});

// This resolves to node:events
assert.deepStrictEqual(await import('events'), Object.defineProperty({
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It resolves to node:events, you mean, it resolves to the mocked version?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes

__proto__: null,
EventEmitter: 'This is mocked!'
}, Symbol.toStringTag, {
enumerable: false,
value: 'Module'
}));

const mutator = mock('node:events', {
EventEmitter: 'This is mocked v2!'
});

const mockedV2 = await import('node:events');
assert.deepStrictEqual(mockedV2, Object.defineProperty({
__proto__: null,
EventEmitter: 'This is mocked v2!'
}, Symbol.toStringTag, {
enumerable: false,
value: 'Module'
}));

mutator.EventEmitter = 'This is mocked v3!';
assert.deepStrictEqual(mockedV2, Object.defineProperty({
__proto__: null,
EventEmitter: 'This is mocked v3!'
}, Symbol.toStringTag, {
enumerable: false,
value: 'Module'
}));
136 changes: 136 additions & 0 deletions test/fixtures/es-module-loaders/mock-loader.mjs
@@ -0,0 +1,136 @@
import {receiveMessageOnPort} from 'worker_threads';
bmeck marked this conversation as resolved.
Show resolved Hide resolved
const mockedModuleExports = new Map();
let currentMockVersion = 0;

/**
* FIXME: this is a hack to workaround loaders being
* single threaded for now
*/
function doDrainPort() {
let msg;
while (msg = receiveMessageOnPort(preloadPort)) {
onPreloadPortMessage(msg.message);
}
}
function onPreloadPortMessage({
mockVersion, resolved, exports
}) {
currentMockVersion = mockVersion;
mockedModuleExports.set(resolved, exports);
}
let preloadPort;
export function globalPreload({port}) {
preloadPort = port;
port.on('message', onPreloadPortMessage);
port.unref();
return `(${()=>{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This string wrapper leads to some awkward internals and interferes with syntax highlighting. Could this instead be a regular function declaration and return fn.toString()?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My syntax highlighting works with this XD. I avoid fn.toString usually since I don't like calling .toString after years of JS and it biting me on occasion, but for an example/test this is fine.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated to use String() so readers can see it is a string without reading to the bottom.

let mockedModules = new Map();
let mockVersion = 0;
const doMock = (resolved, replacementProperties) => {
let exports = Object.keys(replacementProperties);
let namespace = Object.create(null);
let listeners = [];
for (const name of exports) {
let currentValue = replacementProperties[name];
Object.defineProperty(namespace, name, {
enumerable: true,
get() {
return currentValue;
},
set(v) {
currentValue = v;
for (let fn of listeners) {
try {
fn(name);
} catch {
}
}
}
});
}
mockedModules.set(resolved, {
namespace,
listeners
});
mockVersion++;
port.postMessage({mockVersion, resolved, exports });
bmeck marked this conversation as resolved.
Show resolved Hide resolved
return namespace;
}
setImportMetaCallback((meta, context, parent) => {
if (context.url === 'node:mock') {
meta.doMock = doMock;
return;
}
if (context.url.startsWith('mock:')) {
let [proto, version, encodedTargetURL] = context.url.split(':');
let decodedTargetURL = decodeURIComponent(encodedTargetURL);
if (mockedModules.has(decodedTargetURL)) {
meta.mock = mockedModules.get(decodedTargetURL);
return;
}
}
parent(meta, context);
});
}})()`;
}
GeoffreyBooth marked this conversation as resolved.
Show resolved Hide resolved


// rewrites node: loading to mock: so that it can be intercepted
export function resolve(specifier, context, defaultResolve) {
if (specifier === 'node:mock') {
return {
url: specifier
};
}
doDrainPort();
const def = defaultResolve(specifier, context);
if (context.parentURL?.startsWith('mock:')) {
// do nothing, let it get the "real" module
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The mock: prefix returns the unmocked module?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could rename it to unmocked:? both are confusing to me

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or original: maybe? Or unmocked: is fine. Jest calls it actual: https://jestjs.io/uk/docs/bypassing-module-mocks

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actual: seems fine

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-reading, no mock: is the mocked version and without the mock: is the actual one @_o , this was done because there can be many versions of mocking the module so it needed to have things like a mock version number.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😵‍💫

I thought the same as Geoffrey—I thought I left a question/comment about it, but I can't find it now.

I'm more confused by the re-affirmation. Why would a user import a mock in their source code?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the user would never use mock: in a specifier, the underlying cache is using mock: for all the replacements of the "real" module. There can be N number of mocks of a given module, for example mock:1:node%3Afs would be 1 mocked version of fs while mock:2:node%3Afs would be a different mock, after the 1st one.

Basically this is needed for when multiple calls to import.meta.mock('node:fs', ...) create multiple modules. We can't actually replace the existing modules or unlink them from ESM graphs that already have been bound.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ohhhh, okay that makes sense! Might that be worth documenting?

I'm thinking this is too valuable to leave buried as a test fixture forever—at a later point, perhaps this should be surfaced in a more public-facing way (either in a doc, a quasi-official working example like our test loaders repo, etc).

} else if (mockedModuleExports.has(def.url)) {
return {
url: `mock:${currentMockVersion}:${encodeURIComponent(def.url)}`
};
};
return {
url: `${def.url}`
};
}

export function load(url, context, defaultLoad) {
doDrainPort();
if (url === 'node:mock') {
return {
source: 'export default import.meta.doMock',
format: 'module'
};
}
if (url.startsWith('mock:')) {
let [proto, version, encodedTargetURL] = url.split(':');
let ret = generateModule(mockedModuleExports.get(
decodeURIComponent(encodedTargetURL)
));
return {
source: ret,
format: 'module'
};
}
return defaultLoad(url, context);
}

function generateModule(exports) {
let body = 'export {};let mapping = {__proto__: null};'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
let body = 'export {};let mapping = {__proto__: null};'
let body = 'export {}; const mapping = {__proto__: null};'

You might want to put \n at the end of each of these string lines so that the mock source isn’t all on one line, to make debugging and stack traces easier.

for (const [i, name] of Object.entries(exports)) {
let key = JSON.stringify(name);
body += `var _${i} = import.meta.mock.namespace[${key}];`
body += `Object.defineProperty(mapping, ${key}, {enumerable: true,set(v) {_${i} = v;}});`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
body += `Object.defineProperty(mapping, ${key}, {enumerable: true,set(v) {_${i} = v;}});`
body += `Object.defineProperty(mapping, ${key}, {enumerable: true, set(v) {_${i} = v;}});`

body += `export {_${i} as ${name}};`;
}
body += `import.meta.mock.listeners.push(${
() => {
for (var k in mapping) {
mapping[k] = import.meta.mock.namespace[k];
}
}
});`
return body;
}
bmeck marked this conversation as resolved.
Show resolved Hide resolved