New @nrwl/node:node doesn't link transitive dependencies correctly

[jahumes]

Current Behavior

After upgrading to 13.8.4, the new @nrwl/node:node executor is doing something to our code that is causing the linking of our transitive dependencies to not be linked correctly. An example is that we include rxjs@7.4.0 in our application, but one of our main dependencies depends on rxjs@5.4.3. After the 13.8.4 version was implemented with the new version, when our node executor runs to serve our application for development, our dependency gets rxjs@7.4.0 which immediately causes it to crash.

We spent about 6 hours digging into everything we can in our application and have figured out that the only difference between when it works and when it breaks is the upgrade of nx from 13.8.3 to 13.8.4 (quick side note, this change doesn't really abide by semantic versioning because the API changed 😄 ). Our current theory is that something is brakes when the new executor creates child processes for node applications.

Expected Behavior

This is a regression and the expected behavior is that our dependencies' dependencies are linked correctly.

Steps to Reproduce

I will attempt to reproduce this, but this is internal code that I can't share. Since the basic functionality is that a dependency is getting the application's version of one of its' dependencies, hopefully, this isn't too hard to reproduce.

Failure Logs

Possibly unhandled TypeError: rxjs_1.Observable.timer is not a function

Environment

   Node : 14.19.0
   OS   : darwin x64
   yarn : 3.2.0

   nx : 13.8.8
   @nrwl/angular : undefined
   @nrwl/cli : 13.8.8
   @nrwl/cypress : 13.8.8
   @nrwl/detox : undefined
   @nrwl/devkit : 13.8.8
   @nrwl/eslint-plugin-nx : 13.8.8
   @nrwl/express : 13.8.8
   @nrwl/jest : 13.8.8
   @nrwl/js : 13.8.8
   @nrwl/linter : 13.8.8
   @nrwl/nest : undefined
   @nrwl/next : undefined
   @nrwl/node : 13.8.8
   @nrwl/nx-cloud : undefined
   @nrwl/react : 13.8.8
   @nrwl/react-native : undefined
   @nrwl/schematics : undefined
   @nrwl/storybook : 13.8.8
   @nrwl/tao : 13.8.8
   @nrwl/web : 13.8.8
   @nrwl/workspace : 13.8.8
   typescript : 4.4.4
   rxjs : 7.5.5
   ---------------------------------------
   Community plugins:

[jahumes]

It does appear that by combining the js and node packages and using primarily the js executor to serve node packages, that module loading has been changed. It was this before:

nx/packages/node/src/executors/execute/execute.impl.ts

Lines 76 to 79 in 7a982af

	subProcess = fork(event.outfile, options.args, {
	execArgv: getExecArgv(options),
	});
	}

And is now this:

nx/packages/node/src/executors/node/node.impl.ts

Lines 88 to 100 in fd64546

	subProcess = fork(
	joinPathFragments(__dirname, 'node-with-require-overrides'),
	options.args,
	{
	execArgv: getExecArgv(options),
	stdio: 'inherit',
	env: {
	...process.env,
	NX_FILE_TO_RUN: event.outfile,
	NX_MAPPINGS: JSON.stringify(mappings),
	},
	}
	);

I am wondering if this is what has changed that could possibly cause this issue.

nx/packages/node/src/executors/node/node-with-require-overrides.ts

Lines 1 to 20 in fd64546

	const Module = require('module');
	const originalLoader = Module._load;
	const mappings = JSON.parse(process.env.NX_MAPPINGS);
	const keys = Object.keys(mappings);
	const fileToRun = process.env.NX_FILE_TO_RUN;
	Module._load = function (request, parent) {
	if (!parent) return originalLoader.apply(this, arguments);
	const match = keys.find((k) => request === k);
	if (match) {
	const newArguments = [...arguments];
	newArguments[0] = mappings[match];
	return originalLoader.apply(this, newArguments);
	} else {
	return originalLoader.apply(this, arguments);
	}
	};
	require(fileToRun);

[jahumes]

I have verified that changing the fork to the old code causes this issue to go away. I changed the output of 13.8.8 from

function runProcess(event, options, mappings) {
    return (0, tslib_1.__awaiter)(this, void 0, void 0, function* () {
        subProcess = (0, child_process_1.fork)((0, devkit_1.joinPathFragments)(__dirname, 'node-with-require-overrides'), options.args, {
            execArgv: getExecArgv(options),
            stdio: 'inherit',
            env: Object.assign(Object.assign({}, process.env), { NX_FILE_TO_RUN: event.outfile, NX_MAPPINGS: JSON.stringify(mappings) }),
        });
        if (!options.watch) {
            return new Promise((resolve, reject) => {
                subProcess.on('exit', (code) => {
                    if (code === 0) {
                        resolve(undefined);
                    }
                    else {
                        reject();
                    }
                });
            });
        }
    });
}

to

function runProcess(event, options, mappings) {
    return (0, tslib_1.__awaiter)(this, void 0, void 0, function* () {
      if (subProcess || !event.success) {
        return;
      }
      subProcess = (0, child_process_1.fork)(event.outfile, options.args, {
        execArgv: getExecArgv(options),
      });
    });
}

And now everything works just fine

[jahumes]

@jaysoo I believe that I have figured out what is causing the issue. Basically, when the code is being run, it gets a list of mappings for nx specific libraries in the code below:

nx/packages/node/src/executors/node/node.impl.ts

Lines 61 to 81 in 1b7a3b2

	function calculateResolveMappings(
	context: ExecutorContext,
	options: NodeExecutorOptions
	) {
	const projectGraph = readCachedProjectGraph();
	const parsed = parseTargetString(options.buildTarget);
	const { dependencies } = calculateProjectDependencies(
	projectGraph,
	context.root,
	parsed.project,
	parsed.target,
	parsed.configuration
	);
	return dependencies.reduce((m, c) => {
	if (!c.outputs[0] && c.node.type === 'npm') {
	c.outputs[0] = `node_modules/${c.node.data.packageName}`;
	}
	m[c.name] = joinPathFragments(context.root, c.outputs[0]);
	return m;
	}, {});
	}

Once it has these libraries, it runs all require/import statements through an extended module loader at the location below:

nx/packages/node/src/executors/node/node-with-require-overrides.ts

Lines 1 to 20 in 1b7a3b2

	const Module = require('module');
	const originalLoader = Module._load;
	const mappings = JSON.parse(process.env.NX_MAPPINGS);
	const keys = Object.keys(mappings);
	const fileToRun = process.env.NX_FILE_TO_RUN;
	Module._load = function (request, parent) {
	if (!parent) return originalLoader.apply(this, arguments);
	const match = keys.find((k) => request === k);
	if (match) {
	const newArguments = [...arguments];
	newArguments[0] = mappings[match];
	return originalLoader.apply(this, newArguments);
	} else {
	return originalLoader.apply(this, arguments);
	}
	};
	require(fileToRun);

When it sees an import that has the same name as one of nx mapping generated above, it replaces the import with that mapping. An example of what an obfuscated version of this json looks like is below:

{
  "<Obfuscated>": "<Obfuscated>/node_modules/qauthentication",
  "csurf": "<Obfuscated>/node_modules/csurf",
  "express": "<Obfuscated>/node_modules/express",
  "crypto-js": "<Obfuscated>/node_modules/crypto-js",
  "axios": "<Obfuscated>/node_modules/axios",
  "date-fns": "<Obfuscated>/node_modules/date-fns",
  "<Obfuscated>/translations": "<Obfuscated>/dist/libs/translations",
  "glob": "<Obfuscated>/node_modules/glob",
  "dot-prop": "<Obfuscated>/node_modules/dot-prop",
  "yargs": "<Obfuscated>/node_modules/yargs",
  "deepmerge": "<Obfuscated>/node_modules/deepmerge",
  "catalog-ui-context": "<Obfuscated>/node_modules/catalog-ui-context",
  "qs": "<Obfuscated>/node_modules/qs",
  "rxjs": "<Obfuscated>/node_modules/rxjs",
  "<Obfuscated>/plugin-manager": "<Obfuscated>/node_modules/<Obfuscated>/plugin-manager",
  "react": "<Obfuscated>/node_modules/react",
  "lodash.clonedeep": "<Obfuscated>/node_modules/lodash.clonedeep",
  "clone": "<Obfuscated>/node_modules/clone",
  "lru-cache": "<Obfuscated>/node_modules/lru-cache",
  "winston": "<Obfuscated>/node_modules/winston",
  "winston-daily-rotate-file": "<Obfuscated>/node_modules/winston-daily-rotate-file",
  "morgan": "<Obfuscated>/node_modules/morgan",
  "uuid": "<Obfuscated>/node_modules/uuid",
  "prom-client": "<Obfuscated>/node_modules/prom-client",
  "@testing-library/react": "<Obfuscated>/node_modules/@testing-library/react",
  "node-cache": "<Obfuscated>/node_modules/node-cache",
  "cookie-parser": "<Obfuscated>/node_modules/cookie-parser",
  "apollo-server-express": "<Obfuscated>/node_modules/apollo-server-express",
  "apollo-server": "<Obfuscated>/node_modules/apollo-server",
  "graphql": "<Obfuscated>/node_modules/graphql",
  "apollo-datasource-rest": "<Obfuscated>/node_modules/apollo-datasource-rest",
  "dotenv": "<Obfuscated>/node_modules/dotenv"
}

This code is replacing our dependency's module resolution for rxjs (which goes to the node_modules directory of that dependency since it has a mismatch) with Nx's version of rxjs.

[hellivan]

#9284 should fix this issue

[FrozenPandaz]

Closed with #9284

[github-actions]

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.