Repeated package.json reads in ts-loader ts api wrapper compared to tsc --build

[JasonKleban]

I'm noticing extreme performance problems comparing a tsc --build to webpack with ts-loader of the same project. Sysinternals Process Monitor is showing about 11k file reads from a 1 minute tsc --build and nearly 10x ReadFile events through ts-loader, with matchingly awful CPU and memory work before an ultimately successful build hours later. I am pursuing this as a ts-loader optimization problem, but I'm not sure if the ts api gives enough control to succeed.

TypeScript/src/compiler/moduleNameResolver.ts

Lines 963 to 966 in d36df0d

	const { path: candidate, parts } = normalizePathAndParts(combinePaths(containingDirectory, moduleName));
	const resolved = nodeLoadModuleByRelativeName(extensions, candidate, /*onlyRecordFailures*/ false, state, /*considerPackageJson*/ true);
	// Treat explicit "node_modules" import as an external library import.
	return resolved && toSearchResult({ resolved, isExternalLibraryImport: contains(parts, "node_modules") });

Perhaps it's exactly what it needs to do, but this /*considerPackageJson*/ true here is causing repeated reads/parsing of package.json files from dependencies in a row. I can override the injected readFile() and possibly avoid the read with a cache, but I don't think we can avoid the json parse because that is done internal to typescript.

Common dependencies with many internal .d.ts files are costing 6000+ read/parse of their package.json files in a run, with a long tail down to single reads.

Here is the call to compiler.resolveModuleName()

https://github.com/TypeStrong/ts-loader/blob/b8a70f91aa4a450603342e62b8c03bdd09c7a979/src/servicesHost.ts#L1175-L1199

Here is the injected readFile that is called back to read the contents of many package.json again and again, with each relative path import of a .d.ts file within its library:

https://github.com/TypeStrong/ts-loader/blob/b8a70f91aa4a450603342e62b8c03bdd09c7a979/src/servicesHost.ts#L96-L104

Once the packageJsonInfo is extracted, I suspect additional waste but one step at a time.

CC @sheetalkamat

[JasonKleban]

@sheetalkamat @RyanCavanaugh I created a self-contained repro for demonstrating this behavior here: https://github.com/JasonKleban/practices-2021

[JasonKleban]

When comparing the output of traceResolutions: true between the two build methods of the linked sample repo, tsc has 898 lines of output while the webpack & ts=loader version generates 6201 lines. Here is the first significant divergence and it may contain a clue?

and this is the next difference suggesting that tsc is getting a cache hit whereas ts-loader way is not attempting that (maybe, but perhaps the diff proximity is misleading here):

[sheetalkamat]

This is a multi repo issue and i suggest we fix this with:

1. Module resolution to cache readJson as well as getPackageJsonInfo and handle those things. Need to see when and which kind of invalidation should happen in watch and tsserver mode.
2. ts-loader could use module resolution cache that gets cleared like its other cache.
3. API in typescript to provide a way for ts-loader to use efficient module resolution cache we have in ts build state where if module resolution options dont change the module resolution cache is reused.

Also 1 might not be needed with 2 and 3

[sheetalkamat]

@JasonKleban i have PR #43700 and TypeStrong/ts-loader#1287 which i verified with your sample repro.
Can you check if the builds from these two help with your scenario and report. Thanks