Bug: pack:win fails (silently) when node_modules path is >260 characters

[ericis]

Issue

The makensis command fails silently when building the ".nsi" file fails because of a "client" subdirectory path that is longer than 260 characters.

Steps to replicate

1. On a Windows machine...
2. Create a source directory path that is approximately 250 characters in length *This is just the simplest way to replicate the >260 characters issue.
3. Create a new Oclif project
4. Run yarn install
5. Add a custom node dependency with a lot of dependencies
6. Package with yarn oclif pack:win
7. This issue should appear indirectly after failing silently with a subsequent error that it cannot find the "installer.exe" file (because compiling it with makensis failed due to the "MAX_PATH" issue described here)

What is the current behavior?

The command that fails silently is located here:

oclif/src/commands/pack/win.ts

Lines 318 to 320 in 81222ca

	await exec(
	`makensis ${installerBase}/${config.bin}.nsi | grep -v "\\[compress\\]" | grep -v "^File: Descending to"`,
	)

The offending NSI configuration for copying the "client" directory contents is located here:

oclif/src/commands/pack/win.ts

Line 76 in 81222ca

File /r client

A very similar issue is explained in the NSIS forums. However, the comments offer no resolution or further insight.

Also, a related, but completely different explanation of the issue can be found on StackOverflow.

The official Microsoft documentation for this behavior and the [in] lpFileName parameter.

By default, the name is limited to MAX_PATH characters. To extend this limit to 32,767 wide characters, prepend "\?" to the path. For more information, see Naming Files, Paths, and Namespaces.

As well as a note on current Windows builds:

Starting with Windows 10, Version 1607, you can opt-in to remove the MAX_PATH limitation without prepending "\?". See the "Maximum Path Length Limitation" section of Naming Files, Paths, and Namespaces for details.

I attempted to modify the local source of Oclif's "win.ts" file including debugging the const { stdout, stderr} = await exec(makensis ...) command output.

makensis stdout

Processing config: C:\Program Files (x86)\NSIS\nsisconf.nsh
Processing script file: "C:\Users\255450\src\carmax\github\devex-localhost\src\kmx-cli\tmp\windows-x86-installer/kmx.nsi" (ACP)

makensis stderr

File: failed opening file ".\client\node_modules\@opentelemetry\auto-instrumentations-node\node_modules\@opentelemetry\api-logs\node_modules\@opentelemetry\api\build\esm\baggage\context-helpers.js.map"
Error in script "C:\Users\NONEEDTOKNOWTHIS\src\my-cli\tmp\windows-x86-installer/my-cli.nsi" on line 34 -- aborting creation process

The "node_modules" directory structure and "context-helpers.js.map" file mentioned in the error do indeed exist and simply exceed the "MAX_PATH" issue that Microsoft has documented.

While debugging, I unsuccessfully attempted to modify the File /r client to instead read File /r \\\\?\\client as well as File /r \\\\?\\.\\client. The makensis command simply failed at that line with the error:

File: "\\?\.\client" -> no files found.
Usage: File [/nonfatal] [/a] ([/r] [/x filespec [...]] filespec [...] |
   /oname=outfile one_file_only)
Error in script "C:\Users\NONEEDTOKNOWTHIS\src\my-cli\tmp\windows-x86-installer/my-cli.nsi" on line 34 -- aborting creation process

While I am on Windows 11 and should not generally have this "MAX_PATH" issue (according to Microsoft's documentation). I also edited my local registry to explicitly enable long paths with the PowerShell command:

New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" `
-Name "LongPathsEnabled" -Value 1 -PropertyType DWORD -Force

What is the expected behavior?

Oclif packaging for Windows works with long paths. The "node_modules" directory structure is rather infamous for lengthy sub-directories and file paths with a thread of Windows-specific issues.

[ericis]

I thought that I had found a work-around (the silent failure of await exec(makensis ...) would still need to be fixed).

I tried:

1. Upgrading the oclif CLI project to use Yarn 4+

2. Configuring yarn to use classic "node_modules" setup:

   nodeLinker: node-modules
   nmHoistingLimits: dependencies

3. Explicitly installing the offending "long path" dependency e.g. "@opentelemetry\api" in my case

4. Re-running yarn oclif pack:win

🔥 However, the "./tmp/client/node_modules" directory structure does not match the project's yarn install directory structure.

Example of project's "node_modules" structure

*This example shows that both "api" and "api-logs" dependencies have been hoisted out of the "@opentelemetry/auto-instrumentations-node" dependency and up to the top.

Example of oclif's "tmp/my-cli/node_modules" structure

*This example shows that the incredibly deep, classic "node_modules" directory structure has magically reappeared and highlights the longest file path!

⁉ How does oclif generate these "tmp" directories? It doesn't seem to use the already existing "node_modules" directory as-is...

[ericis]

A lucky workaround for our specific project was to move the entire project to the shortest root path on disk as possible e.g. under "C:\src" with a short project directory name.