Long node_modules paths cannot be found on Windows when LongPathsEnabled enabled

[karlhorky]

Version

v20.9.0

Platform

Microsoft Windows NT 10.0.20348.0 x64

Subsystem

No response

What steps will reproduce the bug?

I created a reproduction repo at https://github.com/karlhorky/node-js-max_path-windows-bug . The steps in the GitHub Actions workflow file reproduce the bug:

# Test whether Windows support for long paths is enabled
(Get-ItemProperty "HKLM:System\CurrentControlSet\Control\FileSystem").LongPathsEnabled
# 1

# Clone, setup Node.js

# Install dependencies with npm
npm install

# Run program in short path (✅ works)
node eslint.config.js

# Make directory with long path and copy all files there
mkdir long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\
cp eslint.config.js long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\
cp index.js long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\
cp package.json long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\

# Change to directory with long path
cd long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\

# Install dependencies with npm
npm install

# Run program in long path (💥 crashes)
node eslint.config.js

The error that shows up is this:

node:internal/process/esm_loader:[4](https://github.com/karlhorky/node-js-max_path-windows-bug/actions/runs/6890011394/job/18742165029#step:11:5)0
      internalBinding('errors').triggerUncaughtException(
                                ^

Error: Cannot find package 'D:\a\node-js-max_path-windows-bug\node-js-max_path-windows-bug\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\node_modules\eslint-plugin-import\package.json' imported from D:\a\node-js-max_path-windows-bug\node-js-max_path-windows-bug\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\eslint.config.js
Did you mean to import eslint-plugin-import/lib/index.js?
    at legacyMainResolve (node:internal/modules/esm/resolve:189:26)
    at packageResolve (node:internal/modules/esm/resolve:776:14)
    at moduleResolve (node:internal/modules/esm/resolve:838:20)
    at defaultResolve (node:internal/modules/esm/resolve:1043:11)
    at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:383:12)
    at ModuleLoader.resolve (node:internal/modules/esm/loader:3[5](https://github.com/karlhorky/node-js-max_path-windows-bug/actions/runs/6890011394/job/18742165029#step:11:6)2:25)
    at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:228:38)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:85:39)
    at link (node:internal/modules/esm/module_job:84:3[6](https://github.com/karlhorky/node-js-max_path-windows-bug/actions/runs/6890011394/job/18742165029#step:11:7)) {
  code: 'ERR_MODULE_NOT_FOUND'
}

Node.js v20.[9](https://github.com/karlhorky/node-js-max_path-windows-bug/actions/runs/6890011394/job/18742165029#step:11:10).0
Error: Process completed with exit code 1.

How often does it reproduce? Is there a required condition?

Always

What is the expected behavior? Why is that the expected behavior?

If Windows support for long paths is turned on (see LongPathsEnabled check above showing that it's enabled) I would expect Node.js to also support long paths (longer than the Windows ~260 characters MAX_PATH limit)

What do you see instead?

The crash and error above

Additional information

More information about LongPathsEnabled :

• MAX_PATH lengths on Windows environment actions/runner-images#1052 (comment)

Context:

I'm here after originally reporting the bug in ESLint (I thought that this comment by @bzoz meant that Node.js is not affected by the 260 character path limit on Windows :

• Bug: Long node_modules plugin paths cannot be found on Windows eslint/eslint#17763

[MrJithil]

cc: @nodejs/platform-windows

[StefanStojanovic]

I took a look at it, but found another issue instead. @karlhorky I forked your repo and the failing run can be found here.

I made long path longer, because the original one was shorter then 260 characters. After I did it, Even running npm install (or anything else, example here) started failing. Since I'm seeing same error regardless if I'm using CMD or PowerShell it seems safe to assume this is something Windows specific.

Back to the issue you reported, importer uses file URLs internally, thus cannot use \\?\ prefix required for handling long paths. I'll investigate this further, and add another comment based on my findings.

[karlhorky]

Thanks for the response, and for the investigation! 🙌

I made long path longer, because the original one was shorter then 260 characters. After I did it, Even running npm install (or anything else, example here) started failing

I guess you mean "anything else Node.js-related"? Because the cp command on the previous step doesn't fail:

cp eslint.config.js long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\

But maybe the way that GitHub Actions is launching PowerShell + CMD has a problem - maybe a separate GitHub Actions-related problem cc @al-cheb @MaksimZhukov @chrispat:

An error occurred trying to start process 'C:\Program Files\PowerShell\7\pwsh.EXE' with working directory 'D:\a\node-js-max_path-windows-bug\node-js-max_path-windows-bug\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\'. The directory name is invalid.

I'll try to run various commands from a directory with a long path on my Windows machine locally and see what I find.

[karlhorky]

Ok, I ran some commands on a Windows 11 Pro x64 machine (both node and also cmd) I can confirm that running commands from within deep directories fails with The directory name is invalid:

PS C:\> node -v
v20.5.1

PS C:\> mkdir long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\


    Directory: C:\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\l
    ong-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-p
    ath\long-path\long-path\long-path\long-path


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----        17.11.2023     16:27                long-path


PS C:\> cd long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\
PS C:\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path> node -v
Program 'node.exe' failed to run: The directory name is invalidAt line:1 char:1
+ node -v
+ ~~~~~~~.
At line:1 char:1
+ node -v
+ ~~~~~~~
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed

Commands like cd still work:

PS C:\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path> cd .

cmd.exe also doesn't run:

PS C:\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path> cmd
Program 'cmd.exe' failed to run: The directory name is invalidAt line:1 char:1
+ cmd
+ ~~~.
At line:1 char:1
+ cmd
+ ~~~
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed

Also when running C:\Windows\System32\cmd.exe with the full path - still has a problem with the current directory being invalid:

PS C:\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path\long-path> C:\Windows\System32\cmd.exe
Program 'cmd.exe' failed to run: The directory name is invalidAt line:1 char:1
+ C:\Windows\System32\cmd.exe
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~.
At line:1 char:1
+ C:\Windows\System32\cmd.exe
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed

Wonder if someone from the Windows team involved in interop / this MAX_PATH limitation could confirm that running - I don't know of contacts who work on the Windows team, but maybe @JeremyKuhne (because of this blog post) or @iSazonov (because of involvement in PowerShell/PowerShell#17197 and PowerShell/PowerShell#13955) or @SteveL-MSFT (because of PowerShell/PowerShell involvement) ?

---

Anyway, back to the import / long path file reading issue, maybe at least this part could be supported by Node.js...

[StefanStojanovic]

Ok, I ran some commands on a Windows 11 Pro x64 machine (both node and also cmd) I can confirm that running commands from within deep directories fails with The directory name is invalid:

That is correct. cp is a command, not an executable, so I assume only applications are facing that issue.

Back to the import with long paths problem. Investigating a bit longer got me to one of my first PRs in this repo, especially to changes in the manifest file. Applying that change solved an issue for me, but I'd like to investigate a bit more to find the root cause. The original PR was dropped in favor of this one as that was enough to fix the issue and something similar should probably be possible here. I'll get back to this later this week to continue my work.

[karlhorky]

Regarding the "PowerShell unable to launch programs from long path" problem:

I've commented in an existing MAX_PATH issue in the PowerShell/PowerShell repo:

• Incorrect behavior running executable if current location path is longer than 256 characters PowerShell/PowerShell#13955 (comment)

I may need to open a new issue for this as well.

[StefanStojanovic]

After further investigation, I saw that some parts of the code responsible for module loading, do not add the required \\?\ prefix for Windows long path support. I was able to make a PoC fixing it without the changes to the manifest file, but that's just a starting point for further work.

[karlhorky]

Workaround (for pnpm)

pnpm install can sometimes create file / folder structures that have long paths, which can lead to confusing errors only on Windows such as "Cannot find package" with ESLint:

$ pnpm eslint . --max-warnings 0

Oops! Something went wrong! :(

ESLint: 8.53.0

Error: Cannot find package 'C:\Users\Victor\upleveled\testing-projects\next-js-example-winter-2024-atvie\node_modules\.pnpm\eslint-config-upleveled@8.0.0_@babel+core@7.24.4_@types+eslint@8.56.7_@types+node@20.12.4_@ty_lxumkk76msp4b3xunkneyh2dsu\node_modules\@next\eslint-plugin-next\package.json' imported from C:\Users\Victor\upleveled\testing-projects\next-js-example-winter-2024-atvie\node_modules\.pnpm\eslint-config-upleveled@8.0.0_@babel+core@7.24.4_@types+eslint@8.56.7_@types+node@20.12.4_@ty_lxumkk76msp4b3xunkneyh2dsu\node_modules\eslint-config-upleveled\index.js
Did you mean to import "@next/eslint-plugin-next/dist/index.js"?
    at legacyMainResolve (node:internal/modules/esm/resolve:215:26)
    at packageResolve (node:internal/modules/esm/resolve:841:14)
    at moduleResolve (node:internal/modules/esm/resolve:927:18)
    at defaultResolve (node:internal/modules/esm/resolve:1157:11)
    at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:390:12)
    at ModuleLoader.resolve (node:internal/modules/esm/loader:359:25)
    at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:234:38)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:87:39)
    at link (node:internal/modules/esm/module_job:86:36)

These errors are also caused by the Node.js incompatibility with long paths on Windows.

If you're encountering this, one thing you can try is to use pnpm with node-linker=hoisted in an .npmrc file in your project:

.npmrc

node-linker=hoisted

Source: https://github.com/upleveled/preflight-test-project-next-js-passing/blob/419e89d0eb4067d63373aa7daf703114f949896d/.github/workflows/ci.yml#L77-L81

cc @zkochan