'spack install' on Windows errors with DLL trying to link to itself

[danlipsa]

Steps to reproduce

spack uninstall --all
spack install paraview

Error message

[spack] C:\Users\dan.lipsa\projects\spack>spack install paraview
==> Installing bzip2-1.0.8-lua5n47dgpejpbfinuqewbceq3yoaca3 [1/38]
==> No binary for bzip2-1.0.8-lua5n47dgpejpbfinuqewbceq3yoaca3 found: installing from source
==> Using cached archive: C:\Users\dan.lipsa\projects\spack\var\spack\cache_source-cache\archive\ab\ab5a03176ee106d3f0fa90e381da478ddae405918153cca248e682cd0c4a2269.tar.gz
==> Ran patch() for bzip2
==> bzip2: Successfully installed bzip2-1.0.8-lua5n47dgpejpbfinuqewbceq3yoaca3
Stage: 0.10s. Install: 6.62s. Post-install: 0.14s. Total: 6.90s
[+] C:\Users\dan.lipsa\projects\spack\opt\spack\windows-windows10.0.22631-skylake\msvc-19.39.33523\bzip2-1.0.8-lua5n47dgpejpbfinuqewbceq3yoaca3
[+] C:\Program Files\cmake (external cmake-3.29.0-rc1-fj44ws7t4rj44farqvfogb2nqewueexs)
[+] C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja (external ninja-1.11.0-3mpj42ci3xgxgmwygavomw57fsc3ipir)
==> Error: [WinError 5] Access is denied: 'C:\Program Files (x86)\Windows Kits\10\bin\10.0.17763.0\x86\Microsoft.WindowsAzure.Storage.dll' -> 'C:\Program Files (x86)\Windows Kits\10\bin\Microsoft.WindowsAzure.Storage.dll'

Information on your system

• Spack: 0.22.0.dev0 (125206d)
• Python: 3.11.9
• Platform: windows-windows10.0.22631-skylake
• Concretizer: clingo

General information

• I have run spack debug report and reported the version of Spack/Python/Platform
• I have searched the issues of this repo and believe this is not a duplicate
• I have run the failing commands in debug mode and reported the output

[danlipsa]

The commit that introduces this error is:
125206d

[danlipsa]

@johnwparent This is the result of my git bisect.

[johnwparent]

Thanks for spending the time to track that down Dan!

[johnwparent]

attn @scheibelp

[johnwparent]

Three steps to proper resolution:

• The changes in python: always use a venv #40773 relating to Windows RPaths should not have been made in that PR and should be reverted in general because it breaks behavior. See Revert: c3b131e #44164
• Mitigations for linking against or for Windows system packages should be introduced (this will be done as part of Gitlab CI: Windows Configs #43967)
• Determine why python: always use a venv #40773 broke behavior and address any issues with our impl there when the change is not breaking develop.

[scheibelp]

Determine why #40773 broke behavior and address any issues with our impl there when the change is not breaking develop

You mean beyond the reasoning given in #42445 (i.e. that the generated manifest file is wrong if you generate additional paths after hooks are run)?

I'm confused why moving windows_establish_runtime_linkage slightly earlier would cause this problem: I would have expected that hook-execution time would be an appropriate time to execute; I don't see any files being renamed/removed as part of hooks that would make it succeed.

Overall I'm fine with #44164, but the PR description should explain that it isn't clear why #44164 is needed (i.e. above reasoning suggests the changes in c3b131e should be fine), and that it is a temporary fix to accommodate working CI in Windows.

[johnwparent]

You mean beyond the reasoning given in #42445

#42445 does not provide an explanation for why #40773 breaks functionality on Windows (I agree it doesn't make a lot of sense on the surface as to why that change breaks anything), just why that change was needed (although made in an decidedly sub-optimal context that prevented any testing) in the first place.

Change is not required for Windows CI but for developers trying to get Paraview working on Windows on develop (i.e. Dan).
CI will land shortly, so we want to be able to capitalize immediately when Dan gets Paraview working as that has been notoriously difficult to keep stable in develop w/o CI, and blockers like this will slow that down.

Will update PR description.

[scheibelp]

#42445 does not provide an explanation for why #40773 breaks functionality on Windows

Sorry by that you mean you don't consider

i.e. that the generated manifest file is wrong if you generate additional paths after hooks are run

a reason?

[johnwparent]

#42445 does not provide an explanation for why #40773 breaks functionality on Windows

Sorry by that you mean you don't consider

i.e. that the generated manifest file is wrong if you generate additional paths after hooks are run

a reason?

It's the reason why c3b131e was required in the first place. Not why that change is causing the issue Dan points out above, which I think we agree is somewhat of a head-scratcher at the moment, or am I missing something?

To be perfectly clear, I do not disagree with the need to resolve #42445, not do I disagree with c3b131e, I just do not see a clear reason why that change is causing the bug that inspired this issue, nor do I see how that first issue explains this bug, unless I'm misunderstanding either this bug, or the issue in #42445.
More tersely: It's not clear to me how fixing this bug (#42445) causes this one (#44163)

[scheibelp]

It's the reason why c3b131e was required in the first place.

OK cool

Not why that change is causing the issue Dan points out above,

OK yep, sounds like we agree that part's a mystery

Thanks for the clarification including links etc.

[johnwparent]

c3b131e Solves a bug where python_platlib is not available in the install subprocess. However, it introduces another bug that causes Windows externals to be involved as the root nodes in the RPath symlinking we perform, and this results in errors as we often do not have permissions to modify externals (this is good, the externals are often system libs that should not be modified by Spack, also in general it's bad practice to modify the users non spack installations). This is because we run the post install hooks on externals when they are part of the build dag after they are processed, which, thanks to c3b131e, includes the Windows RPath symlinks.
The solution, which is part of #43967, simply skips any RPath generation for externals on Windows.