Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows Platform Toolchain Resolution Broken #22164

Open
jxy-s opened this issue Apr 27, 2024 · 7 comments
Open

Windows Platform Toolchain Resolution Broken #22164

jxy-s opened this issue Apr 27, 2024 · 7 comments
Labels
P3 We're not considering working on this, but happy to review a PR. (No assignee) type: bug

Comments

@jxy-s
Copy link

jxy-s commented Apr 27, 2024

Description of the bug:

There appears to be a bug in toolchain resolution for bazel on Windows. There are two observed problems:

  1. --platforms for a target other than the host fails to resolve the toolchain.
  2. On an ARM64 Windows host the output binary is x64 when no platform constraints apply.

The consequence of this is that the toolchain resolution is incapable of producing a binary for anything other than x64. Counterintuitively, the ARM64 host produces an x64 binary

I've tested this on my Windows x64 and ARM64 machines. Both have all the MSVC installations necessary to cross-compile to another architecture. I believe the platform constraints should be applying here and the toolchain resolution should be locating the appropriate toolchain to use to compile for a target architecture other than the host. At very least it seems like the toolchain resolution on an ARM64 host should compile natively for that platform rather than produce a x64 binary.

Which category does this issue belong to?

C++ Rules

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I have put together a minimal example project here: https://github.com/jxy-s/bazel_win_xcomp

Which operating system are you running Bazel on?

Windows

What is the output of bazel info release?

release 7.1.1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

https://github.com/jxy-s/bazel_win_xcomp.git
dc6fc3cc910c94a799157a2d8516595cc01bcd7d

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

I believe so, I haven't been able to locate the commit yet.

Have you found anything relevant by searching the web?

I have found some suggestions and discussions on this topic. But nothing with concrete explanations as to why this is happening. This seems like a bug in the toolchain resolution. I do recognize that I could have a configuration problem in my example project. If that is the case I would appreciate guidance on what needs corrected.

#10963
#11938
#11973

Any other information, logs, or outputs that you want to share?

➜ Get-ChildItem -Include cl.exe -Recurse "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools" | ForEach-Object{echo $_.FullName}
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\Hostx64\arm\cl.exe
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\Hostx64\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\Hostx64\x64\cl.exe
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\Hostx64\x86\cl.exe
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\Hostx86\arm\cl.exe
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\Hostx86\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\Hostx86\x64\cl.exe
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\Hostx86\x86\cl.exe

➜ Get-ChildItem -Include vcvar*.bat -Recurse "C:\Program Files\Microsoft Visual Studio\2022\Community\VC" | ForEach-Object{echo $_.FullName}
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvars32.bat
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvars64.bat
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsamd64_arm.bat
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsamd64_arm64.bat
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsamd64_x86.bat
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsx86_amd64.bat
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsx86_arm.bat
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsx86_arm64.bat
➜ bazel query @local_config_cc//:toolchain --output=build
Starting local Bazel server and connecting to it...
# C:/users/jxy/_bazel_jxy/hvmoy72a/external/local_config_cc/BUILD:66:19
cc_toolchain_suite(
  name = "toolchain",
  tags = ["__CC_RULES_MIGRATION_DO_NOT_USE_WILL_BREAK__"],
  generator_name = "toolchain",
  generator_function = "cc_toolchain_suite",
  generator_location = "C:/users/jxy/_bazel_jxy/hvmoy72a/external/local_config_cc/BUILD:66:19",
  toolchains = {"armeabi-v7a|compiler": "@local_config_cc//:cc-compiler-armeabi-v7a", "x64_windows|msvc-cl": "@local_config_cc//:cc-compiler-x64_windows", "x64_x86_windows|msvc-cl": "@local_config_cc//:cc-compiler-x64_x86_windows", "x64_arm_windows|msvc-cl": "@local_config_cc//:cc-compiler-x64_arm_windows", "x64_arm64_windows|msvc-cl": "@local_config_cc//:cc-compiler-arm64_windows", "arm64_windows|msvc-cl": "@local_config_cc//:cc-compiler-arm64_windows", "x64_windows|msys-gcc": "@local_config_cc//:cc-compiler-x64_windows_msys", "x64_windows|mingw-gcc": "@local_config_cc//:cc-compiler-x64_windows_mingw", "x64_windows|clang-cl": "@local_config_cc//:cc-compiler-x64_windows-clang-cl", "x64_windows_msys": "@local_config_cc//:cc-compiler-x64_windows_msys", "x64_windows": "@local_config_cc//:cc-compiler-x64_windows", "x64_x86_windows": "@local_config_cc//:cc-compiler-x64_x86_windows", "x64_arm_windows": "@local_config_cc//:cc-compiler-x64_arm_windows", "x64_arm64_windows": "@local_config_cc//:cc-compiler-arm64_windows", "arm64_windows": "@local_config_cc//:cc-compiler-arm64_windows", "x64_arm64_windows|clang-cl": "@local_config_cc//:cc-compiler-arm64_windows-clang-cl", "arm64_windows|clang-cl": "@local_config_cc//:cc-compiler-arm64_windows-clang-cl", "armeabi-v7a": "@local_config_cc//:cc-compiler-armeabi-v7a"},
)
# Rule toolchain instantiated at (most recent call last):
#   C:/users/jxy/_bazel_jxy/hvmoy72a/external/local_config_cc/BUILD:66:19  in <toplevel>
#   C:/users/jxy/_bazel_jxy/hvmoy72a/external/rules_cc~/cc/defs.bzl:155:30 in cc_toolchain_suite
{
    "armeabi-v7a|compiler": "@local_config_cc//:cc-compiler-armeabi-v7a",
    "x64_windows|msvc-cl": "@local_config_cc//:cc-compiler-x64_windows",
    "x64_x86_windows|msvc-cl": "@local_config_cc//:cc-compiler-x64_x86_windows",
    "x64_arm_windows|msvc-cl": "@local_config_cc//:cc-compiler-x64_arm_windows",
    "x64_arm64_windows|msvc-cl": "@local_config_cc//:cc-compiler-arm64_windows",
    "arm64_windows|msvc-cl": "@local_config_cc//:cc-compiler-arm64_windows",
    "x64_windows|msys-gcc": "@local_config_cc//:cc-compiler-x64_windows_msys",
    "x64_windows|mingw-gcc": "@local_config_cc//:cc-compiler-x64_windows_mingw",
    "x64_windows|clang-cl": "@local_config_cc//:cc-compiler-x64_windows-clang-cl",
    "x64_windows_msys": "@local_config_cc//:cc-compiler-x64_windows_msys",
    "x64_windows": "@local_config_cc//:cc-compiler-x64_windows",
    "x64_x86_windows": "@local_config_cc//:cc-compiler-x64_x86_windows",
    "x64_arm_windows": "@local_config_cc//:cc-compiler-x64_arm_windows",
    "x64_arm64_windows": "@local_config_cc//:cc-compiler-arm64_windows",
    "arm64_windows": "@local_config_cc//:cc-compiler-arm64_windows",
    "x64_arm64_windows|clang-cl": "@local_config_cc//:cc-compiler-arm64_windows-clang-cl",
    "arm64_windows|clang-cl": "@local_config_cc//:cc-compiler-arm64_windows-clang-cl",
    "armeabi-v7a": "@local_config_cc//:cc-compiler-armeabi-v7a"
}
➜ bazel build //... --platforms=:windows_arm64 --toolchain_resolution_debug=.*
WARNING: Build option --toolchain_resolution_debug has changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed).
INFO: ToolchainResolution: Target platform //:windows_arm64: Selected execution platform @@local_config_platform//:host,
INFO: ToolchainResolution: Performing resolution of @@bazel_tools//tools/cpp:toolchain_type for target platform //:windows_arm64
      ToolchainResolution:   Rejected toolchain @@bazel_tools~cc_configure_extension~local_config_cc//:cc-compiler-armeabi-v7a; mismatching values: armv7, android
      ToolchainResolution:   Rejected toolchain @@bazel_tools~cc_configure_extension~local_config_cc//:cc-compiler-x64_windows; mismatching values: x86_64
      ToolchainResolution:   Rejected toolchain @@bazel_tools~cc_configure_extension~local_config_cc//:cc-compiler-armeabi-v7a; mismatching values: armv7, android
      ToolchainResolution:   Rejected toolchain @@bazel_tools~cc_configure_extension~local_config_cc//:cc-compiler-x64_windows; mismatching values: x86_64
      ToolchainResolution:   Rejected toolchain @@local_config_cc//:cc-compiler-armeabi-v7a; mismatching values: armv7, android
      ToolchainResolution:   Rejected toolchain @@local_config_cc//:cc-compiler-x64_windows; mismatching values: x86_64
      ToolchainResolution: No @@bazel_tools//tools/cpp:toolchain_type toolchain found for target platform //:windows_arm64.
INFO: ToolchainResolution: Target platform //:windows_arm64: Selected execution platform @@local_config_platform//:host,
ERROR: C:/users/jxy/_bazel_jxy/veelwacq/external/bazel_tools/tools/cpp/BUILD:58:19: in cc_toolchain_alias rule @@bazel_tools//tools/cpp:current_cc_toolchain:
Traceback (most recent call last):
        File "/virtual_builtins_bzl/common/cc/cc_toolchain_alias.bzl", line 26, column 48, in _impl
        File "/virtual_builtins_bzl/common/cc/cc_helper.bzl", line 219, column 17, in _find_cpp_toolchain
Error in fail: Unable to find a CC toolchain using toolchain resolution. Target: @@bazel_tools//tools/cpp:current_cc_toolchain, Platform: @@//:windows_arm64, Exec platform: @@local_config_platform//:host
ERROR: C:/users/jxy/_bazel_jxy/veelwacq/external/bazel_tools/tools/cpp/BUILD:58:19: Analysis of target '@@bazel_tools//tools/cpp:current_cc_toolchain' failed
ERROR: Analysis of target '//source:example' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.226s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully
@jxy-s
Copy link
Author

jxy-s commented Apr 27, 2024

Another data point I want to add here is that compiling with --cpu=x64_x86_windows locates the wrong toolchain and produces an x64 binary:

➜ bazel build //... --cpu=x64_x86_windows --toolchain_resolution_debug=.* -s
INFO: ToolchainResolution: Target platform @@local_config_platform//:host: Selected execution platform @@local_config_platform//:host,
INFO: ToolchainResolution: Performing resolution of @@bazel_tools//tools/cpp:toolchain_type for target platform @@local_config_platform//:host
      ToolchainResolution:   Rejected toolchain @@bazel_tools~cc_configure_extension~local_config_cc//:cc-compiler-armeabi-v7a; mismatching values: armv7, android
      ToolchainResolution:   Toolchain @@bazel_tools~cc_configure_extension~local_config_cc//:cc-compiler-x64_windows is compatible with target plaform, searching for execution platforms:
      ToolchainResolution:     Compatible execution platform @@local_config_platform//:host
      ToolchainResolution:   All execution platforms have been assigned a @@bazel_tools//tools/cpp:toolchain_type toolchain, stopping
      ToolchainResolution: Recap of selected @@bazel_tools//tools/cpp:toolchain_type toolchains for target platform @@local_config_platform//:host:
      ToolchainResolution:   Selected @@bazel_tools~cc_configure_extension~local_config_cc//:cc-compiler-x64_windows to run on execution platform @@local_config_platform//:host
INFO: ToolchainResolution: Target platform @@local_config_platform//:host: Selected execution platform @@local_config_platform//:host, type @@bazel_tools//tools/cpp:toolchain_type -> toolchain @@bazel_tools~cc_configure_extension~local_config_cc//:cc-compiler-x64_windows
INFO: ToolchainResolution: Target platform @@local_config_platform//:host: Selected execution platform @@local_config_platform//:host,
INFO: ToolchainResolution: Target platform @@local_config_platform//:host: Selected execution platform @@local_config_platform//:host,
INFO: Analyzed target //source:example (67 packages loaded, 302 targets configured).
SUBCOMMAND: # //source:example [action 'Compiling source/main.cpp', configuration: b604a8eea7993d2c3838e73c2e9a47dc2a8837d32db4ab1fffb8dda52c4bebc5, execution platform: @@local_config_platform//:host, mnemonic: CppCompile]
cd /d C:/users/jxy/_bazel_jxy/veelwacq/execroot/_main
  SET INCLUDE=C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\include;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\ATLMFC\include;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\VS\include;C:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um
    SET PATH=C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\HostX64\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\VC\VCPackages;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\TestWindow;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer;C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Current\bin\Roslyn;C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8 Tools\x64\;C:\Program Files (x86)\HTML Help Workshop;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\FSharp\Tools;C:\Program Files\Microsoft Visual Studio\2022\Community\Team Tools\DiagnosticsHub\Collector;C:\Program Files (x86)\Windows Kits\10\bin\10.0.22621.0\\x64;C:\Program Files (x86)\Windows Kits\10\bin\\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\\MSBuild\Current\Bin\amd64;C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\Tools\;;C:\WINDOWS\system32;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\VC\Linux\bin\ConnectionManagerExe;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\vcpkg
    SET PWD=/proc/self/cwd
    SET RUNFILES_MANIFEST_ONLY=1
    SET TEMP=C:\Users\jxy\AppData\Local\Temp
    SET TMP=C:\Users\jxy\AppData\Local\Temp
    SET VSLANG=1033
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\HostX64\x64\cl.exe @bazel-out/x64_x86_windows-fastbuild/bin/source/_objs/example/main.obj.params
# Configuration: b604a8eea7993d2c3838e73c2e9a47dc2a8837d32db4ab1fffb8dda52c4bebc5
# Execution platform: @@local_config_platform//:host
SUBCOMMAND: # //source:example [action 'Linking source/example.exe', configuration: b604a8eea7993d2c3838e73c2e9a47dc2a8837d32db4ab1fffb8dda52c4bebc5, execution platform: @@local_config_platform//:host, mnemonic: CppLink]
cd /d C:/users/jxy/_bazel_jxy/veelwacq/execroot/_main
  SET LIB=C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\ATLMFC\lib\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\lib\x64;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64;C:\Program Files (x86)\Windows Kits\10\lib\10.0.22621.0\ucrt\x64;C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64
    SET PATH=C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\HostX64\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\VC\VCPackages;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\TestWindow;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer;C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Current\bin\Roslyn;C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8 Tools\x64\;C:\Program Files (x86)\HTML Help Workshop;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\FSharp\Tools;C:\Program Files\Microsoft Visual Studio\2022\Community\Team Tools\DiagnosticsHub\Collector;C:\Program Files (x86)\Windows Kits\10\bin\10.0.22621.0\\x64;C:\Program Files (x86)\Windows Kits\10\bin\\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\\MSBuild\Current\Bin\amd64;C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\Tools\;;C:\WINDOWS\system32;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\VC\Linux\bin\ConnectionManagerExe;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\vcpkg
    SET PWD=/proc/self/cwd
    SET RUNFILES_MANIFEST_ONLY=1
    SET TEMP=C:\Users\jxy\AppData\Local\Temp
    SET TMP=C:\Users\jxy\AppData\Local\Temp
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\HostX64\x64\link.exe @bazel-out/x64_x86_windows-fastbuild/bin/source/example.exe-2.params
# Configuration: b604a8eea7993d2c3838e73c2e9a47dc2a8837d32db4ab1fffb8dda52c4bebc5
# Execution platform: @@local_config_platform//:host
INFO: Found 1 target...
Target //source:example up-to-date:
  bazel-bin/source/example.exe
INFO: Elapsed time: 1.865s, Critical Path: 1.43s
INFO: 7 processes: 5 internal, 2 local.
INFO: Build completed successfully, 7 total actions

@jxy-s
Copy link
Author

jxy-s commented Apr 27, 2024

I tested --cpu=x64_x86_windows on 6.5.0 and it worked then, so seems like a regression.

➜ bazel info release
release 6.5.0
➜ bazel build //... --cpu=x64_x86_windows --toolchain_resolution_debug=.* -s
Starting local Bazel server and connecting to it...
INFO: ToolchainResolution: Target platform @local_config_platform//:host: Selected execution platform @local_config_platform//:host,
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-armeabi-v7a; mismatching values: armv7, android
INFO: ToolchainResolution:   Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: execution @local_config_platform//:host: Selected toolchain @local_config_cc//:cc-compiler-x64_windows
INFO: ToolchainResolution: Target platform @local_config_platform//:host: Selected execution platform @local_config_platform//:host, type @bazel_tools//tools/cpp:toolchain_type -> toolchain @local_config_cc//:cc-compiler-x64_windows
INFO: ToolchainResolution: Target platform @local_config_platform//:host: Selected execution platform @local_config_platform//:host,
INFO: ToolchainResolution: Target platform @local_config_platform//:host: Selected execution platform @local_config_platform//:host,
INFO: Analyzed target //source:example (37 packages loaded, 175 targets configured).
INFO: Found 1 target...
SUBCOMMAND: # //source:example [action 'Compiling source/main.cpp', configuration: e4f2f605dc2afa5bf669f12664e665936e77cccf3ca3a99716e501506fa71141, execution platform: @local_config_platform//:host]
cd /d C:/users/jxy/_bazel_jxy/veelwacq/execroot/__main__
  SET INCLUDE=C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\include;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\ATLMFC\include;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\VS\include;C:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um
    SET PATH=C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\HostX64\x86;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\HostX64\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\VC\VCPackages;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\TestWindow;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer;C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Current\bin\Roslyn;C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8 Tools\x64\;C:\Program Files (x86)\HTML Help Workshop;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\FSharp\Tools;C:\Program Files\Microsoft Visual Studio\2022\Community\Team Tools\DiagnosticsHub\Collector;C:\Program Files (x86)\Windows Kits\10\bin\10.0.22621.0\\x64;C:\Program Files (x86)\Windows Kits\10\bin\\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\\MSBuild\Current\Bin\amd64;C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\Tools\;;C:\WINDOWS\system32;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\VC\Linux\bin\ConnectionManagerExe;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\vcpkg
    SET PWD=/proc/self/cwd
    SET RUNFILES_MANIFEST_ONLY=1
    SET TEMP=C:\Users\jxy\AppData\Local\Temp
    SET TMP=C:\Users\jxy\AppData\Local\Temp
    SET VSLANG=1033
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\HostX64\x86\cl.exe /nologo /DCOMPILER_MSVC /DNOMINMAX /D_WIN32_WINNT=0x0601 /D_CRT_SECURE_NO_DEPRECATE /D_CRT_SECURE_NO_WARNINGS /bigobj /Zm500 /EHsc /wd4351 /wd4291 /wd4250 /wd4996 /I. /Ibazel-out/x64_x86_windows-fastbuild/bin /Iexternal/bazel_tools /Ibazel-out/x64_x86_windows-fastbuild/bin/external/bazel_tools /DBAZEL_CURRENT_REPOSITORY="" /showIncludes /MD /Od /Z7 /wd4117 -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" /Fobazel-out/x64_x86_windows-fastbuild/bin/source/_objs/example/main.obj /c source/main.cpp
# Configuration: e4f2f605dc2afa5bf669f12664e665936e77cccf3ca3a99716e501506fa71141
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # //source:example [action 'Linking source/example.exe', configuration: e4f2f605dc2afa5bf669f12664e665936e77cccf3ca3a99716e501506fa71141, execution platform: @local_config_platform//:host]
cd /d C:/users/jxy/_bazel_jxy/veelwacq/execroot/__main__
  SET LIB=C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\ATLMFC\lib\x86;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\lib\x86;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x86;C:\Program Files (x86)\Windows Kits\10\lib\10.0.22621.0\ucrt\x86;C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x86
    SET PATH=C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\HostX64\x86;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\HostX64\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\VC\VCPackages;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\TestWindow;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer;C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Current\bin\Roslyn;C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8 Tools\x64\;C:\Program Files (x86)\HTML Help Workshop;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\FSharp\Tools;C:\Program Files\Microsoft Visual Studio\2022\Community\Team Tools\DiagnosticsHub\Collector;C:\Program Files (x86)\Windows Kits\10\bin\10.0.22621.0\\x64;C:\Program Files (x86)\Windows Kits\10\bin\\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\\MSBuild\Current\Bin\amd64;C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\Tools\;;C:\WINDOWS\system32;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\VC\Linux\bin\ConnectionManagerExe;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\vcpkg
    SET PWD=/proc/self/cwd
    SET RUNFILES_MANIFEST_ONLY=1
    SET TEMP=C:\Users\jxy\AppData\Local\Temp
    SET TMP=C:\Users\jxy\AppData\Local\Temp
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\HostX64\x86\link.exe @bazel-out/x64_x86_windows-fastbuild/bin/source/example.exe-2.params
# Configuration: e4f2f605dc2afa5bf669f12664e665936e77cccf3ca3a99716e501506fa71141
# Execution platform: @local_config_platform//:host
Target //source:example up-to-date:
  bazel-bin/source/example.exe
INFO: Elapsed time: 16.426s, Critical Path: 1.56s
INFO: 6 processes: 4 internal, 2 local.
INFO: Build completed successfully, 6 total actions

Although if I specify the platform constraints using 6.5.0 it doesn't do the needful:

➜ bazel build //... --platforms=:windows_x86 --toolchain_resolution_debug=.* -s
INFO: Build options --cpu and --platforms have changed, discarding analysis cache.
INFO: ToolchainResolution: Target platform //:windows_x86: Selected execution platform @local_config_platform//:host,
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform //:windows_x86: Rejected toolchain @local_config_cc//:cc-compiler-armeabi-v7a; mismatching values: armv7, android
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform //:windows_x86: Rejected toolchain @local_config_cc//:cc-compiler-x64_windows; mismatching values: x86_64
INFO: ToolchainResolution:   Type @bazel_tools//tools/cpp:toolchain_type: target platform //:windows_x86: No toolchains found.
INFO: ToolchainResolution: Target platform //:windows_x86: Selected execution platform @local_config_platform//:host,
INFO: ToolchainResolution: Target platform @local_config_platform//:host: Selected execution platform @local_config_platform//:host,
INFO: Analyzed target //source:example (0 packages loaded, 176 targets configured).
INFO: Found 1 target...
SUBCOMMAND: # //source:example [action 'Compiling source/main.cpp', configuration: f7eb43f60db399aa1e3c6eeb902137b974d432bca91e621dbcf1238df6a5e5ae, execution platform: @local_config_platform//:host]
cd /d C:/users/jxy/_bazel_jxy/veelwacq/execroot/__main__
  SET INCLUDE=C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\include;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\ATLMFC\include;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\VS\include;C:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um
    SET PATH=C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\HostX64\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\VC\VCPackages;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\TestWindow;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer;C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Current\bin\Roslyn;C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8 Tools\x64\;C:\Program Files (x86)\HTML Help Workshop;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\FSharp\Tools;C:\Program Files\Microsoft Visual Studio\2022\Community\Team Tools\DiagnosticsHub\Collector;C:\Program Files (x86)\Windows Kits\10\bin\10.0.22621.0\\x64;C:\Program Files (x86)\Windows Kits\10\bin\\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\\MSBuild\Current\Bin\amd64;C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\Tools\;;C:\WINDOWS\system32;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\VC\Linux\bin\ConnectionManagerExe;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\vcpkg
    SET PWD=/proc/self/cwd
    SET RUNFILES_MANIFEST_ONLY=1
    SET TEMP=C:\Users\jxy\AppData\Local\Temp
    SET TMP=C:\Users\jxy\AppData\Local\Temp
    SET VSLANG=1033
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\HostX64\x64\cl.exe /nologo /DCOMPILER_MSVC /DNOMINMAX /D_WIN32_WINNT=0x0601 /D_CRT_SECURE_NO_DEPRECATE /D_CRT_SECURE_NO_WARNINGS /bigobj /Zm500 /EHsc /wd4351 /wd4291 /wd4250 /wd4996 /I. /Ibazel-out/x64_windows-fastbuild/bin /Iexternal/bazel_tools /Ibazel-out/x64_windows-fastbuild/bin/external/bazel_tools /DBAZEL_CURRENT_REPOSITORY="" /showIncludes /MD /Od /Z7 /wd4117 -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" /Fobazel-out/x64_windows-fastbuild/bin/source/_objs/example/main.obj /c source/main.cpp
# Configuration: f7eb43f60db399aa1e3c6eeb902137b974d432bca91e621dbcf1238df6a5e5ae
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # //source:example [action 'Linking source/example.exe', configuration: f7eb43f60db399aa1e3c6eeb902137b974d432bca91e621dbcf1238df6a5e5ae, execution platform: @local_config_platform//:host]
cd /d C:/users/jxy/_bazel_jxy/veelwacq/execroot/__main__
  SET LIB=C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\ATLMFC\lib\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\lib\x64;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64;C:\Program Files (x86)\Windows Kits\10\lib\10.0.22621.0\ucrt\x64;C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64
    SET PATH=C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\HostX64\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\VC\VCPackages;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\TestWindow;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer;C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Current\bin\Roslyn;C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8 Tools\x64\;C:\Program Files (x86)\HTML Help Workshop;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\FSharp\Tools;C:\Program Files\Microsoft Visual Studio\2022\Community\Team Tools\DiagnosticsHub\Collector;C:\Program Files (x86)\Windows Kits\10\bin\10.0.22621.0\\x64;C:\Program Files (x86)\Windows Kits\10\bin\\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\\MSBuild\Current\Bin\amd64;C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\Tools\;;C:\WINDOWS\system32;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja;C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\VC\Linux\bin\ConnectionManagerExe;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\vcpkg
    SET PWD=/proc/self/cwd
    SET RUNFILES_MANIFEST_ONLY=1
    SET TEMP=C:\Users\jxy\AppData\Local\Temp
    SET TMP=C:\Users\jxy\AppData\Local\Temp
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\HostX64\x64\link.exe @bazel-out/x64_windows-fastbuild/bin/source/example.exe-2.params
# Configuration: f7eb43f60db399aa1e3c6eeb902137b974d432bca91e621dbcf1238df6a5e5ae
# Execution platform: @local_config_platform//:host
Target //source:example up-to-date:
  bazel-bin/source/example.exe
INFO: Elapsed time: 1.672s, Critical Path: 1.39s
INFO: 6 processes: 4 internal, 2 local.
INFO: Build completed successfully, 6 total actions

@jxy-s
Copy link
Author

jxy-s commented Apr 28, 2024

This does not explain the cross compilation issue on Windows x64. But, I tracked down one commit that is related for Windows ARM64: https://github.com/bazelbuild/bazel/pull/14738/files

@StarlarkMethod(
name = "arch",
structField = true,
doc =
"A string identifying the architecture Bazel is running on (the value of the \"os.arch\""
+ " Java property converted to lower case).")
public String getArch() {
return System.getProperty("os.arch").toLowerCase(Locale.ROOT);
}
}

Since on an ARM64 machine running an x64 Bazl binary runs under x64 emulation on the host OS - this is always going to assume the host is x64. This might seem nominal but when selecting the toolchain this causes a problem. Few ways to address this...

  • detect the actual host OS architecture using PROCESSOR_IDENTIFIER (not PROCESSOR_ARCHITECTURE)
  • use an ARM64 version of Bazel

I am using bazelisk which doesn't provide an ARM64 version (or doesn't install the host-native Bazel). I tested with the ARM64 Windows version of Bazel and it does detect the host OS correctly and actually produces a native binary.

Perhaps an issue needs raised over on the bazelisk repo for it to correctly detect the host architecture and install/use the ARM64 Bazel?

With respect to cross compiling on Windows ARM64 with a native ARM64 Bazel, testing shows something is still broken:

➜ bazel build //... --platforms=:windows_x86 --toolchain_resolution_debug=.* -s
WARNING: Build option --toolchain_resolution_debug has changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed).
INFO: ToolchainResolution: Target platform //:windows_x86: Selected execution platform @@local_config_platform//:host,
INFO: ToolchainResolution: Performing resolution of @@bazel_tools//tools/cpp:toolchain_type for target platform //:windows_x86
      ToolchainResolution:   Rejected toolchain @@bazel_tools~cc_configure_extension~local_config_cc//:cc-compiler-arm64_windows; mismatching values: aarch64
      ToolchainResolution:   Rejected toolchain @@bazel_tools~cc_configure_extension~local_config_cc//:cc-compiler-armeabi-v7a; mismatching values: armv7, android
      ToolchainResolution:   Rejected toolchain @@bazel_tools~cc_configure_extension~local_config_cc//:cc-compiler-arm64_windows; mismatching values: aarch64
      ToolchainResolution:   Rejected toolchain @@bazel_tools~cc_configure_extension~local_config_cc//:cc-compiler-armeabi-v7a; mismatching values: armv7, android
      ToolchainResolution:   Rejected toolchain @@local_config_cc//:cc-compiler-arm64_windows; mismatching values: aarch64
      ToolchainResolution:   Rejected toolchain @@local_config_cc//:cc-compiler-armeabi-v7a; mismatching values: armv7, android
      ToolchainResolution: No @@bazel_tools//tools/cpp:toolchain_type toolchain found for target platform //:windows_x86.
INFO: ToolchainResolution: Target platform //:windows_x86: Selected execution platform @@local_config_platform//:host,
ERROR: C:/users/jxy/_bazel_jxy/f63mpks4/external/bazel_tools/tools/cpp/BUILD:58:19: in cc_toolchain_alias rule @@bazel_tools//tools/cpp:current_cc_toolchain:
Traceback (most recent call last):
        File "/virtual_builtins_bzl/common/cc/cc_toolchain_alias.bzl", line 26, column 48, in _impl
        File "/virtual_builtins_bzl/common/cc/cc_helper.bzl", line 219, column 17, in _find_cpp_toolchain
Error in fail: Unable to find a CC toolchain using toolchain resolution. Target: @@bazel_tools//tools/cpp:current_cc_toolchain, Platform: @@//:windows_x86, Exec platform: @@local_config_platform//:host
ERROR: C:/users/jxy/_bazel_jxy/f63mpks4/external/bazel_tools/tools/cpp/BUILD:58:19: Analysis of target '@@bazel_tools//tools/cpp:current_cc_toolchain' failed
ERROR: Analysis of target '//source:example' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.294s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully

@fmeum
Copy link
Collaborator

fmeum commented May 2, 2024

As the author of #14738, I would strongly suggest fixing this in Bazelisk. Many parts of Bazel use the architecture of the Bazel binary to reason about the host OS, so even if we reverted a part of this PR, we would likely still be facing other issues.

@jxy-s
Copy link
Author

jxy-s commented May 2, 2024

Agree with addressing the architecture problem in bazelisk. I don't think that resolves the toolchain resolution issue. Unclear yet if platforms ever worked correctly for Windows.

@comius comius added P3 We're not considering working on this, but happy to review a PR. (No assignee) and removed team-Rules-CPP Issues for C++ rules untriaged labels May 3, 2024
@sluongng
Copy link
Contributor

sluongng commented May 6, 2024

It's not a problem with platform toolchain resolution.
This seems to be a problem with @local_config_cc toolchain, the default "host toolchain" Bazel setup, made a false impression that it's multi-platforms while it's not.

The host toolchain is almost always made with the assumption that host = exec = target, so cross-compilation is usually out of the scope of such a toolchain. I think the easiest way for you to resolve this problem is to define your own toolchain that support cross-compilation.


Another problem here is that you are confusing the old toolchain-crosstool resolution, which is based on the cc_toolchain_suite and corresponding --cpu flag, And the new toolchain-platforms resolution, which is based on the --platforms and --extra_toolchains and --extra_execution_platforms flags.

These 2 mechanisms serve the same purpose, but the latter is newer and more flexible while the former will be deprecated away soon. Once you start to roll your cc toolchain, you would want to use the toolchain-platform resolution approach and stay away from --cpu flag as much as possible.

@jxy-s
Copy link
Author

jxy-s commented May 6, 2024

@sluongng thanks for the information. I'll look into this further when I have time. It is fairly common to compile for a target that is not the host on Windows. It would be nice if Bazel continued to support this out of the box.

These 2 mechanisms serve the same purpose, but the latter is newer and more flexible while the former will be deprecated away soon.

It seemed to have worked in the past with --cpu and it now seems to be a regression. I recognize that will be removed in the future. But there is now no replacement, outside of rolling your own toolchain.

I merged a patch to the EWDK toolchain repo to suit my needs for now: 0xf005ba11/bazel_ewdk_cc@1e0bd43

Perhaps I'll come back and contribute here when I have time to fully read through the Windows toolchain code in this repo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P3 We're not considering working on this, but happy to review a PR. (No assignee) type: bug
Projects
None yet
Development

No branches or pull requests

7 participants