cgo+windows broken in 4.0.0 (error: error reading '.'; iquote)

[johnskopis]

What version of rules_go are you using?

v0.25.1 (also tried master)

What version of gazelle are you using?

v0.22.3

What version of Bazel are you using?

4.0.0

Does this issue reproduce with the latest releases of all the above?

I suspect, yes.

What operating system and processor architecture are you using?

This is a x86_64 windows 10 machine.

Any other potentially useful information about your toolchain?

I haven't explicitly configured a toolchain. I launch bazel in a docker image and use local_cc.
This works fine in linux with clang or gcc but windows is a different beast.

An aside, are there any examples to run RBE locally? RBE seems very promising but it's undocumented

What did you do?

I am trying to build https://github.com/mattn/go-sqlite3 in windows.

TBH, I'm not familiar enough w/ windows to say for sure whether it's a bug or PEBKAC.

I created a repro here https://github.com/johnskopis/repro.

I was able to find a few related issues:

• cgo: use -I instead of -iquote for source header directories #2687 it's not the same issue but it looks similar to the last one -- it could also be an issue w/ my sandbox.
• PROTOBUF_MAYBE_CONSTEXPR should not be constexpr on MinGW protocolbuffers/protobuf#8049

The first issue pevents me from compiling using mingw.
The second issue seems similar to the one I'm experiencing using msvc/clang/clang-cl, albeit at a different callsite.

What did you expect to see?

cgo should work on windows

What did you see instead?

cgo doesn't work

Random thoughts

I've been ruminating on a few approaches and I'm wondering what you think.

1. Fix protobuf so that it builds with gcc (8049)
   a. can we just add mutable modifier to the mutex?
   b. is there a const modifier we can remove?
   c. I clearly didn't check :) hopefully it's a simple fix though.

2. define"target overrides" / support multiple compilers. What I'm thinking is a json/proto, e.g.:

{
  'overrides': {
    '//pkg:target' : {
      'cc_compiler': '...'   // or cc toolchain
    }
  }
}

For #2 , I'm very curious to hear your thoughts. I realize this is not really in the spirit of blaze. but I'm not really sure how I'm supposed to override the compiler used for "that" go package when "that" is generated code, i.e. from vendor (/vendor/...).

Personally, I don't like approach 2. Mixing objects from multiple compilers wouldprobably work but it introduces complexity...(and it just feels like a hack)

cc @jayconrod

$ /c/bin/bazelisk.exe --bazelrc ./tools/bazelrc.windows run //repro --sandbox_debug --verbose_failures  --compiler=clang-cl
Loading:
Loading: 0 packages loaded
INFO: Build options --compiler and --host_compiler have changed, discarding analysis cache.
Analyzing: target //repro:repro (0 packages loaded, 0 targets configured)
INFO: Analyzed target //repro:repro (0 packages loaded, 7029 targets configured).
INFO: Found 1 target...
[0 / 6] [Prepa] BazelWorkspaceStatusAction stable-status.txt
ERROR: C:/users/johnskopis/src/repro/vendor/github.com/mattn/go-sqlite3/BUILD.bazel:3:11: GoCompilePkg vendor/github.com/mattn/go-sqlite3/go-sqlite3.a failed: (Exit 1): builder.exe failed: error executing command
  cd C:/tmp/q56i5uwg/execroot/repro
  SET CC=C:/LLVM/bin/clang-cl.exe
    SET CGO_ENABLED=1
    SET GOARCH=amd64
    SET GOOS=windows
    SET GOPATH=
    SET GOROOT=external/go_sdk
    SET GOROOT_FINAL=GOROOT
    SET INCLUDE=C:\vs2019tools\VC\Tools\MSVC\14.28.29333\ATLMFC\include;C:\vs2019tools\VC\Tools\MSVC\14.28.29333\include;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um;C:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt;C:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared;C:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um;C:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt;C:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt;C:\LLVM\lib\clang\11.0.1\include
    SET LIB=C:\vs2019tools\VC\Tools\MSVC\14.28.29333\ATLMFC\lib\x64;C:\vs2019tools\VC\Tools\MSVC\14.28.29333\lib\x64;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64;C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64;C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64;C:\LLVM\lib\clang\11.0.1\lib\windows;C:\LLVM\lib\clang\11.0.1\lib\windows
    SET PATH=;C:/LLVM/bin;C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8 Tools\x64\;C:\Program Files (x86)\Windows Kits\10\bin\10.0.18362.0\x64;C:\Program Files (x86)\Windows Kits\10\bin\x64;C:\WINDOWS\system32;C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\vs2019tools\Common7\IDE\;C:\vs2019tools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin;C:\vs2019tools\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja;C:\vs2019tools\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer;C:\vs2019tools\Common7\IDE\CommonExtensions\Microsoft\TestWindow;C:\vs2019tools\Common7\IDE\VC\VCPackages;C:\vs2019tools\Common7\Tools\;C:\vs2019tools\MSBuild\Current\bin\Roslyn;C:\vs2019tools\VC\Tools\Llvm\x64\bin;C:\vs2019tools\VC\Tools\MSVC\14.28.29333\bin\HostX64\x64;C:\vs2019tools\\MSBuild\Current\Bin
    SET TEMP=C:\Users\johnskopis\AppData\Local\Temp
    SET TMP=C:\Users\johnskopis\AppData\Local\Temp
  bazel-out/host/bin/external/go_sdk/builder.exe compilepkg -sdk external/go_sdk -installsuffix windows_amd64 -src vendor/github.com/mattn/go-sqlite3/backup.go -src vendor/github.com/mattn/go-sqlite3/callback.go -src vendor/github.com/mattn/go-sqlite3/convert.go -src vendor/github.com/mattn/go-sqlite3/doc.go -src vendor/github.com/mattn/go-sqlite3/error.go -src vendor/github.com/mattn/go-sqlite3/sqlite3.go -src vendor/github.com/mattn/go-sqlite3/sqlite3_context.go -src vendor/github.com/mattn/go-sqlite3/sqlite3_func_crypt.go -src vendor/github.com/mattn/go-sqlite3/sqlite3_go18.go -src vendor/github.com/mattn/go-sqlite3/sqlite3_load_extension.go -src vendor/github.com/mattn/go-sqlite3/sqlite3_opt_preupdate.go -src vendor/github.com/mattn/go-sqlite3/sqlite3_opt_preupdate_omit.go -src vendor/github.com/mattn/go-sqlite3/sqlite3_opt_userauth_omit.go -src vendor/github.com/mattn/go-sqlite3/sqlite3_other.go -src vendor/github.com/mattn/go-sqlite3/sqlite3_solaris.go -src vendor/github.com/mattn/go-sqlite3/sqlite3_type.go -src vendor/github.com/mattn/go-sqlite3/sqlite3_usleep_windows.go -src vendor/github.com/mattn/go-sqlite3/sqlite3_windows.go -src vendor/github.com/mattn/go-sqlite3/static_mock.go -src vendor/github.com/mattn/go-sqlite3/sqlite3-binding.c -src vendor/github.com/mattn/go-sqlite3/sqlite3_opt_unlock_notify.c -src vendor/github.com/mattn/go-sqlite3/sqlite3-binding.h -src vendor/github.com/mattn/go-sqlite3/sqlite3ext.h -importpath github.com/mattn/go-sqlite3 -p github.com/johnskopis/repro/vendor/github.com/mattn/go-sqlite3 -package_list bazel-out/host/bin/external/go_sdk/packages.txt -o bazel-out/x64_windows-opt/bin/vendor/github.com/mattn/go-sqlite3/go-sqlite3.a -x bazel-out/x64_windows-opt/bin/vendor/github.com/mattn/go-sqlite3/go-sqlite3.x -gcflags  -asmflags  -cppflags -I vendor/github.com/mattn/go-sqlite3 -iquote . -cflags /nologo /DCOMPILER_MSVC /DNOMINMAX /D_WIN32_WINNT=0x0601 /D_CRT_SECURE_NO_DEPRECATE /D_CRT_SECURE_NO_WARNINGS /bigobj /Zm500 /EHsc /wd4351 /wd4291 /wd4250 /wd4996 /showIncludes /MD /O2 /Oy- /DNDEBUG /wd4117 -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" -Wno-builtin-macro-redefined /Gy /Gw -DHAVE_USLEEP=1 -DSQLITE_DEFAULT_WAL_SYNCHRONOUS=1 -DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_RTREE -DSQLITE_ENABLE_UPDATE_DELETE_LIMIT -DSQLITE_OMIT_DEPRECATED -DSQLITE_THREADSAFE=1 -DSQLITE_TRACE_SIZE_LIMIT=15 -Wno-deprecated-declarations -std=gnu99 -Ivendor/github.com/mattn/go-sqlite3 -fno-stack-check -fno-stack-protector -mno-stack-arg-probe -fPIC -cxxflags /nologo /DCOMPILER_MSVC /DNOMINMAX /D_WIN32_WINNT=0x0601 /D_CRT_SECURE_NO_DEPRECATE /D_CRT_SECURE_NO_WARNINGS /bigobj /Zm500 /EHsc /wd4351 /wd4291 /wd4250 /wd4996 /showIncludes /MD /O2 /Oy- /DNDEBUG /wd4117 -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" -Wno-builtin-macro-redefined /Gy /Gw -fPIC -objcflags /nologo /DCOMPILER_MSVC /DNOMINMAX /D_WIN32_WINNT=0x0601 /D_CRT_SECURE_NO_DEPRECATE /D_CRT_SECURE_NO_WARNINGS /bigobj /Zm500 /EHsc /wd4351 /wd4291 /wd4250 /wd4996 /showIncludes /MD /O2 /Oy- /DNDEBUG /wd4117 -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" -Wno-builtin-macro-redefined /Gy /Gw -DHAVE_USLEEP=1 -DSQLITE_DEFAULT_WAL_SYNCHRONOUS=1 -DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_RTREE -DSQLITE_ENABLE_UPDATE_DELETE_LIMIT -DSQLITE_OMIT_DEPRECATED -DSQLITE_THREADSAFE=1 -DSQLITE_TRACE_SIZE_LIMIT=15 -Wno-deprecated-declarations -std=gnu99 -Ivendor/github.com/mattn/go-sqlite3 -fno-stack-check -fno-stack-protector -mno-stack-arg-probe -fPIC -objcxxflags /nologo /DCOMPILER_MSVC /DNOMINMAX /D_WIN32_WINNT=0x0601 /D_CRT_SECURE_NO_DEPRECATE /D_CRT_SECURE_NO_WARNINGS /bigobj /Zm500 /EHsc /wd4351 /wd4291 /wd4250 /wd4996 /showIncludes /MD /O2 /Oy- /DNDEBUG /wd4117 -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" -Wno-builtin-macro-redefined /Gy /Gw -fPIC -ldflags /nologo /SUBSYSTEM:CONSOLE /MACHINE:X64 /DEFAULTLIB:clang_rt.builtins-x86_64.lib /DEFAULTLIB:msvcrt.lib /OPT:ICF /OPT:REF -lmingwex -lmingw32
Execution platform: @local_config_platform//:host
clang-cl: warning: unknown argument ignored in clang-cl: '-dM' [-Wunknown-argument]
clang-cl: warning: unknown argument ignored in clang-cl: '-iquote' [-Wunknown-argument]
clang-cl: warning: unknown argument ignored in clang-cl: '-iquote' [-Wunknown-argument]
clang-cl: warning: unknown argument ignored in clang-cl: '-iquote' [-Wunknown-argument]
clang-cl: warning: unknown argument ignored in clang-cl: '-std=gnu99' [-Wunknown-argument]
clang-cl: warning: unknown argument ignored in clang-cl: '-fno-stack-check' [-Wunknown-argument]
clang-cl: warning: unknown argument ignored in clang-cl: '-fno-stack-protector' [-Wunknown-argument]
clang-cl: warning: unknown argument ignored in clang-cl: '-mno-stack-arg-probe' [-Wunknown-argument]
clang-cl: warning: unknown argument ignored in clang-cl: '-fPIC' [-Wunknown-argument]
error: error reading '.'
1 error generated.
compilepkg: error running subcommand external\go_sdk\pkg\tool\windows_amd64\cgo.exe: exit status 2
Target //repro:repro failed to build
INFO: Elapsed time: 0.654s, Critical Path: 0.11s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully
ERROR: Build failed. Not running target
FAILED: Build did NOT complete successfully

[jayconrod]

This is a pretty broad issue. If you can narrow it to a specific error or file multiple issues for specific errors, that would make it more actionable.

• You said you're getting errors similar to cgo: use -I instead of -iquote for source header directories #2687. What are they? cgo: use -I instead of -iquote for source header directories #2687 should be fixed in v0.25.1.
• You mentioned you're running inside Docker. Do you get the same error outside of Docker? I believe Docker runs a small Linux VM on Windows, so it's a very different environment.
• I can't do much about PROTOBUF_MAYBE_CONSTEXPR should not be constexpr on MinGW protocolbuffers/protobuf#8049. It needs to be fixed there. I'd suggest downgrading to an older protobuf version (lower than 3.14 I think) that's not affected.
• Your error messages show clang-cl. Unfortunately, cgo doesn't support MSVC. This is a cgo limitation, not rules_go. cgo determines types of C/C++ definitions using error messages, and it doesn't understand messages from MSVC.

Using rules_go on Windows is the main thing I can recommend. Unfortunately, it's pretty difficult to get it working. Anything outside the narrow happy path tends to break.

[adam-azarchs]

The problem here is that when it is treating clang-cl as if it were clang, which it isn't. So when it sees -iquote . it ignores the -iquote (with a warning) and then tries to treat . as a source file.

This is a general problem of rules_go's cgo support not handling clang-cl as a compiler. clang-cl is not MSVC (though it depends on an MSVC installation for things like the C runtime library) - it's just a wrapper around clang that knows how to translate arguments from the form that MSVC's cl.exe expects them into the form that clang expects them.

As an alternative to fixing this on the rules_go side, bazel's autoconfigured c++ toolchain could maybe be taught how to use clang directly on windows, without the cl wrapper? Or rules_go could just prefix clang args with /clang: so they get passed through directly.