Clang-16.0.4-GCCcore-12.2.0 build failure in nvptx on CentOS 7

[verdurin]

Building Clang to test something else and saw this error (there's no GPU on this node):

[ 53%] Optimizing LLVM bitcode libomptarget-nvptx-sm_70.bc
opt: /dev/shm/Clang/16.0.4/GCCcore-12.2.0/llvm-project-16.0.4.src/llvm/lib/Support/StringMap.cpp:185: void llvm::StringMapImpl::RemoveKey(llvm::StringMapEntryBase*): Assertion `V
 == V2 && "Didn't find key?"' failed.
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: /dev/shm/Clang/16.0.4/GCCcore-12.2.0/llvm.obj.1/bin/opt -O3 -openmp-opt-disable -attributor-enable=module /dev/shm/Clang/16.0.4/GCCcore-12.2.0/llvm.obj
.1/runtimes/runtimes-bins/openmp/libomptarget/DeviceRTL/internalized_libomptarget-nvptx-sm_70.bc -o /dev/shm/Clang/16.0.4/GCCcore-12.2.0/llvm.obj.1/runtimes/runtimes-bins/openmp/
libomptarget/DeviceRTL/libomptarget-nvptx-sm_70.bc
 #0 0x0000000002087b5e llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/dev/shm/Clang/16.0.4/GCCcore-12.2.0/llvm.obj.1/bin/opt+0x2087b5e)
 #1 0x00000000020854cb SignalHandler(int) Signals.cpp:0:0
 #2 0x00007f97cf833630 __restore_rt sigaction.c:0:0
 #3 0x00007f97cd99a387 raise (/lib64/libc.so.6+0x36387)
 #4 0x00007f97cd99ba78 abort (/lib64/libc.so.6+0x37a78)
 #5 0x00007f97cd9931a6 __assert_fail_base (/lib64/libc.so.6+0x2f1a6)
 #6 0x00007f97cd993252 (/lib64/libc.so.6+0x2f252)
 #7 0x00000000020188f1 llvm::StringMapImpl::RemoveKey(llvm::StringMapEntryBase*) (/dev/shm/Clang/16.0.4/GCCcore-12.2.0/llvm.obj.1/bin/opt+0x20188f1)
 #8 0x0000000001755393 llvm::Instruction::eraseFromParent() (/dev/shm/Clang/16.0.4/GCCcore-12.2.0/llvm.obj.1/bin/opt+0x1755393)
 #9 0x00000000019df7ff runIPSCCP(llvm::Module&, llvm::DataLayout const&, llvm::AnalysisManager<llvm::Function>*, std::function<llvm::TargetLibraryInfo const& (llvm::Function&)>,
std::function<llvm::TargetTransformInfo& (llvm::Function&)>, std::function<llvm::AssumptionCache& (llvm::Function&)>, llvm::function_ref<llvm::AnalysisResultsForFn (llvm::Functio
n&)>, bool) SCCP.cpp:0:0
#10 0x00000000019e0e3d llvm::IPSCCPPass::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) (/dev/shm/Clang/16.0.4/GCCcore-12.2.0/llvm.obj.1/bin/opt+0x19e0e3d)
#11 0x00000000024203dd llvm::detail::PassModel<llvm::Module, llvm::IPSCCPPass, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisMan
ager<llvm::Module>&) (/dev/shm/Clang/16.0.4/GCCcore-12.2.0/llvm.obj.1/bin/opt+0x24203dd)
#12 0x00000000017e9b09 llvm::PassManager<llvm::Module, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) (/dev/shm/Clang/16.0.4/GCCco
re-12.2.0/llvm.obj.1/bin/opt+0x17e9b09)
#13 0x000000000080aa2d llvm::runPassPipeline(llvm::StringRef, llvm::Module&, llvm::TargetMachine*, llvm::TargetLibraryInfoImpl*, llvm::ToolOutputFile*, llvm::ToolOutputFile*, llv
m::ToolOutputFile*, llvm::StringRef, llvm::ArrayRef<llvm::PassPlugin>, llvm::opt_tool::OutputKind, llvm::opt_tool::VerifierKind, bool, bool, bool, bool, bool, bool) (/dev/shm/Cla
ng/16.0.4/GCCcore-12.2.0/llvm.obj.1/bin/opt+0x80aa2d)
#14 0x0000000000755c5b main (/dev/shm/Clang/16.0.4/GCCcore-12.2.0/llvm.obj.1/bin/opt+0x755c5b)
#15 0x00007f97cd986555 __libc_start_main (/lib64/libc.so.6+0x22555)
#16 0x00000000007fc716 _start (/dev/shm/Clang/16.0.4/GCCcore-12.2.0/llvm.obj.1/bin/opt+0x7fc716)
/bin/sh: line 1: 23487 Aborted                 /dev/shm/Clang/16.0.4/GCCcore-12.2.0/llvm.obj.1/bin/opt -O3 -openmp-opt-disable -attributor-enable=module /dev/shm/Clang/16.0.4/GCC
core-12.2.0/llvm.obj.1/runtimes/runtimes-bins/openmp/libomptarget/DeviceRTL/internalized_libomptarget-nvptx-sm_70.bc -o /dev/shm/Clang/16.0.4/GCCcore-12.2.0/llvm.obj.1/runtimes/r
untimes-bins/openmp/libomptarget/DeviceRTL/libomptarget-nvptx-sm_70.bc
make[5]: *** [openmp/libomptarget/DeviceRTL/libomptarget-nvptx-sm_70.bc] Error 134
make[5]: Leaving directory `/dev/shm/Clang/16.0.4/GCCcore-12.2.0/llvm.obj.1/runtimes/runtimes-bins'
make[4]: *** [openmp/libomptarget/DeviceRTL/CMakeFiles/omptarget-nvptx-sm_70-bc.dir/all] Error 2
make[4]: Leaving directory `/dev/shm/Clang/16.0.4/GCCcore-12.2.0/llvm.obj.1/runtimes/runtimes-bins'
make[3]: *** [all] Error 2
make[3]: Leaving directory `/dev/shm/Clang/16.0.4/GCCcore-12.2.0/llvm.obj.1/runtimes/runtimes-bins'
make[2]: *** [runtimes/runtimes-stamps/runtimes-build] Error 2
make[2]: Leaving directory `/dev/shm/Clang/16.0.4/GCCcore-12.2.0/llvm.obj.1'
make[1]: *** [runtimes/CMakeFiles/runtimes.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

[Micket]

I'd double check that it didn't just run out of memory or something. I doubt this needs cuda drivers, it's just optimizing it's own llvm bitcode (.bc file).

[migueldiascosta]

fwiw, I've used configopts = '-DLIBOMPTARGET_BUILD_DEVICERTL_BCLIB=OFF ' to get around that...

[verdurin]

Don't think it would be memory.
Trying again now on a CentOS 8 node in case that's relevant, and will try what Miguel suggested too.

[verdurin]

Confirmed that it worked on CentOS 7 with Miguel's fix, and worked without that fix on CentOS 8 Stream.

[verdurin]

This also appears to work for Clang-16.0.6.