represent c++ abi compatibility by adding "compiler variants" to the spec model

[cosmicexplorer]

This discussion proposes the introduction of "compiler variants" as an extension of spec variants. This is motivated by the desire to ensure spack doesn't try to build C++ packages with different values of the glibcxx abi together in the same environment. The necessary changes are detailed in the "work scope" section.

Problem

In response to a separate discussion over Slack, @dpzmick wrote:

I'm not sure that "apply this constraint to my whole environment" as a one time thing at install time is entirely sufficient. If the anon constraint is adding something like cppflags=-D_GLIBCXX_USE_CXX11_ABI=0 (changing the C++ abi used for strings and other important things), then the same constraint must continue to be used on any future spack install as well, otherwise any newly installed/concretized packages will have abi compatibility issues with ones already in the env.

When I worked in packaging tensorflow's native libraries at Twitter, I recall vividly how google's python packages were officially documented to have -D_GLIBCXX_USE_CXX11_ABI=0, but in practice, specific releases would have =1, which caused a compile error: see my failed attempt to get them to fix this at tensorflow/tensorflow#27078. This kind of compatibility issue seems like exactly what Spack is supposed to solve.

Existing Workarounds

Using cppflags

Managing _GLIBCXX_USE_CXX11_ABI via cppflags does not ensure compatibility, since spack does not consider it a conflict if one package is built with cppflags=-D_GLIBCXX_USE_CXX11_ABI=0 and its dependency is built with cppflags=-D_GLIBCXX_USE_CXX11_ABI=1:

# cosmicexplorer@terrestrial-gamma-ray-flash: ~/tools/s1 17:21:55
; spack spec 'raja cppflags=-D_GLIBCXX_USE_CXX11_ABI=0 ^ camp cppflags=-D_GLIBCXX_USE_CXX11_ABI=1'
Input spec
--------------------------------
raja cppflags="-D_GLIBCXX_USE_CXX11_ABI=0"
    ^camp cppflags="-D_GLIBCXX_USE_CXX11_ABI=1"

Concretized
--------------------------------
raja@2022.03.0%gcc@12.1.1 cppflags="-D_GLIBCXX_USE_CXX11_ABI=0" ~cuda+examples+exercises~ipo+openmp~rocm+shared~tests build_system=cmake build_type=RelWithDebInfo arch=linux-alpine3-zen3
    ^blt@0.5.2%gcc@12.1.1 build_system=generic arch=linux-alpine3-zen3
    ^camp@2022.03.2%gcc@12.1.1 cppflags="-D_GLIBCXX_USE_CXX11_ABI=1" ~cuda~ipo+openmp~rocm~tests build_system=cmake build_type=RelWithDebInfo arch=linux-alpine3-zen3
    ^cmake@3.23.1%gcc@12.1.1~doc+ncurses+ownlibs~qt build_type=Release arch=linux-alpine3-zen3

Package Recipes Hardcoding Flags

I was glad to see that the string CXX11_ABI was only found a few times in spack package definitions:

# cosmicexplorer@terrestrial-gamma-ray-flash: ~/tools/s1 16:53:50
; rg 'CXX11_ABI'
var/spack/repos/builtin/packages/julia/fix-gfortran.patch
20: BB_TRIPLET_LIBGFORTRAN_CXXABI := $(shell $(call invoke_python,$(JULIAHOME)/contrib/normalize_triplet.py) $(or $(XC_HOST),$(XC_HOST),$(BUILD_MACHINE)) "$(FC_OR_CC_VERSION)" "$(or $(shell echo '\#include <string>' | $(CXX) $(CXXFLAGS) -x c++ -dM -E - | grep _GLIBCXX_USE_CXX11_ABI | awk '{ print $$3 }' ),1)")

var/spack/repos/builtin/packages/xios/package.py
135:%BASE_CFLAGS    -ansi -w -D_GLIBCXX_USE_CXX11_ABI=0 \

var/spack/repos/builtin/packages/serialbox/package.py
155:                cmake_flags.append("-D_GLIBCXX_USE_CXX11_ABI=0")

However, it's clearly something multiple packages care about, and in the case of julia, it appears the Make.inc file in the julia codebase has a special method to obtain _GLIBCXX_USE_CXX11_ABI from the list of flags (so this value clearly affects how the package is built).

Reported Build Failures

When searching "C++ ABI" or "cxx abi" in this repo, I only found #28541, which I couldn't immediately understand, but also appears to be about gfortran and C++ ABI compatibility. I'm not sure if this is the same issue, but it seems related.

Proposed Solution

Represent C++ ABI configuration as a "compiler variant", and read that config to set appropriate compile flags in the compiler wrapper. This might look something like:

> spack install 'raja%gcc+cxx11_abi'
# Install raja, setting `-D_GLIBCXX_USE_CXX11_ABI=1` for all compiles.

If we tried to do the same thing we did earlier and set different values for the cxx abi in a dependency, spack should error:

> spack spec 'raja%gcc+cxx11_abi ^ camp%gcc~cxx11_abi'
# Should error out with e.g. "incompatible compiler variant values".

Work Scope

This change would require modifying:

1. The spack.spec.Spec object, to retain a separate VariantMap for "compiler variants".
2. concretize.lp, to raise a constraint error when compiler variants do not match.
3. The spec parser, to understand a + or ~ variant after a % as a "compiler variant".
   • Note that we can avoid trying to process any key-value variants for now, since the cxx abi is a binary flag.
4. The spack.compiler.Compiler object, to accept parameters from "compiler variant" values.
5. The cc wrapper, to inject -D_GLIBCXX_USE_CXX11_ABI={0,1} into all gcc compiles depending on the value of the cxx11_abi compiler variant.

Alternatives

Special Variant Name (like cppflags)

We could instead make cxx11_abi into a "special variant name" (like cppflags) to avoid modifying the spec parser. That could look like:

# `cxx11_abi` is *parsed* as a normal variant,
# but when constructing a Spec instance,
# it is processed separately from other variants.
> spack spec 'raja cxx11_abi=1 %gcc ^ camp cxx11_abi=0'
# Should error out with e.g. "incompatible compiler variant values".

Since _GLIBCXX_USE_CXX11_ABI is specific to g++, I think it would be confusing to allow this "compiler variant" to be set without attaching it to a compiler spec. I think it makes more sense to attach "compiler variants" to compiler specs, but if we want to be cautious about changing the spec syntax, we could do this instead.

Separate Compiler Name

We could avoid having to modify the spack.compiler.Compiler object to allow parameters if we instead create a copy of the gcc compiler object named gcc_cxx11 or something. This would probably require a lot of annoying changes to package recipes to explicitly allow the gcc_cxx11 compiler, so we probably shouldn't do this.

[cosmicexplorer]

Ah, @giordano notes in #28541 (comment) that this is probably best done as part of the compiler-as-dependency work, which makes sense. I would prefer to use that solution instead of specially handling this case. I'll try to find where this fits in to that larger work.

[giordano]

The C++ string ABI is one level deep inside the C++ runtime library, since it's an incompatibility internal to libstdc++.

[dpzmick]

Hey, thanks for capturing this out of slack! ABI compatibility is a pretty big deal and def something I'd appreciate having good tools to control. A feature like this might make spack super attractive to us at $work :)

This issue goes a bit deeper than just the CXX11_ABI flag though. GCC has a page noting all the minor tweaks they've made to abi: https://gcc.gnu.org/onlinedocs/gcc/C_002b_002b-Dialect-Options.html#C_002b_002b-Dialect-Options . These are mostly bug fixes, but it's worth considering what the implications are on packaging tools. i.e. consider this one:

Version 17, which first appeared in G++ 12, fixes layout of classes that inherit from aggregate classes with default member initializers in C++14 and up.

which I can imagine would change behavior of certain UB-laden code.

[brevzin]

A different set of flags that are in the same vein would be the sanitizers. For instance, I might want to compile my entire set of dependencies with -fsanitize=address. That's not an ABI flag that I need to even have a sane binary, but it is the kind of thing that would be useful to help track down potential issues.

[alalazo]

This will be handled in the "Compiler as dependencies" project, which is still under development. We have a couple of issues linked to this epic:

• Software model for "compiler as dependencies" #31357

That show which model we are currently trying to achieve with compilers and runtime libraries.

[dpzmick]

Hey @cosmicexplorer / @alalazo. Checking in on ISC I noticed that the spack compiler-as-a-dep has been released! I see notes here: https://github.com/spack/spack/releases/tag/v0.22.0

Does this mean the way to do something like pin the ABI, or pin a certain sanitizer, is to create something like a compiler wrapper that ensures the flag is set, then build everything in a package tree against that special compiler? I don't see a good place in compilers.yaml or packages.yaml to stick these flags

[alalazo]

@dpzmick Sorry to say the compiler as deps is just half deployed. We have modeling for the runtimes, but for what you ask for we still need changes that are slated for v0.23 in november.