Darwin M1 Support

[aherrmann-da]

A new attempt at #11496

This adds support to build and test the daml repository on MacOS on M1.
This was tested on an M1 machine on MacOS 12.2 Monterey.

• Updates or patches rule set dependencies to handle the darwin_arm64 platform and assign appropriate cpu constraints to toolchains.
• Configure GHC to use the LLVM backend on that platform. GHC 9.0's native backend does not yet support M1.
• Make sure to consistently use the linker provided by the Bazel CC toolchain.
• Update NodeJS for M1 compatibility
• Make sure to re-sign binaries using codesign after patching them with install_name_tool. Otherwise the signature test will fail and MacOS will refuse to execute the binary. The symptom is often that the binary fails to load and terminates immediately with exit-code -9 without any further indication. This also happens when other processes attempt to load the binary, e.g. lldb.
• Configure the CC toolchain on MacOS for M1 compatibility.
• Disable create-daml-app tests on MacOS M1, see Update puppeteer for M1 support #13250 for further details.

Note, the NodeJS update seems to cause two issues in the compatibility test:
1.) The install step in the create-daml-app tests fails unless we add node to PATH. Previously it was sufficient to set --scripts-prepend-node-path. See this commit for details.
2.) The install step in create-daml-app tests fails for SDK versions 1.8.1 and 1.9.0 due to conflicting versions constraints on babel-jest. See this commit for details.

Pull Request Checklist

• Read and understand the contribution guidelines
• Include appropriate tests
• Set a descriptive title and thorough description
• Add a reference to the issue this PR will solve, if appropriate
• Include changelog additions in one or more commit message bodies between the CHANGELOG_BEGIN and CHANGELOG_END tags
• Normal production system change, include purpose of change in description
• If you mean to change the status of a component, please make sure you keep the Component Status page up to date.

NOTE: CI is not automatically run on non-members pull-requests for security
reasons. The reviewer will have to comment with /AzurePipelines run to
trigger the build.

[aherrmann-da]

How to proceed with this?
Can we patch or fix the dependency issue, or are these releases incompatible with the more recent NodeJS version?

[aherrmann-da]

I've kept the older NodeJS version for the compat tests specifically for now. This seems to resolve the issue (at least locally, waiting for CI). This gives us the freedom to look into the NodeJS update for the compat tests independently.

[dasormeter]

hiya @aherrmann-da what is it that is missing for build on an M1 machine onmain presently? i was able to run a bazel build //... successfully on an M1 laptop -- i have not yet tried a full bazel test //...

also are we planning to soon have both an arm64 and amd64 version of the linux docker images for the daml-sdk and friends published to dockerhub and/or our private digitalasset artifactory?

[aherrmann-da]

hiya @aherrmann-da what is it that is missing for build on an M1 machine onmain presently? i was able to run a bazel build //... successfully on an M1 laptop -- i have not yet tried a full bazel test //...

@dasormeter This was tested on an M1 machine on MacOS 12.2 Monterey. By that I mean that bazel build //... and bazel test //... pass. So, in that regard, there is nothing that's missing. That said, there is the one issue with create-daml-app, see #13250 for details. This PR works around that by disabling this test on M1.

also are we planning to soon have both an arm64 and amd64 version of the linux docker images for the daml-sdk and friends published to dockerhub and/or our private digitalasset artifactory?

I don't know if there are any plans around that.

[garyverhaegen-da]

Thank you so much for looking into this!

@dasormeter

also are we planning to soon have both an arm64 and amd64 version of the linux docker images for the daml-sdk and friends published to dockerhub and/or our private digitalasset artifactory?

No.

hiya @aherrmann-da what is it that is missing for build on an M1 machine onmain presently? i was able to run a bazel build //... successfully on an M1 laptop -- i have not yet tried a full bazel test //...

@dasormeter This was tested on an M1 machine on MacOS 12.2 Monterey. By that I mean that bazel build //... and bazel test //... pass. So, in that regard, there is nothing that's missing.

I think the question was specifically about main, i.e. why is this PR needed? I have not looked into it myself yet but both @dasormeter and @nycnewman have claimed successful builds on M1 with current main.

My (tenuous) understanding is that main currently builds x86 binaries even when run on M1 machines, and this PR makes us able to generate "native" arm binaries. But happy to be enlightened if that's not the case (or get confirmaton if it is).

[aherrmann-da]

I think the question was specifically about main, i.e. why is this PR needed? I have not looked into it myself yet but both @dasormeter and @nycnewman have claimed successful builds on M1 with current main.

My (tenuous) understanding is that main currently builds x86 binaries even when run on M1 machines, and this PR makes us able to generate "native" arm binaries. But happy to be enlightened if that's not the case (or get confirmaton if it is).

Thanks for clarifying @garyverhaegen-da , indeed, I misunderstood the question.

You're correct, main currently builds in x86_64 mode through Rosetta and this PR enables native ARM64 builds.

[sofiafaro-da]

Great work, thank you!

[sofiafaro-da]

What does this do? Is this related to adding M1 support?

[aherrmann-da]

It disables building haddocks for that package. We already disable it for others. The haddocks build failed on M1, details are in the commit message.

[sofiafaro-da]

Nice, thanks for the explanation.

[garyverhaegen-da]

You're correct, main currently builds in x86_64 mode through Rosetta and this PR enables native ARM64 builds.

So if we want to publish M1 binaries, we'll need to build on M1? Is there any way we could get universal binaries built from both platforms? It's currently a bit tricky to add M1 nodes to CI.

[dasormeter]

Hi @aherrmann when i run bazel build //... on this branch on my M1 machine i get the following error

ERROR: /Users/brianhealey/g/daml/release/BUILD.bazel:9:18: HaskellLinkBinary release/release failed: (Exit 1): ghc_wrapper failed: error executing command
  (cd /private/var/tmp/_bazel_brianhealey/191c77e1bcea6e1b949558cf8ebb3be9/sandbox/processwrapper-sandbox/3861/execroot/com_github_digital_asset_daml && \
  exec env - \
    CC_WRAPPER_CC_PATH=external/local_config_cc/cc_wrapper.sh \
    CC_WRAPPER_CPU=darwin_arm64 \
    CC_WRAPPER_PLATFORM=darwin \
    LANG=C.UTF-8 \
    RULES_HASKELL_DOCDIR_PATH=/nix/store/kd7bjbp0wgz0qjhx6m2fcasp66n0wga2-ghc-native-bignum-9.0.2-doc/share/doc/ghc-native-bignum/html/libraries/base-4.15.1.0 \
    RULES_HASKELL_GHC_PATH=external/rules_haskell_ghc_nixpkgs/bin/ghc \
    RULES_HASKELL_GHC_PKG_PATH=external/rules_haskell_ghc_nixpkgs/bin/ghc-pkg \
    RULES_HASKELL_LIBDIR_PATH=/nix/store/z8577sg7xny6fxhndrrrkpvzhn53w9vc-ghc-native-bignum-9.0.2/lib/ghc-9.0.2 \
  bazel-out/darwin_arm64-opt/bin/external/rules_haskell/haskell/ghc_wrapper bazel-out/darwin_arm64-opt/bin/release/compile_flags_release__HaskellLinkBinary bazel-out/darwin_arm64-opt/bin/release/extra_args_release__HaskellLinkBinary)
Execution platform: @io_tweag_rules_nixpkgs//nixpkgs/platforms:host

Use --sandbox_debug to see verbose messages from the sandbox
ld: warning: directory not found for option '-L/nix/store/xqbgx1y5xh2k13c6nizlprkp0j8mcw6w-clang-wrapper-12.0.1/lib'
ld: warning: directory not found for option '-L/nix/store/xqbgx1y5xh2k13c6nizlprkp0j8mcw6w-clang-wrapper-12.0.1/lib'
/nix/store/cncwqs75v3syxhna1452bfd8wvwrzxxb-clang-wrapper-wrapper-12.0.1/bin/ld: line 256: 51168 Segmentation fault: 11  /nix/store/d853z4g5dp326kc7lqd9n3kh29wic1jf-cctools-binutils-darwin-949.0.1/bin/ld ${extraBefore+"${extraBefore[@]}"} ${params+"${params[@]}"} ${extraAfter+"${extraAfter[@]}"}
clang-12: error: linker command failed with exit code 139 (use -v to see invocation)
`cc_wrapper-python' failed in phase `Linker'. (Exit code: 139)
INFO: Elapsed time: 2133.417s, Critical Path: 1345.88s
INFO: 7331 processes: 3475 internal, 2 local, 3536 processwrapper-sandbox, 318 worker.
FAILED: Build did NOT complete successfully

[garyverhaegen-da]

Blocking accidental merges until we figure out @dasormeter's issue.

[garyverhaegen-da]

@dasormeter There are Nix changes in this PR; can you confirm you get that after having refreshed your Nix configuration? In principle cding out of then back into the directory should be good enough if you're using dev-env, but if the problem persists could you try nuking Nix and see if that helps?

[dasormeter]

@dasormeter There are Nix changes in this PR; can you confirm you get that after having refreshed your Nix configuration? In principle cding out of then back into the directory should be good enough if you're using dev-env, but if the problem persists could you try nuking Nix and see if that helps?

I cd'ed out and back in and kicked off a bazel build //... again
Is there a command to run to verify the expected nix changes have kicked in?

[dasormeter]

Oddly enough the bash, python3.9 and clang that run as part of bazel do not report what kind (intel or mac) they are running -- i guess these are custom jobs we get from nix?

[dasormeter]

I still got the same error after cd'ing out and back in

By nuking Nix do you mean the usual daml repo cleanup commands or something more drastic?

[garyverhaegen-da]

I still got the same error after cd'ing out and back in

By nuking Nix do you mean the usual daml repo cleanup commands or something more drastic?

I'm not using the word "nuke" lightly; what I do is:

rm -rf $HOME/{.nix-channels,.nix-defexpr,.nix-profile,.config/nixpkgs,.cache/nix} && chmod -R +w /nix/store; chmod -R +w /var/tmp/_bazel_garyverhaegen;  rm -rf /nix/store /nix/var /var/tmp/_bazel_garyverhaegen /Users/garyverhaegen/Desktop/code/daml/.bazel-cache; rm -rf /Users/garyverhaegen/Desktop/code/daml/dev-env/jdk

Then reinstall Nix and rebuild everything.

Obviously those commands are a bit specific to my own machine, but you get the idea. I've been told it's unnecessarily drastic and we should never need to do this, but anecdotally it has seemed to solve issues for me a few times. It's mostly psychological though: if something still fails after that I'm pretty confident it's not an issue with caching on my machine.

Do expect about 20 minutes for that + reinstall Nix, so plan for it accordingly if it's your main work machine.

[dasormeter]

Did you also blow out /var/tmp/_bazel__nixbld1/?

❯ ls var/tmp/_bazel
_bazel__nixbld1/ _bazel_brianhealey/

[garyverhaegen-da]

I don't have that. I assume you've done a multi-user install of Nix and nixbld1 is the username of your Nix builder?

[dasormeter]

I don't have that. I assume you've done a multi-user install of Nix and nixbld1 is the username of your Nix builder?

Yeah i just followed default options for nix install

[dasormeter]

I still got the same error after cd'ing out and back in
By nuking Nix do you mean the usual daml repo cleanup commands or something more drastic?

I'm not using the word "nuke" lightly; what I do is:

rm -rf $HOME/{.nix-channels,.nix-defexpr,.nix-profile,.config/nixpkgs,.cache/nix} && chmod -R +w /nix/store; chmod -R +w /var/tmp/_bazel_garyverhaegen;  rm -rf /nix/store /nix/var /var/tmp/_bazel_garyverhaegen /Users/garyverhaegen/Desktop/code/daml/.bazel-cache; rm -rf /Users/garyverhaegen/Desktop/code/daml/dev-env/jdk

Then reinstall Nix and rebuild everything.

Obviously those commands are a bit specific to my own machine, but you get the idea. I've been told it's unnecessarily drastic and we should never need to do this, but anecdotally it has seemed to solve issues for me a few times. It's mostly psychological though: if something still fails after that I'm pretty confident it's not an issue with caching on my machine.

Do expect about 20 minutes for that + reinstall Nix, so plan for it accordingly if it's your main work machine.

Seems to be rebuilding the world from nix derivations when i then run bazel build //... after resintalling nix : is this expected because nix store was blown out?

[garyverhaegen-da]

If you've blown your Nix store (as my commands above do), it is expected that Nix has to do some work to get its store back in shape, but most of it should be downloading pre-built binaries, not building things from scratch.

If you want to separate Bazel work from Nix work, you can do just the Nix work by running:

nix-build --no-out-link nix -A tools -A cached

from the root of the repo.

The caching story may be different on M1 so it's possible it's building more there. When I run it (on my Intel mac) it's basically getting everything from the cache.

[dasormeter]

@aherrmann-da @garyverhaegen-da unfortunately a full nix and bazel nuke and reinstall did not make a difference -- still failing at this same place on an M1 machine when i do bazel build //...

ERROR: /Users/brianhealey/g/daml/daml-assistant/integration-tests/BUILD.bazel:106:16: HaskellLinkBinary daml-assistant/integration-tests/integration-tests failed: (Exit 1): ghc_wrapper failed: error executing command
  (cd /private/var/tmp/_bazel_brianhealey/191c77e1bcea6e1b949558cf8ebb3be9/sandbox/processwrapper-sandbox/5180/execroot/com_github_digital_asset_daml && \
  exec env - \
    CC_WRAPPER_CC_PATH=external/local_config_cc/cc_wrapper.sh \
    CC_WRAPPER_CPU=darwin_arm64 \
    CC_WRAPPER_PLATFORM=darwin \
    LANG=C.UTF-8 \
    RULES_HASKELL_DOCDIR_PATH=/nix/store/kd7bjbp0wgz0qjhx6m2fcasp66n0wga2-ghc-native-bignum-9.0.2-doc/share/doc/ghc-native-bignum/html/libraries/base-4.15.1.0 \
    RULES_HASKELL_GHC_PATH=external/rules_haskell_ghc_nixpkgs/bin/ghc \
    RULES_HASKELL_GHC_PKG_PATH=external/rules_haskell_ghc_nixpkgs/bin/ghc-pkg \
    RULES_HASKELL_LIBDIR_PATH=/nix/store/z8577sg7xny6fxhndrrrkpvzhn53w9vc-ghc-native-bignum-9.0.2/lib/ghc-9.0.2 \
  bazel-out/darwin_arm64-opt/bin/external/rules_haskell/haskell/ghc_wrapper bazel-out/darwin_arm64-opt/bin/daml-assistant/integration-tests/compile_flags_integration-tests__HaskellLinkBinary bazel-out/darwin_arm64-opt/bin/daml-assistant/integration-tests/extra_args_integration-tests__HaskellLinkBinary)
Execution platform: @io_tweag_rules_nixpkgs//nixpkgs/platforms:host

Use --sandbox_debug to see verbose messages from the sandbox
ld: warning: directory not found for option '-L/nix/store/xqbgx1y5xh2k13c6nizlprkp0j8mcw6w-clang-wrapper-12.0.1/lib'
ld: warning: directory not found for option '-L/nix/store/xqbgx1y5xh2k13c6nizlprkp0j8mcw6w-clang-wrapper-12.0.1/lib'
/nix/store/cncwqs75v3syxhna1452bfd8wvwrzxxb-clang-wrapper-wrapper-12.0.1/bin/ld: line 256: 93072 Segmentation fault: 11  /nix/store/d853z4g5dp326kc7lqd9n3kh29wic1jf-cctools-binutils-darwin-949.0.1/bin/ld ${extraBefore+"${extraBefore[@]}"} ${params+"${params[@]}"} ${extraAfter+"${extraAfter[@]}"}
clang-12: error: linker command failed with exit code 139 (use -v to see invocation)
`cc_wrapper-python' failed in phase `Linker'. (Exit code: 139)
INFO: Elapsed time: 2875.536s, Critical Path: 2044.12s
INFO: 11819 processes: 6641 internal, 4 local, 4684 processwrapper-sandbox, 490 worker.
FAILED: Build did NOT complete successfully

[dasormeter]

@aherrmann-da @garyverhaegen-da unfortunately a full nix and bazel nuke and reinstall did not make a difference -- still failing at this same place on an M1 machine when i do bazel build //...

ERROR: /Users/brianhealey/g/daml/daml-assistant/integration-tests/BUILD.bazel:106:16: HaskellLinkBinary daml-assistant/integration-tests/integration-tests failed: (Exit 1): ghc_wrapper failed: error executing command
  (cd /private/var/tmp/_bazel_brianhealey/191c77e1bcea6e1b949558cf8ebb3be9/sandbox/processwrapper-sandbox/5180/execroot/com_github_digital_asset_daml && \
  exec env - \
    CC_WRAPPER_CC_PATH=external/local_config_cc/cc_wrapper.sh \
    CC_WRAPPER_CPU=darwin_arm64 \
    CC_WRAPPER_PLATFORM=darwin \
    LANG=C.UTF-8 \
    RULES_HASKELL_DOCDIR_PATH=/nix/store/kd7bjbp0wgz0qjhx6m2fcasp66n0wga2-ghc-native-bignum-9.0.2-doc/share/doc/ghc-native-bignum/html/libraries/base-4.15.1.0 \
    RULES_HASKELL_GHC_PATH=external/rules_haskell_ghc_nixpkgs/bin/ghc \
    RULES_HASKELL_GHC_PKG_PATH=external/rules_haskell_ghc_nixpkgs/bin/ghc-pkg \
    RULES_HASKELL_LIBDIR_PATH=/nix/store/z8577sg7xny6fxhndrrrkpvzhn53w9vc-ghc-native-bignum-9.0.2/lib/ghc-9.0.2 \
  bazel-out/darwin_arm64-opt/bin/external/rules_haskell/haskell/ghc_wrapper bazel-out/darwin_arm64-opt/bin/daml-assistant/integration-tests/compile_flags_integration-tests__HaskellLinkBinary bazel-out/darwin_arm64-opt/bin/daml-assistant/integration-tests/extra_args_integration-tests__HaskellLinkBinary)
Execution platform: @io_tweag_rules_nixpkgs//nixpkgs/platforms:host

Use --sandbox_debug to see verbose messages from the sandbox
ld: warning: directory not found for option '-L/nix/store/xqbgx1y5xh2k13c6nizlprkp0j8mcw6w-clang-wrapper-12.0.1/lib'
ld: warning: directory not found for option '-L/nix/store/xqbgx1y5xh2k13c6nizlprkp0j8mcw6w-clang-wrapper-12.0.1/lib'
/nix/store/cncwqs75v3syxhna1452bfd8wvwrzxxb-clang-wrapper-wrapper-12.0.1/bin/ld: line 256: 93072 Segmentation fault: 11  /nix/store/d853z4g5dp326kc7lqd9n3kh29wic1jf-cctools-binutils-darwin-949.0.1/bin/ld ${extraBefore+"${extraBefore[@]}"} ${params+"${params[@]}"} ${extraAfter+"${extraAfter[@]}"}
clang-12: error: linker command failed with exit code 139 (use -v to see invocation)
`cc_wrapper-python' failed in phase `Linker'. (Exit code: 139)
INFO: Elapsed time: 2875.536s, Critical Path: 2044.12s
INFO: 11819 processes: 6641 internal, 4 local, 4684 processwrapper-sandbox, 490 worker.
FAILED: Build did NOT complete successfully

I get many warnings about the clang-wrapper-12.0.1/bin directory not being found in many of the targes

INFO: From HaskellLinkBinary compiler/damlc/tests/damlc-lint:
ld: warning: directory not found for option '-L/nix/store/xqbgx1y5xh2k13c6nizlprkp0j8mcw6w-clang-wrapper-12.0.1/lib'
ld: warning: directory not found for option '-L/nix/store/xqbgx1y5xh2k13c6nizlprkp0j8mcw6w-clang-wrapper-12.0.1/lib'
INFO: From HaskellLinkBinary daml-assistant/daml-helper/test-daml-ledger:
ld: warning: directory not found for option '-L/nix/store/xqbgx1y5xh2k13c6nizlprkp0j8mcw6w-clang-wrapper-12.0.1/lib'
ld: warning: directory not found for option '-L/nix/store/xqbgx1y5xh2k13c6nizlprkp0j8mcw6w-clang-wrapper-12.0.1/lib'
INFO: From HaskellLinkBinary compiler/damlc/tests/damlc-cliparser:
ld: warning: directory not found for option '-L/nix/store/xqbgx1y5xh2k13c6nizlprkp0j8mcw6w-clang-wrapper-12.0.1/lib'
ld: warning: directory not found for option '-L/nix/store/xqbgx1y5xh2k13c6nizlprkp0j8mcw6w-clang-wrapper-12.0.1/lib'
INFO: From HaskellLinkBinary compiler/damlc/tests/damlc-test:
ld: warning: directory not found for option '-L/nix/store/xqbgx1y5xh2k13c6nizlprkp0j8mcw6w-clang-wrapper-12.0.1/lib'
ld: warning: directory not found for option '-L/nix/store/xqbgx1y5xh2k13c6nizlprkp0j8mcw6w-clang-wrapper-12.0.1/lib'
INFO: From HaskellLinkBinary compiler/damlc/tests/daml-doc:
ld: warning: directory not found for option '-L/nix/store/xqbgx1y5xh2k13c6nizlprkp0j8mcw6w-clang-wrapper-12.0.1/lib'
ld: warning: directory not found for option '-L/nix/store/xqbgx1y5xh2k13c6nizlprkp0j8mcw6w-clang-wrapper-12.0.1/lib'
INFO: From HaskellLinkBinary compiler/damlc/damlc-bootstrap:
ld: warning: directory not found for option '-L/nix/store/xqbgx1y5xh2k13c6nizlprkp0j8mcw6w-clang-wrapper-12.0.1/lib'
ld: warning: directory not found for option '-L/nix/store/xqbgx1y5xh2k13c6nizlprkp0j8mcw6w-clang-wrapper-12.0.1/lib'

[dasormeter]

On third rebuild this looks to have worked

luaotfload | db : Font names database not found, generating new one.
luaotfload | db : This can take several minutes; please be patient.This is LuaHBTeX, Version 1.13.0 (Web2C 2021/NixOS.org)
 system commands enabled.
INFO: Elapsed time: 318.041s, Critical Path: 301.82s
INFO: 770 processes: 6 internal, 431 processwrapper-sandbox, 333 worker.
INFO: Build completed successfully, 770 total actions

[dasormeter]

Build successfully on M1 laptop
A fully bazel test //... passed as well (took some hrs though :))

[skisel-da]

Build successfully on my M1 laptop as well

[aherrmann-da]

I'm not entirely sure how I feel about @dasormeter's issue just disappearing after three attempts, but I'll leave the decision up to @aherrmann.

That's odd, I tried to reproduce with a clean build without cache on an M1 test machine, but the build went through without error. So, I'm not able to reproduce the issue.

The missing paths warnings shouldn't be the culprit, these are additional paths configured in the Nix clang wrapper. The critical error seems to be the segfault:

/nix/store/cncwqs75v3syxhna1452bfd8wvwrzxxb-clang-wrapper-wrapper-12.0.1/bin/ld: line 256: 93072 Segmentation fault: 11 /nix/store/d853z4g5dp326kc7lqd9n3kh29wic1jf-cctools-binutils-darwin-949.0.1/bin/ld ${extraBefore+"${extraBefore[@]}"} ${params+"${params[@]}"} ${extraAfter+"${extraAfter[@]}"}

I can't really say what's causing this. It doesn't seem worth blocking the PR on it though.
But, if the issue occurs again please do raise it. It would be good to debug it in that case.