Apple silicon builds

[AshCripps]

refs: nodejs/TSC#886 (comment)

We need to kickstart the discussion again on how we plan to support the new Mac hardware which is due to be announced on Nov 10th.

We last had the discussion in the build meeting in july where we had some discussion as to whether to do 2 binaries or to go for fat binaries.

We also need to discuss sourcing hardware - We currently have two DTKs kindly donated by macstadium but no release machine as of yet and these DTKs will go back to apple at some point.

To Do:

• Add 11.0 to versionselector to ensure we build only on master
• Enable the macos-arm job in normal test regressions
• Get the DTKs back into CI for testing
• Test compiling and running node natively on the mac arm hardware
• Test if a current release runs on the new hardware through the translation layer
• Figure out a source of hardware to add to our CI for when our DTKs go back to apple
• Note down size difference in binaries between: intel, arm and fat.

[mhdawson]

I think we'd agree to do fat binaries as they seemed preferable for a number of reasons.

In terms of hardware longer term, we should see if macstadium has a longer term answer for that now as having them provide the hardware is the easiest path for us.

[rvagg]

+1 to fat (again, I think? I don't recall). We used to do fat for x86+x64 until we ripped that out for io.js (and then Node 4). There's still echoes of the code to do it in nodejs/node Makefile and tools/ and it's certainly in the git history.

For hardware, we really should have redundancy here too. Same approach as we have for x64. Rely on macstadium (assuming they'll let us use official hardware for this) but get our own and ask nearForm if they'll kindly host them for us as backups.

[gengjiawen]

We have PRs need to run on silicon too: nodejs/node#35986.

[mhdawson]

@rvagg redundancy makes sense. I don't think we can order ARM mac mini H/W yet, correct me if I'm wrong on that and I can start the discussion with the Foundation.

[rvagg]

No, we're really just waiting to see Apple's announcement tomorrow. Rumours are suggesting laptops, which is probably not going to be helpful. But 🤞 we may get mini's out of this in which case we should jump straight on it.

[AshCripps]

Yeah the event looks to be the announcement of just 2 laptops according to rumours at least - we might be another year or so away from iMacs and mac minis.

[AshCripps]

Turns out they did announce new mac minis so it might be worth speaking with the guys at macstadium again to see their timeline.

[rvagg]

@mhdawson can you start the discussion with the Foundation about getting a couple of minis? And we'll need to make sure we have somewhere to put them, I guess we need to probe nearForm on this unless we have other options.

[AshCripps]

just as an FYI I am working to get the DTKs into our regular regression runs to get some testing at least on the platform - I managed to brick one updating to the latest big sur but macstadium have swapped it out for another one. Im waiting to hear back if its safe to update the second one before I carry on.

[mhdawson]

@rvagg last time we also asked for virtualization software so that we could have a 2 vms on each mac mini so that 1 for test and one for release would be good enough. Do you know if the same software will work on the arm machines?

We'll also need to agree on the specs. From https://www.apple.com/ca/shop/buy-mac/mac-mini there only seem to be 2 choices, one with the only difference being the SSD size.

[AshCripps]

Key difference this time is the minis are 8 core instead of 6 core so we may be able to get away with running 3 VMs on one mini, The next issue is VMware fusion which I cant seem to find any more mention besides "soon" or "we are working on it" for M1 support

[nschonni]

I think there is also supposed to be support for 16GB or ram https://www.anandtech.com/show/16235/apple-intros-first-three-apple-silicon-macs-late-2020-macbook-air-13inch-macbook-pro-mac-mini but it doesn't look like it's on the shop either
The memory option is there once you select the HDD model

[rvagg]

Oh, right yeah, the VM problem. I bet VMWare aren't treating this as a top priority. I wonder if we need to shop around for that. Maybe Orka is the only game in town for now unless we're happy to go bare metal for these?

[nschonni]

Looks like the license was updated around virtualisation https://blog.macstadium.com/blog/developers-big-sur-and-vindication

[rvagg]

So .. here's the key piece that someone needs to figure out - how do we get our toolchain to compile arm64 macos binaries on x64?

Here's how we used to make fat binaries: https://github.com/nodejs/node/blob/archived-io.js-v0.10/Makefile#L291-L301

i.e. we could do a --dest-cpu=ia32 build and a --dest-cpu=x64 build and smush the resulting node binaries together with lipo. This second piece is still the right way as far as I understand, the problem is the --dest-cpu=arm64 bit.

Here's some things I've tinkered with so far on my x64 mac:

• Installed Xcode 12 beta which I believe is the only way to get this to work (I could be corrected on this, the fact that it's still beta has me confused about this point)
• Manually added 'SDKROOT': '/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/' to my common.gypi in the 'target_arch=="arm64"' section for the "mac" block (without this it keeps on defaulting to my system SDK, I assume if we built on the latest OS this would default to the right directory).
• ./configure --dest-cpu=arm64 and make
• Added --without-intl to ./configure because icupkg breaks the build - this is an expected part of cross compiling, we probably need a system ICU installed to make this work (although this isn't a great option for release machines where we want to push the ICU version with the source so we might need to figure out how to first compile it for x64 then use that binary to run against the arm64 build).

But my latest failure is V8 related: OSError: [Errno 86] Bad CPU type in executable: '/Users/rvagg/git/iojs/io.js/out/Release/bytecode_builtins_list_generator'.

I haven't got any further than that. Also, please don't count on me to solve all of these problems, I don't really have time to be doing any of this and was really just checking to see how far off we might be -- this all suggests that the answer is "quite a way off, and it will require coordinated effort across subsystems to make this happy".

[rvagg]

It seems CC_target and CC_host can be (ab)used here to get further: CC_host="ccache cc -arch x86_64" CXX_host="ccache c++ -arch x86_64" CC_target="ccache cc -arch arm64" CXX_target="ccache c++ -arch arm64" ./configure --dest-cpu=arm64, although that gets me to asm errors in out/Release/obj.host/v8_base_without_compiler/deps/v8/src/heap/base/asm/x64/push_registers_asm.o.d.raw during the x64 phase. But that's very late in the build. It could be because this approach results in a double -arch being supplied and maybe the later one is overriding the former.

[AshCripps]

https://developer.apple.com/documentation/xcode/building_a_universal_macos_binary Looks like Apple says to use lipo too but I dont know enough about the node source code to say if we have those conditional macros in place.

[rvagg]

Yeah, it's the two -arch's that are appearing in the compile commands with the later one overriding the first. So if I remove the two lines that add the -arch from tools/gyp/pylib/gyp/xcode_emulation.py and rely on the ones supplied from my CC_host and CC_target:

CC_host="ccache cc -arch x86_64" CXX_host="ccache c++ -arch x86_64" CC_target="ccache cc -arch arm64" CXX_target="ccache c++ -arch arm64" CC="ccache cc -arch arm64" CXX="ccache c++ -arch arm64" ./configure --dest-cpu=arm64

Then I get what appears to be an arm64 binary:

$ file node
node: Mach-O 64-bit executable arm64

That's probably not super useful other than confirming that it is possible to get a cross-compiled binary out of our toolchain, if lightly hacked. It doesn't suggest that path to getting this working properly though. At this stage I'm suspecting that something about the CFLAGS changed in GYP along the way that messed this up. I don't think xcode_emulation.py has really changed much in all that time, but make.py has, including some of the CFLAGS logic (including the _host and _target stuff).

FTR the diff that gets me here is:

diff --git a/common.gypi b/common.gypi
index 4745bb5ac7..1e00c64f0f 100644
--- a/common.gypi
+++ b/common.gypi
@@ -510,6 +510,7 @@
           }],
           ['target_arch=="arm64"', {
             'xcode_settings': {
+              'SDKROOT': '/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/',
               'ARCHS': ['arm64'],
               'OTHER_LDFLAGS!': [
                 '-Wl,-no_pie',
diff --git a/tools/gyp/pylib/gyp/xcode_emulation.py b/tools/gyp/pylib/gyp/xcode_emulation.py
index 8af2b39f9a..fa1bd07ab7 100644
--- a/tools/gyp/pylib/gyp/xcode_emulation.py
+++ b/tools/gyp/pylib/gyp/xcode_emulation.py
@@ -663,7 +663,7 @@ class XcodeSettings(object):
             # TODO: Supporting fat binaries will be annoying.
             self._WarnUnimplemented("ARCHS")
             archs = ["i386"]
-        cflags.append("-arch " + archs[0])
+        # cflags.append("-arch " + archs[0])

         if archs[0] in ("i386", "x86_64"):
             if self._Test("GCC_ENABLE_SSE3_EXTENSIONS", "YES", default="NO"):
@@ -947,7 +947,7 @@ class XcodeSettings(object):
             # TODO: Supporting fat binaries will be annoying.
             self._WarnUnimplemented("ARCHS")
             archs = ["i386"]
-        ldflags.append("-arch " + archs[0])
+        # ldflags.append("-arch " + archs[0])

         # Xcode adds the product directory by default.
         # Rewrite -L. to -L./ to work around http://www.openradar.me/25313838

[rvagg]

Oh, so here's the most plausible explanation as to why it used to work for our fat binary builds but now doesn't - the i386 build, simply using --dest-cpu=ia32 built binaries (including the build-time tooling) that actually ran on the compile host, because the compile host could run i386 binaries as well as x64. So -arch i386 got passed through to everything and it didn't matter, the resulting binaries could run regardless of whether it was for target or host. But now we need differentiation for cross-compiling, like we do for the ARMv7 cross compile, where there's a separation of these arch parameters between _host and _target so we get build-time tooling in the host's architecture and get the proper _target architecture in the resulting binary.

So it's this hard-wired -arch in xcode_emulation.py that's our main problem. Either we need it to properly switch between _host and _target (the elegant way) or we get to override it with the custom environment variables like I've done above (the hacky way).

@nodejs/gyp this is going to have to involve you to some degree, I don't know how familiar you are with xcode_emulation.py.

[rvagg]

A proposal for shoe-horning this in nodejs/gyp-next#78

(doesn't solve the SDKROOT issue though)

[AshCripps]

we held a meeting today and come up with the following next steps as a starter:

• Get the DTKs back into CI for testing
• Test compiling and running node natively on the mac arm hardware
• Test if a current release runs on the new hardware through the translation layer
• Figure out a source of hardware to add to our CI for when our DTKs go back to apple
• Note down size difference in binaries between: intel, arm and fat.

as a note we need to figure out what breaking changes we might introduce with the apple silicon support and plan around targeting a specific release (e.g. 16.x)

Also need to ensure we get a green CI run on the new hardware as there have been some untested (by us) prs around apple silicon land on master recently.

[STRML]

Just want to confirm since I got one in the mail today. No compatibility problems with node, it just works - tested with a large/complex project - and even runs very well compared to Intel chips. So new owners should not be complaining this week.

The work is still worth doing given the ~30% performance boost it unlocks.

[bdkjones]

@nschonni Thanks. From the perspective of an outsider, I don't think Node has really explained their timeline or their progress on this issue. As someone at nodejs/TSC#886 pointed out: there is no single source of truth for those of us trying to get Node running on arm64—you have to cobble together bits and pieces from several threads all over the place. The people with the expertise are in these threads, so it's natural to talk to them there.

Also: These threads are the top results when Googling "Build Node on Apple Silicon" and similar. That's naturally going to funnel people here, unfortunately.

If you want to avoid that in the future, when large-scale platform transitions occur, it would be a good idea to update the "releases" page at nodejs.org with a link that says something like, "Trying to build Node on that new M1 Mac? Here's what you need to know." Centralize the expertise and you won't be bothered by the unwashed idiot masses (me).

[AshCripps]

There is a pinned issue on nodejs/node which is as up to date as to our progress - nodejs/node#37309

[kalwiggins]

anyone get node 10 to work on apple silicon? I haven't been able too

[architchandra]

Can't get Node 10.14 to install using nvm on an M1 Mac Air. Any workarounds?

[billinghamj]

I would say that is expected. Node 15 supports M1. For older versions, use Rosetta

Edit:

Keep in mind this note at the top of #37309:

NOTE: Support is very unlikely to be back ported to 14.x or before due to requiring a SemVer Major V8 update

[arcreigh]

All my M1 mini is in route, while I am not a developer I want to assist in any way that I can. Even if it means picking up a new skill / helping test compilation or running various tests.

[AshCripps]

Closing. Support for Apple silicon is 15.x and above, and the binaries will be released with the 16.x release next week

[marckohlbrugge]

Are the Apple Silicon builds for 15.x still planned? I can see them for 16.x but not for 15.x

[targos]

No, and there are no more releases planned for v15.x

[7rulnik]

@targos what about 14.17+?

[AshCripps]

@7rulnik 14.x releases of apple silicon are currently not planned but the support is in the code base if you wish to build from source.

[tometo-dev]

Can't get Node 10.14 to install using nvm on an M1 Mac Air. Any workarounds?

@architchandra did you find a solution?

[johanneswuerbach]

@tsuki42 a workaround is documented here nvm-sh/nvm#2350 (comment)

[tometo-dev]

@johanneswuerbach That seems to have worked. Thanks! 🙏