Failed to load .wasm on macOS running on Apple silicon

[dio]

Background:

Currently, since the following PR is merged,

• bazel: disable wasm on macOS for now #20889

macOS binaries v1.23.x (e.g. fetched via Homebrew), and main by default doesn't support Wasm.

Relevant:

• Set --define wasm=wasmtime for macOS by default #23235.

Since @keith's CL on v8 upstream is merged. Also, that CL is actually included in the current (also release/1.23) linked v8: 10.4.132.18:

envoy/bazel/repository_locations.bzl

Line 1053 in 6389f16

version = "10.4.132.18",

.

Therefore, now we can successfully build envoy on macOS on Apple silicon (tested on M1) 🎉. However, when we run it to load a .wasm file (from examples/wasm-cc), we caught on: https://chromium.googlesource.com/v8/v8.git/+/refs/tags/10.4.132.18/src/base/platform/platform-posix.cc#478, we expect ENOMEM (12) but got EACCES (13, Permission denied). (full log: https://gist.githubusercontent.com/dio/7ef745c09327aa6da0be3b32ef28016a/raw/d220e891afe52208d346ed0dae0d72220275fe73/v8-log.txt).

#
# Fatal error in external/v8/src/base/platform/platform-posix.cc, line 478
# Check failed: 12 == (*__error()) (12 vs. 13).
#
#
#
#FailureMessage Object: 0x16b63df60
==== C stack trace ===============================

    0   envoy-static                        0x000000010a935be4 v8::base::debug::StackTrace::StackTrace() + 32
    1   envoy-static                        0x000000010a935c20 v8::base::debug::StackTrace::StackTrace() + 28
    2   envoy-static                        0x0000000109e23308 v8::platform::(anonymous namespace)::PrintStackTrace() + 28
    3   envoy-static                        0x000000010a8ff940 V8_Fatal(char const*, int, char const*, ...) + 208
    4   envoy-static                        0x000000010a92fdd0 v8::base::OS::SetPermissions(void*, unsigned long, v8::base::OS::MemoryPermission) + 424
    5   envoy-static                        0x000000010a90fb20 v8::base::PageAllocator::SetPermissions(void*, unsigned long, v8::PageAllocator::Permission) + 44
    6   envoy-static                        0x000000010a8f13f8 v8::base::BoundedPageAllocator::SetPermissions(void*, unsigned long, v8::PageAllocator::Permission) + 252
    7   envoy-static                        0x0000000108a5c2b0 v8::internal::VirtualMemory::SetPermissions(unsigned long, unsigned long, v8::PageAllocator::Permission) + 140
    8   envoy-static                        0x0000000108044b08 v8::internal::MemoryAllocator::PartialFreeMemory(v8::internal::BasicMemoryChunk*, unsigned long, unsigned long, unsigned long) + 520
    9   envoy-static                        0x000000010810aa78 v8::internal::Page::ShrinkToHighWaterMark() + 1136
    10  envoy-static                        0x00000001080961a4 v8::internal::PagedSpaceBase::ShrinkPageToHighWaterMark(v8::internal::Page*) + 36
    11  envoy-static                        0x0000000108096524 v8::internal::PagedSpaceBase::ShrinkImmortalImmovablePages() + 256
    12  envoy-static                        0x0000000107ee83b0 v8::internal::Heap::NotifyDeserializationComplete() + 92
    13  envoy-static                        0x0000000107d21a94 v8::internal::Isolate::Init(v8::internal::SnapshotData*, v8::internal::SnapshotData*, v8::internal::SnapshotData*, bool) + 5192
    14  envoy-static                        0x0000000107d22230 v8::internal::Isolate::InitWithSnapshot(v8::internal::SnapshotData*, v8::internal::SnapshotData*, v8::internal::SnapshotData*, bool) + 216
    15  envoy-static                        0x0000000108a132a0 v8::internal::Snapshot::Initialize(v8::internal::Isolate*) + 792
    16  envoy-static                        0x0000000107862a54 v8::Isolate::Initialize(v8::Isolate*, v8::Isolate::CreateParams const&) + 980
    17  envoy-static                        0x00000001078631e4 v8::Isolate::New(v8::Isolate::CreateParams const&) + 36
    18  envoy-static                        0x00000001072bf48c wasm::Store::make(wasm::Engine*) + 308
    19  envoy-static                        0x00000001072779d4 proxy_wasm::v8::V8::load(std::__1::basic_string_view<char, std::__1::char_traits<char> >, std::__1::basic_string_view<char, std::__1::char_traits<char> >, std::__1::unordered_map<unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::hash<unsigned int>, std::__1::equal_to<unsigned int>, std::__1::allocator<std::__1::pair<unsigned int const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > > const&) + 64
    20  envoy-static                        0x0000000107247b04 proxy_wasm::WasmBase::load(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool) + 1536
    21  envoy-static                        0x000000010724a014 proxy_wasm::createWasm(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<proxy_wasm::PluginBase> const&, std::__1::function<std::__1::shared_ptr<proxy_wasm::WasmHandleBase> (std::__1::basic_string_view<char, std::__1::char_traits<char> >)> const&, std::__1::function<std::__1::shared_ptr<proxy_wasm::WasmHandleBase> (std::__1::shared_ptr<proxy_wasm::WasmHandleBase>)> const&, bool) + 576

That check was introduced by: https://chromium-review.googlesource.com/c/v8/v8/+/3610445. Seems like, when on macOS, it (if (ret != 0) CHECK_EQ(ENOMEM, errno);) should be executed after the "workaround" for macOS on Apple silicon is executed (to use madvise, caused by the mprotect "bug" on Apple silicon since macOS 11.2: https://bugs.chromium.org/p/v8/issues/detail?id=11389#c14. As additional info: this workaround was introduced by https://chromium-review.googlesource.com/c/v8/v8/+/2679688).

The proposed additional entry to bazel/v8.patch:

diff --git a/src/base/platform/platform-posix.cc b/src/base/platform/platform-posix.cc
index 131ff9614e..8699699861 100644
--- a/src/base/platform/platform-posix.cc
+++ b/src/base/platform/platform-posix.cc
@@ -472,11 +472,6 @@ bool OS::SetPermissions(void* address, size_t size, MemoryPermission access) {
   int prot = GetProtectionFromMemoryPermission(access);
   int ret = mprotect(address, size, prot);

-  // Any failure that's not OOM likely indicates a bug in the caller (e.g.
-  // using an invalid mapping) so attempt to catch that here to facilitate
-  // debugging of these failures.
-  if (ret != 0) CHECK_EQ(ENOMEM, errno);
-
   // MacOS 11.2 on Apple Silicon refuses to switch permissions from
   // rwx to none. Just use madvise instead.
 #if defined(V8_OS_DARWIN)
@@ -486,6 +481,11 @@ bool OS::SetPermissions(void* address, size_t size, MemoryPermission access) {
   }
 #endif

+  // Any failure that's not OOM likely indicates a bug in the caller (e.g.
+  // using an invalid mapping) so attempt to catch that here to facilitate
+  // debugging of these failures.
+  if (ret != 0) CHECK_EQ(ENOMEM, errno);
+
   if (ret == 0 && access == OS::MemoryPermission::kNoAccess) {
     // This is advisory; ignore errors and continue execution.
     USE(DiscardSystemPages(address, size));

I tested to build and run the produced binary on macOS running on M1 (it successfully loads the .wasm file):

   wasm-cc (mac-wasm): uname -a
Darwin Dhis-MacBook-Air.local 21.6.0 Darwin Kernel Version 21.6.0: Mon Aug 22 20:20:05 PDT 2022; root:xnu-8020.140.49~2/RELEASE_ARM64_T8101 arm64
   wasm-cc (mac-wasm): file ~/src/envoy/bazel-bin/source/exe/envoy-static
/Users/dio/src/envoy/bazel-bin/source/exe/envoy-static: Mach-O 64-bit executable arm64
   wasm-cc (mac-wasm): pwd
/Users/dio/src/envoy/examples/wasm-cc
   wasm-cc (mac-wasm): ~/src/envoy/bazel-bin/source/exe/envoy-static -c envoy.yaml 
...
[2022-09-25 20:43:18.254][2501953][info][main] [source/server/server.cc:904] starting main dispatch loop

If this patch does make sense, I'd like to raise PRs both for release/1.23 and main branches, since both of them are linked to the same v8 version: 10.4.132.18.

Thanks!

[dio]

Another suggestion is to set v8_enable_external_code_space to true (see: https://chromium-review.googlesource.com/c/v8/v8/+/3610445/comment/6294ada3_a3cab643/), but it seems like v8_enable_external_code_space is part of V8 GN flags that are not currently supported from bazel.

Also need to try to update v8 :) nodejs/node-v8#226 (comment)

[wbpcode]

cc @PiotrSikora