macOS 11.2: Fatal error Check failed: allocator->SetPermissions

[andreialecu]

• Version: v15.6.0
• Platform: Darwin Andreis-Mac-mini.local 20.3.0 Darwin Kernel Version 20.3.0: Thu Jan 14 14:38:22 PST 2021; root:xnu-7195.81.2~2/RELEASE_ARM64_T8101 arm64
• Subsystem: wasm(?)

What steps will reproduce the bug?

Use latest macOS Beta 11.2 (released 24 Jan 2021) and node 15.6.0

npm install -g yarn
cd `mktemp -d`
yarn init -2
yarn add lodash

How often does it reproduce? Is there a required condition?

All the time.

What is the expected behavior?

No crash.

What do you see instead?

#
# Fatal error in , line 0
# Check failed: allocator->SetPermissions(reinterpret_cast<void*>(region.begin()), region.size(), PageAllocator::kNoAccess).
#
#
#
#FailureMessage Object: 0x16f986408
 1: 0x100575758 node::NodePlatform::GetStackTracePrinter()::$_3::__invoke() [/Users/andreialecu/.nvm/versions/node/v15.6.0/bin/node]
 2: 0x1012bdd50 V8_Fatal(char const*, ...) [/Users/andreialecu/.nvm/versions/node/v15.6.0/bin/node]
 3: 0x100c0fb98 v8::internal::wasm::WasmCodeManager::Decommit(v8::base::AddressRegion) [/Users/andreialecu/.nvm/versions/node/v15.6.0/bin/node]
 4: 0x100c13218 v8::internal::wasm::NativeModule::FreeCode(v8::internal::Vector<v8::internal::wasm::WasmCode* const>) [/Users/andreialecu/.nvm/versions/node/v15.6.0/bin/node]
 5: 0x100c22400 v8::internal::wasm::WasmEngine::FreeDeadCodeLocked(std::__1::unordered_map<v8::internal::wasm::NativeModule*, std::__1::vector<v8::internal::wasm::WasmCode*, std::__1::allocator<v8::internal::wasm::WasmCode*> >, std::__1::hash<v8::internal::wasm::NativeModule*>, std::__1::equal_to<v8::internal::wasm::NativeModule*>, std::__1::allocator<std::__1::pair<v8::internal::wasm::NativeModule* const, std::__1::vector<v8::internal::wasm::WasmCode*, std::__1::allocator<v8::internal::wasm::WasmCode*> > > > > const&) [/Users/andreialecu/.nvm/versions/node/v15.6.0/bin/node]
 6: 0x100c20108 v8::internal::wasm::WasmEngine::PotentiallyFinishCurrentGC() [/Users/andreialecu/.nvm/versions/node/v15.6.0/bin/node]
 7: 0x100c2157c v8::internal::wasm::WasmEngine::ReportLiveCodeForGC(v8::internal::Isolate*, v8::internal::Vector<v8::internal::wasm::WasmCode*>) [/Users/andreialecu/.nvm/versions/node/v15.6.0/bin/node]
 8: 0x100c23d20 v8::internal::wasm::(anonymous namespace)::WasmGCForegroundTask::RunInternal() [/Users/andreialecu/.nvm/versions/node/v15.6.0/bin/node]
 9: 0x100573bf4 node::PerIsolatePlatformData::RunForegroundTask(std::__1::unique_ptr<v8::Task, std::__1::default_delete<v8::Task> >) [/Users/andreialecu/.nvm/versions/node/v15.6.0/bin/node]
10: 0x10057288c node::PerIsolatePlatformData::FlushForegroundTasksInternal() [/Users/andreialecu/.nvm/versions/node/v15.6.0/bin/node]
11: 0x100d551a8 uv__async_io [/Users/andreialecu/.nvm/versions/node/v15.6.0/bin/node]
12: 0x100d66d68 uv__io_poll [/Users/andreialecu/.nvm/versions/node/v15.6.0/bin/node]
13: 0x100d55638 uv_run [/Users/andreialecu/.nvm/versions/node/v15.6.0/bin/node]
14: 0x100472bd0 node::SpinEventLoop(node::Environment*) [/Users/andreialecu/.nvm/versions/node/v15.6.0/bin/node]
15: 0x10054caf4 node::NodeMainInstance::Run(node::EnvSerializeInfo const*) [/Users/andreialecu/.nvm/versions/node/v15.6.0/bin/node]
16: 0x1004ecc20 node::Start(int, char**) [/Users/andreialecu/.nvm/versions/node/v15.6.0/bin/node]
17: 0x199435f34 start [/usr/lib/system/libdyld.dylib]

Additional information

This started happening after the macOS beta update I think. I was on beta before, but everything was working fine until after the update.

I'm on an ARM64 M1 chip (mac mini 16gb ram)

node/deps/v8/src/wasm/wasm-code-manager.cc

Lines 1630 to 1631 in 770d9e2

	CHECK(allocator->SetPermissions(reinterpret_cast<void*>(region.begin()),
	region.size(), PageAllocator::kNoAccess));

[andreialecu]

Some additional information:

• It appears that nodejs 15.6 runs on v8 8.6, sharing the same V8 version with Chromium 86.
• I installed Chromium 86 to test WASM, and it crashes there too.
• However, Chromium 88+ which use v8 8.8 do not have the crash.

See microsoft/vscode#113410 (comment) for more details.

[GautierT]

@andreialecu : did you find a workaround ? Thanks !

[andreialecu]

Unfortunately no. I had to revert to nodejs 14, using Rosetta. Hopefully only temporarily.

arch -x86_64 zsh

nvm install 14
nvm alias default 14
exit

[GautierT]

yes i did the same but sad to have to use back Rosetta... Thanks.

[EriksRemess]

Same crash happens on latest MacOS 11.2 RC2 (20D62).

Update:
Node v15.7.0 released today has the same issue.

[MarcoZehe]

Thanks for filing this and investigating the cause, @andreialecu . I am running into this when trying to run the Netlify CLI.

[matinzd]

I also have this problem on apple silicon node v 15.7.0.

System info

Darwin Matins-MacBook-Pro.local 20.3.0 Darwin Kernel Version 20.3.0: Thu Jan 14 14:38:22 PST 2021; root:xnu-7195.81.2~2/RELEASE_ARM64_T8101 arm64

Error:

$ react-native start

#
# Fatal error in , line 0
# Check failed: allocator->SetPermissions(reinterpret_cast<void*>(region.begin()), region.size(), PageAllocator::kNoAccess).
#
#
#
#FailureMessage Object: 0x16efd4748
 1: 0x100f09044 node::NodePlatform::GetStackTracePrinter()::$_3::__invoke() [/opt/homebrew/Cellar/node/15.7.0/bin/node]
 2: 0x101890fdc V8_Fatal(char const*, ...) [/opt/homebrew/Cellar/node/15.7.0/bin/node]

[panr]

Big Sur 11.2 RC3 doesn't change anything. Still get the same error...

[andreialecu]

@aduh95 apologies for tagging you on this, but given your mac arm64 commits, perhaps you would know who to forward this to since it is a very serious problem.

11.2 will be out any day now and this will hit a lot of people.

[panr]

I just reported this directly to Apple. Will see what happens...

[andreialecu]

I also reported it to Apple several days ago but that's usually a black hole.

I'm inclined to believe that v8 is doing something wrong here, because recent Chrome versions don't have this issue, but older Chrome versions do (ones which use the same v8 version as node). See #37061 (comment) and microsoft/vscode#113410 (comment)

[aduh95]

@aduh95 apologies for tagging you on this, but given your mac arm64 commits, perhaps you would know who to forward this to since it is a very serious problem.

11.2 will be out any day now and this will hit a lot of people.

Unfortunately I don't have M1 macs at my disposal anymore.

If the problem is the V8 version, could you try compile node with a more recent one? You can try out the V8 8.8 PR which is almost ready now:

# on a local git repo sync with the latest master
curl -L https://github.com/nodejs/node/pull/36139.patch | git am
make -j8

[andreialecu]

Thanks for the tip @aduh95, I built it but it doesn't seem to be enough:

➜ which node
/Users/andreialecu/Work/contrib/node/node

➜ node -v
v16.0.0-pre

➜ yarn install

...
#
# Fatal error in , line 0
# Check failed: allocator->SetPermissions(reinterpret_cast<void*>(region.begin()), region.size(), PageAllocator::kNoAccess).
#
#
#
#FailureMessage Object: 0x16db051f8
 1: 0x1023feb58 node::NodePlatform::GetStackTracePrinter()::$_3::__invoke() [/Users/andreialecu/Work/contrib/node/out/Release/node]
 2: 0x103150998 V8_Fatal(char const*, ...) [/Users/andreialecu/Work/contrib/node/out/Release/node]
 3: 0x102aa8710 v8::internal::wasm::WasmCodeManager::Decommit(v8::base::AddressRegion) [/Users/andreialecu/Work/contrib/node/out/Release/node]
 4: 0x102aac384 v8::internal::wasm::NativeModule::FreeCode(v8::internal::Vector<v8::internal::wasm::WasmCode* const>) [/Users/andreialecu/Work/contrib/node/out/Release/node]
 5: 0x102abb448 v8::internal::wasm::WasmEngine::FreeDeadCodeLocked(std::__1::unordered_map<v8::internal::wasm::NativeModule*, std::__1::vector<v8::internal::wasm::WasmCode*, std::__1::allocator<v8::internal::wasm::WasmCode*> >, std::__1::hash<v8::internal::wasm::NativeModule*>, std::__1::equal_to<v8::internal::wasm::NativeModule*>, std::__1::allocator<std::__1::pair<v8::internal::wasm::NativeModule* const, std::__1::vector<v8::internal::wasm::WasmCode*, std::__1::allocator<v8::internal::wasm::WasmCode*> > > > > const&) [/Users/andreialecu/Work/contrib/node/out/Release/node]

[aduh95]

@andreialecu What does node -p process.versions.v8 returns? If you visit chrome://version/ on a Chromium-browser that doesn't have the bug, what V8 version is it running?

[andreialecu]

➜ node -p process.versions.v8
8.8.278.14-node.21


Google Chrome | 88.0.4324.96 (Official Build) (arm64)
-- | --
Revision | 68dba2d8a0b149a1d3afac56fa74648032bcf46b-refs/branch-heads/4324@{#1784}
OS | macOS Version 11.2 (Build 20D53)
JavaScript | V8 8.8.278.14

Seems to be exactly the same.

I'm not sure if I tested this properly, or if Chromium 86 had a different issue, but I mentioned here what I did: microsoft/vscode#113410 (comment)

[aduh95]

The yarn install part of the repro makes it a bit tedious to reproduce, any chance you are able to provide a smaller repro? Maybe a simple out/Release/node test/parallel/test-wasm-simple.js would be enough?

[andreialecu]

➜ out/Release/node test/parallel/test-wasm-simple.js seems to complete without errors. I assume there's some different code path that ends up being used when using yarn install.

To others in this thread:

• What are you running to get this crash?
• Is everyone on Apple Silicon or does it happen on Intel as well?

[matinzd]

➜ out/Release/node test/parallel/test-wasm-simple.js seems to complete without errors. I assume there's some different code path that ends up being used when using yarn install.

To others in this thread:

• What are you running to get this crash?
• Is everyone on Apple Silicon or does it happen on Intel as well?

Just Apple Silicon. No problem on Intel. Happens in some projects.

[andreialecu]

@aduh95 Some additional info.

Removing the CHECK() from this line:

node/deps/v8/src/wasm/wasm-code-manager.cc

Lines 1630 to 1631 in 770d9e2

	CHECK(allocator->SetPermissions(reinterpret_cast<void*>(region.begin()),
	region.size(), PageAllocator::kNoAccess));

Resolves the problem, as expected. I'm able to run yarn properly and other commands.

Why it fails though, is a mistery. I went through a bit of a rabbit hole reading source to see where it leads to and found this part:

node/deps/v8/src/base/platform/platform-posix.cc

Lines 417 to 421 in b9dfda9

	int ret = mprotect(address, size, prot);
	if (ret == 0 && access == OS::MemoryPermission::kNoAccess) {
	// This is advisory; ignore errors and continue execution.
	USE(DiscardSystemPages(address, size));
	}

I assume mprotect fails and the entire method returns false.

I found this https://bugs.exim.org/show_bug.cgi?id=2618 mentioning mprotect, JIT and Apple Silicon, which may be relevant.

[aduh95]

Possibly related: Probably related: v8/v8@4e077ff

That's the most recent commit that references JIT and aarm64, and was included in V8 8.8.

Would changing this instruction:

node/deps/v8/src/wasm/wasm-code-manager.cc

Lines 1630 to 1631 in 770d9e2

	CHECK(allocator->SetPermissions(reinterpret_cast<void*>(region.begin()),
	region.size(), PageAllocator::kNoAccess));

to

 CHECK(allocator->SetPermissions(reinterpret_cast<void*>(region.begin()), 
                                 region.size(), PageAllocator::kNoAccessWillJitLater)); 

solve the issue?

[andreialecu]

@aduh95 nope:

# Fatal error in , line 0
# Check failed: allocator->SetPermissions(reinterpret_cast<void*>(region.begin()), region.size(), PageAllocator::kNoAccessWillJitLater).

[andreialecu]

So I just tried some random WASM packages from npm using their example scripts and they seem to run properly on 15.6 and on 16-dev. test/parallel/test-wasm-simple.js also succeeds.

So WASM does work to some extent, until that Decommit code path gets called.

I can reproduce it 100% using yarn@^2. I'm not entirely sure what it does differently vs other WASM modules I tested.

This may also indicate that Chrome and recent v8 versions also have the bug, just that it doesn't reproduce as obviously there.

[andreialecu]

Here's an easy way to get the patch applied:

Ensure you have nvm installed:

# install nvm if you don't have it:
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.37.2/install.sh | bash

I made a gist of the patch for easy download:

cd ~
curl https://gist.githubusercontent.com/andreialecu/54b17d7872d047a61e85dcf0261a6105/raw/da5b55457533397e178a021ccfbf9cb827da9d8b/node-arm64.diff > node-patch.diff

nvm install 15.8

(if you already have node installed via nvm, you may need to uninstall it first - nvm uninstall 15)

Now, wait for a bit for nvm to download the sources. Keep an eye on the terminal output.

Once it starts compiling, open a second terminal and run the following. You should have plenty of time before it starts compiling the relevant files.

cd ~/.nvm/.cache/src/node-v15.8.0/files
patch -p1 < ~/node-patch.diff

After it's done you can use nvm use 15 before running node commands to ensure it works. If you want to make it the default version for your shells, run nvm alias default 15.8.0

[carloskiki]

I built node from source using homebrew brew reinstall --build-from-source node and it seems like it doesn't work.

I still get the error when I use npm start on a react app. Can somebody explain to me why?

[goranmoomin]

@Carlosted I haven't used homebrew for a few months (I switched to MacPorts), but you should add --HEAD to your command to compile with the latest source.

[andresilva-cc]

@Carlosted I haven't used homebrew for a few months (I switched to MacPorts), but you should add --HEAD to your command to compile with the latest source.

I can confirm that using the command brew install --HEAD node installs v16.0.0-head which already has this fix.

[Virgil-N]

New version of Node (15.8.0) fixes issue for me.

Not for me.😂

[jakobkummerow]

@adietrichs : That's an unrelated issue. Please file a bug at crbug.com/v8/new and include as much detail as you can (ideally: repro instructions).

[aduh95]

The fix should land in Node.js v15.9.0 (expected release date: today).

Please stop commenting the issue is still happening with Node.js v15.8.0, we know, the fix is coming. In the mean time, please wait for v15.9.0 to be released, or build from source, or use the x86_64 version with Rosetta.

[etiennecossart]

Just upgraded to v15.9.0 and it resolved the Safari localhost connection issue

[karlhorky]

Homebrew Pull Request: Homebrew/homebrew-core#71400

[andresilva-cc]

I've uninstalled node --HEAD via brew and installed it again (node@15), and I can confirm that it is working as expected. Thanks, everyone!

[m-chhapariya]

I was also facing the similar issue with nodejs 15.3.0 on mac M1 chip

Starting the development server...

# Fatal error in , line 0
# Check failed: allocator->SetPermissions(reinterpret_cast<void*>(region.begin()), region.size(), PageAllocator::kNoAccess)

I tried installing node 17.2.0 and it is working fine.

Successful logs:

Compiled successfully!

You can now view my-react-app in the browser.

  Local:            http://localhost:3000
  On Your Network:  http://192.168.0.102:3000

Note that the development build is not optimized.
To create a production build, use npm run build.

[dio]

@matinzd or @targos if you don't mind, could you help to provide the gist.githubusercontent.com/targos/8ca8f7a0c76847f29f5658a93c30af1c/raw/adf420ca34e1e0a74c5a70ece059c3d0d2c5c043/crash.js, currently it is 404, attached to #37276? Wanted to learn about this :). Thanks!