Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Insiders on macOS ARM64: Extension Host crashes (SIGILL, EFAULT, EPIPE) #113410

Closed
andreialecu opened this issue Dec 25, 2020 · 89 comments
Closed
Assignees
Labels
🍎 si Issues related to apple silicon bug Issue identified by VS Code Team member as probable bug extension-host Extension host issues freeze-slow-crash-leak VS Code crashing, performance, freeze and memory leak issues

Comments

@andreialecu
Copy link

andreialecu commented Dec 25, 2020

Issue Type: Bug

I haven't seen this reported anywhere, so here goes.

Using the latest insiders version on macOS, I noticed that there are stability issues with extensions.

After some time running vscode, the extension host either crashes and needs to be restarted, or the code actions that trigger on saving a document start hanging.

I noticed that in the Developer Tools it frequently logs messages like:

Uncaught Exception:  Error: read EFAULT
    at Pipe.onStreamRead (internal/stream_base_commons.js:205:27)
t.log @ console.ts:137

Screenshot 2020-12-25 at 10 14 32

The EFAULT exceptions might be triggering this behavior.

This also hangs and spins forever:
Screenshot 2020-12-25 at 10 10 39

If I cancel it, then run Developer: Restart Extension Host it will then save properly.

(It's possible one of my extensions might be interfering as well, as I have a ton of them enabled - but I don't remember seeing this on Stable VSCode with Rosetta.)

VS Code version: Code - Insiders 1.53.0-insider (4a875e2, 2020-12-21T12:34:09.548Z)
OS version: Darwin arm64 20.2.0

System Info
Item Value
CPUs undefined
GPU Status 2d_canvas: enabled
gpu_compositing: enabled
metal: disabled_off
multiple_raster_threads: enabled_on
oop_rasterization: enabled
opengl: enabled_on
protected_video_decode: unavailable_off
rasterization: enabled
skia_renderer: disabled_off_ok
video_decode: enabled
webgl: enabled
webgl2: enabled
Load (avg) 2, 3, 3
Memory (System) 8.00GB (0.17GB free)
Process Argv --crash-reporter-id ea63df4b-e690-4545-96f3-6944dca865ad
Screen Reader no
VM 0%
Extensions (58)
Extension Author (truncated) Version
better-comments aar 2.1.0
vscode-base64 ada 0.1.0
vsc-rename-files alf 2.1.0
ng-template Ang 0.1100.2
vscode-color ans 0.4.5
vscode-apollo apo 1.18.0
vscode-zipfs arc 2.2.2
folder-source-actions bie 0.2.0
markdown-preview-github-styles bie 0.1.6
path-intellisense chr 2.3.0
react-toolkit Cod 1.1.23
angular-schematics cyr 4.7.0
vscode-eslint dba 2.1.14
emulate Die 1.4.0
es7-react-js-snippets dsz 3.1.0
gitlens eam 11.1.0
tsl-problem-matcher eam 0.3.1
prettier-vscode esb 5.8.0
vscode-todo-plus fab 4.18.0
flow-for-vscode flo 1.5.0
vscode-yarn gam 1.7.1
beautify Hoo 1.5.0
vscode-edit-csv jan 0.5.3
Angular2 joh 11.0.0
vscode-peacock joh 3.9.1
chat kar 0.35.0
graphql-for-vscode kum 1.15.3
vscode-wsl-workspacefolder lfu 1.1.2
ftp-sync luk 0.3.9
auto-barrel mik 1.7.1
dotenv mik 1.0.1
mongodb-vscode mon 0.3.0
vscode-csscomb mrm 5.3.2
vscode-docker ms- 1.9.0
vscode-edge-devtools ms- 1.1.1
remote-wsl ms- 0.52.0
vscode-js-profile-flame ms- 0.0.13
vsliveshare ms- 1.0.3375
vsliveshare-audio ms- 0.1.91
vsliveshare-pack ms- 0.4.0
debugger-for-chrome msj 4.12.11
vscode-react-native msj 1.2.0
angular2-inline nat 0.0.17
vscode-ios-common-files Ort 1.0.5
vscode-react-native-storybooks Ort 2.7.1
format-code-action roh 0.0.1
vscode-coverage-gutters rya 2.6.0
vscode-hexdump sle 1.8.1
move-ts str 1.12.0
theshukran-react-utils the 1.0.2
gitflow vec 1.2.1
es-quotes vil 0.2.6
vscodeintellicode Vis 1.2.10
vscode-wakatime Wak 4.0.10
vscode-html-pug-convertor way 0.1.1
vscode-ruby win 0.27.0
code-template-tool yua 0.6.2
sort-js-object-keys zen 1.0.6

(2 theme extensions excluded)

A/B Experiments
vsliv695:30137379
vsins829:30139715
vsliv368:30146709
vsreu685:30147344
openlogontheside:30221882
python763:30178808
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
python504:30227505
vswsl492:30208929
wsl2promptcf:30219163
vstry914:30230485
pythonvsdeb440:30224570
unusedpromptcf:30219165
folderexplorercf:30219167
openfilemenucf:30219169
pythonvsded773:30223139
core-portspanel:30233467

@andreialecu
Copy link
Author

Possibly relevant, this is also in the developer tools log:

 WARN [File Watcher (chokidar)] Watcher is not using native fsevents library and is falling back to unefficient polling.

@andreialecu
Copy link
Author

Also these from when Extension Host actually crashes:

Screenshot 2020-12-25 at 10 22 39

@andreialecu andreialecu changed the title Insiders on macOS ARM64: Error: read EFAULT Insiders on macOS ARM64: Extension Host crashes (SIGILL, EFAULT, EPIPE) Dec 25, 2020
@andreialecu
Copy link
Author

andreialecu commented Dec 26, 2020

One other error I've been seeing in developer tools prior to an Extension Host crash:

ERR Request textDocument/codeAction failed with message: Cannot read .eslintignore file: /Users/andreialecu/Work/proj/.eslintignore
Error: EFAULT: bad address in system call argument, read: Error: Request textDocument/codeAction failed with message: Cannot read .eslintignore file: /Users/andreialecu/Work/proj/.eslintignore
Error: EFAULT: bad address in system call argument, read
	at /Users/andreialecu/.vscode-insiders/extensions/dbaeumer.vscode-eslint-2.1.14/client/out/extension.js:1:52031
	at /Users/andreialecu/.vscode-insiders/extensions/dbaeumer.vscode-eslint-2.1.14/client/out/extension.js:1:52325
	at Immediate.<anonymous> (/Users/andreialecu/.vscode-insiders/extensions/dbaeumer.vscode-eslint-2.1.14/client/out/extension.js:1:52690)
	at processImmediate (internal/timers.js:456:21)

@andreialecu
Copy link
Author

New one:

[Extension Host] Error: Illegal argument    at Object.t.illegalArgument (file:///Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:50:803)    at file:///Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1705:214    at d.invokeFunction (file:///Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1759:614)    at u._tryExecuteCommand (file:///Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:4214:45)    at file:///Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:4213:814 (at _executeContributedCommand (/Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/services/extensions/node/extensionHostProcess.js:707:780))
t.log @ /Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:336

@andreialecu
Copy link
Author

Possibly relevant, this is also in the developer tools log:

 WARN [File Watcher (chokidar)] Watcher is not using native fsevents library and is falling back to unefficient polling.

Likely relevant for this issue: fsevents/fsevents#349

@andreialecu
Copy link
Author

andreialecu commented Jan 4, 2021

@deepak1556 if you need any further information kindly let me know.

I'm curious about one thing
Screenshot 2021-01-04 at 16 03 39

This seems to specify that it's running on node 12, but node 12 isn't compatible with arm64 (at least not the general purpose one). Only node 15 is.

Is this right?

Also, fsevents/fsevents#350 has been merged and pending release as fsevents 2.3.0 which will hopefully happen today, and might help with the chokidar issue.

@deepak1556
Copy link
Contributor

Can you run the arm64 insiders in a new user data dir --user-data-dir <some-absolute-path> and see if the crash continues.

This seems to specify that it's running on node 12, but node 12 isn't compatible with arm64 (at least not the general purpose one). Only node 15 is.

The runtime is patched to work in electron.

@deepak1556 deepak1556 added 🍎 si Issues related to apple silicon info-needed Issue requires more information from poster labels Jan 4, 2021
@andreialecu
Copy link
Author

andreialecu commented Jan 5, 2021

@deepak1556 I ran:

code-insiders --user-data-dir `mktemp -d` .

Then had to use Reload Window from the command pallette due to some issues with extensions.

Still got the crash:

(hopefully Reload Window didn't mess with the data dir)

Screenshot 2021-01-05 at 11 26 19

Note that it happens randomly, I've seen it mostly happen when the angular app I'm running in the integrated terminal detects a code change and starts compiling. However, it can happen without any integrated terminal tasks running as well.

What I've been doing and helps a bit, was to run Workspaces: Duplicate in new window and run my Run task there (which spawns about 3 nodejs processes).

It's possible it is related to file change detection though. In this particular crash above, it crashed immediately after saving a file, and when angular started recompiling (running via ng serve).

@andreialecu
Copy link
Author

andreialecu commented Jan 5, 2021

Screenshot 2021-01-05 at 11 45 13

Also, at times, this pops up when saving a file and either: a) takes a long time, b) doesn't finish at all (waiting for minutes). I have to cancel it and Reload Window.

While this one was spinning, there was nothing relevant logged in the Developer tools, no warnings or errors.

@andreialecu
Copy link
Author

Attaching the CPU profile of the vscode.git warning, which has been popping up repeatedly. git integration definitely seems slow, but not sure if it's git itself being slow or something else.

exthost-840d83.cpuprofile.zip

@deepak1556
Copy link
Contributor

Can you also try to remove the .vscode-insiders folder under home directory and see if the extension host crashes.

As for the git integration slowdown, its triggered by child_process.spawn calls based on your profile and the root issue is tracked in upstream electron/electron#26143

@andreialecu
Copy link
Author

I have disabled the Live Share Extension Pack which I wasn't really using and it seems that things are a lot better. I noticed it was taking part in the cpuprofile attached previously. 🤔

Not sure if this fixes it yet, but there haven't been any crashes in a while.

@andreialecu
Copy link
Author

Can you also try to remove the .vscode-insiders folder under home directory and see if the extension host crashes.

FWIW, both VSCode Exploration and Insiders exhibit the problem, and I installed both fresh arm64 on a fresh macOS installation. This was a new Mac, so I didn't restore from anything.

The crashes have been super consistent for me, at least once every 10 minutes or so. If no one else reported this, it's probably a combination of the electron issue you linked and some extensions I'm using.

I'll keep monitoring whether disabling Live Share helped and report back.

@deepak1556
Copy link
Contributor

If the crash continues, can you start the app with --crash-reporter-directory <some-absolute-path>, crash dumps will be generated in that location, please attach it here. Thanks!

@andreialecu
Copy link
Author

andreialecu commented Jan 5, 2021

I've been trying real hard for the past hour to make it crash since disabling the Live Share pack, and I wasn't able to.

I assume the pack installs a code lens provider that runs git when hovering certain things in the editor. This seems somewhat inefficient on its part, besides the libuv issue you mentioned.

The performance, prior to disabling it, was horrible overall. I can't believe how smooth it is now. Thanks for pointing out that bug.

Just a heads up that fsevents@2.3.0 is out now with the arm64/x86_64 universal binary. I'm not sure what chokidar is used for, but perhaps bumping fsevents might fix the warning and improve performance as well.

@deepak1556
Copy link
Contributor

Just a heads up that fsevents@2.3.0 is out now with the arm64/x86_64 universal binary. I'm not sure what chokidar is used for, but perhaps bumping fsevents might fix the warning and improve performance as well.

Thanks for the work! chokidar just released a version with it. Will update shortly.

@andreialecu
Copy link
Author

andreialecu commented Jan 6, 2021

If the crash continues, can you start the app with --crash-reporter-directory <some-absolute-path>, crash dumps will be generated in that location, please attach it here. Thanks!

It looks like eventually it still crashed, even without the live share extension pack. It took way longer though. 🤔

Screenshot 2021-01-06 at 10 46 00

Going to run with crash dump reporting now.


Still seeing this logged btw:

   ERR Request textDocument/codeAction failed with message: Cannot read config file: /Users/andreialecu/Work/x/x-monorepo/packages/x-shared/package.json
Error: EFAULT: bad address in system call argument, read: Error: Request textDocument/codeAction failed with message: Cannot read config file: /Users/andreialecu/Work/x/x-monorepo/packages/x-shared/package.json
Error: EFAULT: bad address in system call argument, read
	at /Users/andreialecu/.vscode-insiders/extensions/dbaeumer.vscode-eslint-2.1.14/client/out/extension.js:1:52031
	at /Users/andreialecu/.vscode-insiders/extensions/dbaeumer.vscode-eslint-2.1.14/client/out/extension.js:1:52325
	at Immediate.<anonymous> (/Users/andreialecu/.vscode-insiders/extensions/dbaeumer.vscode-eslint-2.1.14/client/out/extension.js:1:52690)
	at processImmediate (internal/timers.js:456:21)

Have seen it a bunch of times, always related to dbaeumer.vscode-eslint.

@andreialecu
Copy link
Author

@deepak1556 I just had another crash of the Extension Host, this time with a new code. SIGSEGV instead of SIGILL (first time I'm seeing that):

Screenshot 2021-01-06 at 15 01 11

Attaching crash dump:

09b42d44-5dd7-42ae-b463-3342d4c0e9aa.dmp.zip

That file was in the pending directory. The other directories were empty. Hopefully this is the right one.

@andreialecu
Copy link
Author

Attaching a dump from a crash with code SIGILL this time.

ecc52cbe-9a56-4e06-828f-93fad3721831.dmp.zip

@andreialecu
Copy link
Author

@deepak1556
Copy link
Contributor

That file was in the pending directory. The other directories were empty. Hopefully this is the right one.

Yes thats the correct directory to look for dumps, symbolicated trace for SIGSEGV crash

Crash reason:  EXC_BAD_ACCESS / KERN_INVALID_ADDRESS
Crash address: 0x110
Process uptime: 14992 seconds

Thread 35 (crashed)
 0  Electron Framework!node::inspector::TcpHolder::WriteRaw(std::__1::vector<char, std::__1::allocator<char> > const&, void (*)(uv_write_s*, int)) [inspector_socket.cc : 679 + 0x0]
     x0 = 0x00000001389dbe40    x1 = 0x000000000000000f
     x2 = 0x00000001389dbf30    x3 = 0x0000000000000007
     x4 = 0x00000001389f7220    x5 = 0x0000000000000001
     x6 = 0x0000000000000000    x7 = 0x0000000177021d7c
     x8 = 0x0000000000000000    x9 = 0x000000000000a007
    x10 = 0x0000000001040800   x11 = 0x0000000000040000
    x12 = 0x0000000000040000   x13 = 0x0000000001040800
    x14 = 0x0000000092b1b743   x15 = 0x0000000000007756
    x16 = 0x0000000000000d7f   x17 = 0x0000000000040000
    x18 = 0x0000000000000000   x19 = 0x00000001389dbe40
    x20 = 0x0000000102b8644c   x21 = 0x00000001389c9e00
    x22 = 0x0000000000000000   x23 = 0x0000000177022210
    x24 = 0x0000000000000269   x25 = 0x0000000137e90480
    x26 = 0x0000000127f2bb80   x27 = 0x0000000000000000
    x28 = 0x0000000137e90480    fp = 0x00000001770221f0
     lr = 0x0000000102b84e10    sp = 0x00000001770221b0
     pc = 0x0000000102b84e14
    Found by: given as instruction pointer in context
 1  Electron Framework!node::inspector::TcpHolder::WriteRaw(std::__1::vector<char, std::__1::allocator<char> > const&, void (*)(uv_write_s*, int)) [inspector_socket.cc : 679 + 0x4]
     fp = 0x0000000177022260    lr = 0x0000000102b86914
     sp = 0x0000000177022200    pc = 0x0000000102b84e10
    Found by: previous frame's frame pointer
 2  Electron Framework!node::inspector::(anonymous namespace)::WsHandler::Write(std::__1::vector<char, std::__1::allocator<char> >) [inspector_socket.cc : 612 + 0xc]
     fp = 0x00000001770222b0    lr = 0x0000000102b85414
     sp = 0x0000000177022270    pc = 0x0000000102b86914
    Found by: previous frame's frame pointer
 3  Electron Framework!node::inspector::InspectorSocket::Write(char const*, unsigned long) [inspector_socket.cc : 765 + 0x10]
     fp = 0x0000000177022360    lr = 0x0000000102b7f6dc
     sp = 0x00000001770222c0    pc = 0x0000000102b85414
    Found by: previous frame's frame pointer
 4  Electron Framework!node::inspector::(anonymous namespace)::RequestQueueData::RequestQueueData(uv_loop_s*)::'lambda'(uv_async_s*)::__invoke(uv_async_s*) [inspector_io.cc : 85 + 0xc]
     fp = 0x00000001770227c0    lr = 0x00000001003b202c
     sp = 0x0000000177022370    pc = 0x0000000102b7f6dc
    Found by: previous frame's frame pointer
 5  Electron Framework!uv__async_io [async.c : 163 + 0x4]
     fp = 0x000000017702a8a0    lr = 0x00000001003c23c8
     sp = 0x00000001770227d0    pc = 0x00000001003b202c
    Found by: previous frame's frame pointer
 6  Electron Framework!uv__io_poll [kqueue.c : 0 + 0x8]
     fp = 0x000000017702a920    lr = 0x00000001003b2468
     sp = 0x000000017702a8b0    pc = 0x00000001003c23c8
    Found by: previous frame's frame pointer
 7  Electron Framework!uv_run [core.c : 381 + 0x4]
     fp = 0x000000017702afc0    lr = 0x0000000102b7eeec
     sp = 0x000000017702a930    pc = 0x00000001003b2468
    Found by: previous frame's frame pointer
 8  Electron Framework!node::inspector::InspectorIo::ThreadMain() [inspector_io.cc : 321 + 0x8]
     fp = 0x000000017702afe0    lr = 0x000000019c75506c
     sp = 0x000000017702afd0    pc = 0x0000000102b7eeec
    Found by: previous frame's frame pointer
 9  libsystem_pthread.dylib!_pthread_start + 0x13c
     fp = 0x0000000000000000    lr = 0x3f4100019c74fda0
     sp = 0x000000017702aff0    pc = 0x000000019c75506c
    Found by: previous frame's frame pointer

this would be nodejs/node#34833 , that issue reports its fixed in v14. We can ignore this for now.

@deepak1556
Copy link
Contributor

As for the SIGILL dumps,

Crash reason:  0x00000000 / 0x00000000
Crash address: 0x19c7230f8
Process uptime: 9411 seconds

Thread 0 (crashed)
 0  libsystem_kernel.dylib!__fork + 0x14
     x0 = 0x000000000000a0e3    x1 = 0x0000000000000000
     x2 = 0x0000000000000303    x3 = 0x000000000000002c
     x4 = 0x0000000000000507    x5 = 0x0000000000000000
     x6 = 0x0000000000000000    x7 = 0x0000002800000000
     x8 = 0x0000000000000000    x9 = 0x0000000104677e20
    x10 = 0x00000001fc650338   x11 = 0x0000000000087bbb
    x12 = 0x0000000000000002   x13 = 0x0000000000000000
    x14 = 0x0000000000000000   x15 = 0x0000000000000001
    x16 = 0x0000000000000002   x17 = 0x00000002021398c0
    x18 = 0x0000000000000000   x19 = 0x000000016b9545c8
    x20 = 0x000000016b954500   x21 = 0x0000000000000003
    x22 = 0x000000010b3f6e90   x23 = 0x000000016b954518
    x24 = 0x0000000000000003   x25 = 0x0000000000000000
    x26 = 0x000000010b3f6fe0   x27 = 0x0000000000000003
    x28 = 0x000000010a6a1154    fp = 0x000000016b9544b0
     lr = 0xff7a80019c6355bc    sp = 0x000000016b9544b0
     pc = 0x000000019c7230f8
    Found by: given as instruction pointer in context
 1  0xff7a80019c6355b8
     fp = 0x000000016b9544d0    lr = 0xff7a80019c6355bc
     sp = 0x000000016b9544c0    pc = 0xff7a80019c6355bc
    Found by: previous frame's frame pointer
 2  0xff7a80019c6355b8
     fp = 0x000000016b9545a0    lr = 0xaa468001046d4e90
     sp = 0x000000016b9544e0    pc = 0xff7a80019c6355bc
    Found by: previous frame's frame pointer
 3  0xaa468001046d4e8c
     fp = 0x000000016b9552b0    lr = 0x0000000106e757c8
     sp = 0x000000016b9545b0    pc = 0xaa468001046d4e90
    Found by: previous frame's frame pointer
 4  Electron Framework!node::(anonymous namespace)::ProcessWrap::Spawn(v8::FunctionCallbackInfo<v8::Value> const&) [process_wrap.cc : 258 + 0x4]
     fp = 0x000000016b955390    lr = 0x0000000104faa6f8
     sp = 0x000000016b9552c0    pc = 0x0000000106e757c8
    Found by: previous frame's frame pointer
 5  Electron Framework!v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo) [api-arguments-inl.h : 158 + 0x4]
     fp = 0x000000016b955450    lr = 0x0000000104faa4b8
     sp = 0x000000016b9553a0    pc = 0x0000000104faa6f8
    Found by: previous frame's frame pointer
 6  Electron Framework!v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) [builtins-api.cc : 111 + 0x8]
     fp = 0x000000016b9554b0    lr = 0x0000000104fa9c78
     sp = 0x000000016b955460    pc = 0x0000000104faa4b8
    Found by: previous frame's frame pointer
 7  Electron Framework!v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) [builtins-api.cc : 141 + 0x14]
     fp = 0x000000016b9554f0    lr = 0x000000010557276c
     sp = 0x000000016b9554c0    pc = 0x0000000104fa9c78
    Found by: previous frame's frame pointer
 8  Electron Framework!Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit + 0x68
     fp = 0x000000016b9555d0    lr = 0x00000001055028f4
     sp = 0x000000016b955500    pc = 0x000000010557276c
    Found by: previous frame's frame pointer
 9  Electron Framework!Builtins_InterpreterEntryTrampoline + 0x110
     fp = 0x000000016b955650    lr = 0x00000001055028f4
     sp = 0x000000016b9555e0    pc = 0x00000001055028f4
    Found by: previous frame's frame pointer
10  Electron Framework!Builtins_InterpreterEntryTrampoline + 0x110
     fp = 0x000000016b9556f0    lr = 0x00000001055028f4
     sp = 0x000000016b955660    pc = 0x00000001055028f4
    Found by: previous frame's frame pointer
11  Electron Framework!Builtins_InterpreterEntryTrampoline + 0x110
     fp = 0x000000016b955730    lr = 0x00000001054f9a38
     sp = 0x000000016b955700    pc = 0x00000001055028f4
    Found by: previous frame's frame pointer
12  Electron Framework!Builtins_ArgumentsAdaptorTrampoline + 0xd4
     fp = 0x000000016b955820    lr = 0x00000001055028f4
     sp = 0x000000016b955740    pc = 0x00000001054f9a38
    Found by: previous frame's frame pointer
13  Electron Framework!Builtins_InterpreterEntryTrampoline + 0x110
     fp = 0x000000016b955860    lr = 0x00000001054f9a38
     sp = 0x000000016b955830    pc = 0x00000001055028f4
    Found by: previous frame's frame pointer
14  Electron Framework!Builtins_ArgumentsAdaptorTrampoline + 0xd4
     fp = 0x000000016b955940    lr = 0x00000001055028f4
     sp = 0x000000016b955870    pc = 0x00000001054f9a38
    Found by: previous frame's frame pointer
15  Electron Framework!Builtins_InterpreterEntryTrampoline + 0x110
     fp = 0x000000016b955990    lr = 0x00000001054f9a38
     sp = 0x000000016b955950    pc = 0x00000001055028f4
    Found by: previous frame's frame pointer
16  Electron Framework!Builtins_ArgumentsAdaptorTrampoline + 0xd4
     fp = 0x000000016b955a10    lr = 0x00000001055028f4
     sp = 0x000000016b9559a0    pc = 0x00000001054f9a38
    Found by: previous frame's frame pointer
17  Electron Framework!Builtins_InterpreterEntryTrampoline + 0x110
     fp = 0x000000016b955a80    lr = 0x00000001055028f4
     sp = 0x000000016b955a20    pc = 0x00000001055028f4
    Found by: previous frame's frame pointer
18  Electron Framework!Builtins_InterpreterEntryTrampoline + 0x110
     fp = 0x000000016b955ac0    lr = 0x00000001055c245c
     sp = 0x000000016b955a90    pc = 0x00000001055028f4
    Found by: previous frame's frame pointer
19  Electron Framework!Builtins_PromiseFulfillReactionJob + 0x38
     fp = 0x000000016b955b20    lr = 0x0000000105523920
     sp = 0x000000016b955ad0    pc = 0x00000001055c245c
    Found by: previous frame's frame pointer
20  Electron Framework!Builtins_RunMicrotasks + 0x29c
     fp = 0x000000016b955b68    lr = 0x00000001054fff48
     sp = 0x000000016b955b30    pc = 0x0000000105523920
    Found by: previous frame's frame pointer
21  Electron Framework!Builtins_JSRunMicrotasksEntry + 0xa4
     fp = 0xffffffffffffffff    lr = 0x0000298c0000a6d8
     sp = 0x000000016b955b78    pc = 0x00000001054fff48
    Found by: previous frame's frame pointer
22  Electron Framework!Builtins_JSConstructEntry + 0x11c
     sp = 0x000000016b955b98    pc = 0x00000001054ffea0
    Found by: stack scanning
23  Electron Framework!v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [simulator.h : 142 + 0x4]
     sp = 0x000000016b955bd0    pc = 0x0000000104ffff5c
    Found by: stack scanning
24  Electron Framework!v8::internal::LoadIC::Load(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Name>, bool, v8::internal::Handle<v8::internal::Object>) [ic.cc : 477 + 0x4]
     sp = 0x000000016b955c30    pc = 0x00000001050b8534
    Found by: stack scanning
25  Electron Framework!v8::internal::Runtime_LoadIC_Miss(int, unsigned long*, v8::internal::Isolate*) [ic.cc : 2343 + 0x14]
     sp = 0x000000016b955d00    pc = 0x00000001050c0634
    Found by: stack scanning
26  Electron Framework!v8::internal::Runtime_DefineDataPropertyInLiteral(int, unsigned long*, v8::internal::Isolate*) [runtime-object.cc : 953 + 0x14]
     sp = 0x000000016b955d10    pc = 0x0000000105231084
    Found by: stack scanning
27  Electron Framework!v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [execution.cc : 428 + 0x8]
     sp = 0x000000016b955da0    pc = 0x0000000105000424
    Found by: stack scanning
28  Electron Framework!Builtins_InterpreterEntryTrampoline + 0xfe
     sp = 0x000000016b955dd0    pc = 0x00000001055028e2
    Found by: stack scanning
29  Electron Framework!v8::internal::Execution::TryRunMicrotasks(v8::internal::Isolate*, v8::internal::MicrotaskQueue*, v8::internal::MaybeHandle<v8::internal::Object>*) [execution.cc : 505 + 0x4]
     sp = 0x000000016b955e00    pc = 0x00000001050004ec
    Found by: stack scanning
30  Electron Framework!v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*) [microtask-queue.cc : 165 + 0xc]
     sp = 0x000000016b955e60    pc = 0x0000000105015128
    Found by: stack scanning
31  Electron Framework!Builtins_InterpreterEntryTrampoline + 0x110
     sp = 0x000000016b955e80    pc = 0x00000001055028f4
    Found by: stack scanning
32  Electron Framework!Builtins_InterpreterEntryTrampoline + 0x110
     sp = 0x000000016b955ef0    pc = 0x00000001055028f4
    Found by: stack scanning
33  Electron Framework!v8::internal::MicrotaskQueue::PerformCheckpoint(v8::Isolate*) [microtask-queue.cc : 117 + 0x0]
     sp = 0x000000016b955f70    pc = 0x0000000105014f50
    Found by: stack scanning
34  Electron Framework!Builtins_CallApiCallback + 0xd0
     sp = 0x000000016b955f90    pc = 0x0000000105503bb4
    Found by: stack scanning
35  Electron Framework!Builtins_CallApiCallback + 0xd0
     sp = 0x000000016b955f98    pc = 0x0000000105503bb4
    Found by: stack scanning
36  Electron Framework!Builtins_JSEntryTrampoline + 0xa8
     sp = 0x000000016b9560a0    pc = 0x000000010550006c
    Found by: stack scanning
37  Electron Framework!Builtins_JSEntry + 0xa4
     sp = 0x000000016b9560d0    pc = 0x00000001054ffd08
    Found by: stack scanning
38  Electron Framework!Builtins_ConstructProxy + 0x39c
     sp = 0x000000016b956130    pc = 0x00000001054ffc60
    Found by: stack scanning
39  Electron Framework!v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [simulator.h : 142 + 0x14]
     sp = 0x000000016b956170    pc = 0x0000000104ffffd0
    Found by: stack scanning
40  Electron Framework!v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [simulator.h : 142 + 0x14]
     sp = 0x000000016b9561e0    pc = 0x0000000104ffffd0
    Found by: stack scanning
41  Electron Framework!v8::internal::LookupIterator::Delete() [lookup.cc : 669 + 0xc]
     sp = 0x000000016b9562a0    pc = 0x0000000105174148
    Found by: stack scanning
42  Electron Framework!v8::internal::Object::AddDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::Maybe<v8::internal::ShouldThrow>, v8::internal::StoreOrigin) [objects.cc : 2816 + 0xc]
     sp = 0x000000016b9562b0    pc = 0x0000000105191864
    Found by: stack scanning
43  Electron Framework!v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) [execution.cc : 462 + 0x4]
     sp = 0x000000016b956340    pc = 0x0000000104fff4f8
    Found by: stack scanning
44  Electron Framework!v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) [api.cc : 4994 + 0x14]
     sp = 0x000000016b9563d0    pc = 0x0000000104f86f48
    Found by: stack scanning
45  Electron Framework!node::InternalCallbackScope::Close() [callback.cc : 144 + 0x8]
     sp = 0x000000016b9564f0    pc = 0x0000000106d5b6ec
    Found by: stack scanning
46  Electron Framework!node::InternalMakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context) [callback.cc : 197 + 0x4]
     sp = 0x000000016b956560    pc = 0x0000000106d5ba68
    Found by: stack scanning
47  Electron Framework!v8::internal::JSArrayBuffer::Attach(std::__1::shared_ptr<v8::internal::BackingStore>) [js-array-buffer.cc : 113 + 0x4]
     sp = 0x000000016b9565b0    pc = 0x0000000105140bc0
    Found by: stack scanning
48  Electron Framework!v8::internal::JSArrayBuffer::Setup(v8::internal::SharedFlag, std::__1::shared_ptr<v8::internal::BackingStore>) [js-array-buffer.cc : 51 + 0x8]
     sp = 0x000000016b9565f0    pc = 0x0000000105140af8
    Found by: stack scanning
49  Electron Framework!node::AsyncWrap::MakeCallback(v8::Local<v8::Function>, int, v8::Local<v8::Value>*) [async_wrap.cc : 769 + 0x1c]
     sp = 0x000000016b956690    pc = 0x0000000106d6c4c0
    Found by: stack scanning
50  Electron Framework!node::StreamBase::CallJSOnreadMethod(long, v8::Local<v8::ArrayBuffer>, unsigned long, node::StreamBase::StreamBaseJSChecks) [stream_base.cc : 346 + 0x10]
     sp = 0x000000016b9566f0    pc = 0x0000000106e7a02c
    Found by: stack scanning
51  Electron Framework!node::EmitToJSStreamListener::OnStreamRead(long, uv_buf_t const&) [stream_base.cc : 0 + 0x10]
     sp = 0x000000016b956720    pc = 0x0000000106e7bed8
    Found by: stack scanning
52  Electron Framework!node::LibuvStreamWrap::OnUvRead(long, uv_buf_t const*) [stream_base-inl.h : 126 + 0x10]
     sp = 0x000000016b956790    pc = 0x0000000106e80710
    Found by: stack scanning
53  Electron Framework!node::LibuvStreamWrap::ReadStart()::$_1::__invoke(uv_handle_s*, unsigned long, uv_buf_t*) [stream_wrap.cc : 218 + 0x4]
     sp = 0x000000016b9567c0    pc = 0x0000000106e80ed0
    Found by: stack scanning
54  Electron Framework!uv__stream_io [stream.c : 1239 + 0xc]
     sp = 0x000000016b956810    pc = 0x00000001046d6f80
    Found by: stack scanning

@deepak1556
Copy link
Contributor

Both the dumps indicate the crash happens after call to fork. I am not sure what the cause is here, but it would be a side effect of having all those child process spawn calls being delayed. In your case two extensions definitely standout from the devtools log, can you disable vscode.git and eamodio.gitlens ?

@deepak1556 deepak1556 added freeze-slow-crash-leak VS Code crashing, performance, freeze and memory leak issues and removed info-needed Issue requires more information from poster labels Jan 6, 2021
@andreialecu
Copy link
Author

andreialecu commented Feb 10, 2021

The EFAULT errors don't produce any logs that I can see. And SIGILL has only happened that one time, but it's concerning because it means it still exists. It also seems to be really hard to run into on the 16GB M1 (vs the 8GB where I got it extremely frequently)

Do you think there's any way to get it to produce a stack trace or more useful logs for the EFAULT errors?

I can reproduce those relatively easy under memory pressure.

(PS: I will run with --crash-reporter-directory enabled all the time now)

@vassudanagunta
Copy link

vassudanagunta commented Feb 10, 2021

The warning disappeared, but the extension host reports the same crash as in the screenshot I attached earlier.

@andreialecu I also have a crazy-ish idea. What if you disable JIT by passing --js-flags=--jitless flag?

No difference. Same error as above, and an additional one which seems to confirm my previous comment

@andreialecu maybe because the setting isn't getting picked up by the extension. All my problems are fixed, but running my code and tests via bash command line.

/cc @huming2207

@huming2207
Copy link

@vassudanagunta I've tested it. Some VSCode plugins have WebAssembly stuff in there. If you set V8 in VSCode to JIT-less, the WS related API will be disabled. So it's not an option here.

@deepak1556
Copy link
Contributor

@vassudanagunta @huming2207 there should be no wasm related crashes with latest insiders, this issue is mainly tracking a crash under memory pressure.

@huming2207
Copy link

@deepak1556 Yes I agree. I think this is off-topic as it's related to the permission issue nodejs/node#37061 only.

@richiksc
Copy link

node v15.9.0 has been released with the fix for nodejs/node#37061.

@deepak1556 deepak1556 removed the info-needed Issue requires more information from poster label Feb 24, 2021
@deepak1556 deepak1556 added this to the March 2021 milestone Feb 24, 2021
@deepak1556
Copy link
Contributor

@andreialecu the applications with which you see EFAULT errors do they use wasm modules ? Is there a sample test case I can try. Thanks!

@andreialecu
Copy link
Author

@deepak1556 I haven't really seen EFAULT in the wild in nodejs apps lately. Actually not at all since upgrading to the 16GB M1 like previously mentioned.

But I have been able to reproduce EFAULT fairly consistently even on the 16GB M1 under memory pressure in VSCode, via the steps in: #113410 (comment)

Unfortunately I don't have further info besides what people in nodejs/node#36826 have been posting.

Overall Insiders has been pretty stable lately. I did not see any additional SIGILL crashes besides that single one 14 days ago.

But again, I suspect these errors may be more prevalent on the 8GB of RAM M1.

If I open Developer Tools I see some random errors logged there most of the time. But things seem to be fine.

Here's one example, but I'm not sure if these are related to this particular issue:

Screenshot 2021-02-24 at 22 18 37

@deepak1556
Copy link
Contributor

Thanks for the details, that helps.

@purge
Copy link

purge commented Mar 9, 2021

I'm experiencing this regularly on the stable release (I was previously insiders)

Version: 1.54.1 (Universal)
Commit: f30a9b7
Date: 2021-03-04T22:42:18.719Z (4 days ago)
Electron: 11.3.0
Chrome: 87.0.4280.141
Node.js: 12.18.3
V8: 8.7.220.31-electron.0
OS: Darwin arm64 20.3.0

I will provide a stack trace when it happens again if thats useful

@andreialecu
Copy link
Author

andreialecu commented Mar 9, 2021

@purge what's the RAM capacity of your M1? And which error, SIGILL?

@purge
Copy link

purge commented Mar 10, 2021

@andreialecu it was EFAULT (I do experience SIGILLs in node though) i have 16Gb and it definitely appears to be related to memory pressure.

@purge
Copy link

purge commented Mar 11, 2021

this happened again today with SIGILL. It may just be nodejs/node#36826
so out of your control.

/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1857 Extension host terminated unexpectedly. Code:  null  Signal:  SIGILL

@andreialecu
Copy link
Author

@purge see: #113410 (comment)

The command I was using was: code-insiders --crash-reporter-directory ~/vscode-crash ..

I haven't personally seen any SIGILLs in a long time, but if they happen frequently for you, it would likely help a lot if you can attach the crash dumps.

@joaomoreno joaomoreno added the bug Issue identified by VS Code Team member as probable bug label Mar 26, 2021
@deepak1556 deepak1556 removed this from the March 2021 milestone Mar 26, 2021
@alexdima alexdima added the extension-host Extension host issues label Apr 28, 2021
@bpasero
Copy link
Member

bpasero commented Apr 28, 2021

Lately I have seen a crash of the extension host on macOS 11.x ARM almost daily.

@bl-ue
Copy link

bl-ue commented Jun 10, 2021

Have we determined that it's memory related? Or is it something like network sockets timing out/disconnecting/becoming invalidated?

@bpasero
Copy link
Member

bpasero commented Jun 10, 2021

Since I posted in April, I have not seen a single extension host crash ever since. I am still using macOS ARM.

@purge
Copy link

purge commented Jun 10, 2021

Agreed, i have not seen it for months either. not sure whether its was node or macos update that fixed it.

@deepak1556
Copy link
Contributor

Is anyone in this thread actively experiencing this issue on latest macOS Big Sur 11.4 and VS Code version 1.57.1 ?

@andreialecu
Copy link
Author

andreialecu commented Jun 29, 2021

@deepak1556 There has been some activity here nodejs/node#36826 (comment) and a report of a SIGSEGV with the extension host under high memory pressure.

Luckily a nodejs contributor is looking into it, so perhaps we'll have some answers soon.

Personally, I haven't seen almost any crashes in a long time. I use VSCode 8+ hrs per day and in the past 3 months or so I only got one extension host crash or maybe two. (It might still happen more often on a 8GB machine though.)

Edit: just noticed you were involved in that same discussion, and the conclusion was that macOS 11.4 might have fixed it.

@deepak1556
Copy link
Contributor

Yup I am now looking for users who are hitting this issue on 11.4 to validate the nodejs issue findings. I will give this issue some time before closing as fixed.

@deepak1556
Copy link
Contributor

Closing per above comment.

@github-actions github-actions bot locked and limited conversation to collaborators Aug 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
🍎 si Issues related to apple silicon bug Issue identified by VS Code Team member as probable bug extension-host Extension host issues freeze-slow-crash-leak VS Code crashing, performance, freeze and memory leak issues
Projects
None yet
Development

No branches or pull requests