Docker crashes when renderer process eats up too much memory

[brian-mann]

Related to #348 and #349.

When running headlessly on very long and memory intense applications we are seeing renderer crashes with Docker.

[brian-mann]

This is actually not indicative of a memory leak inside of Cypress (nor your application). This has to do with the way that browsers work under the hood.

Luckily, we have a found a simple one line fix for this problem - simply add the flag --ipc=host.

This option is documented here.

The Slack app and Atom has also been documented to crash here, here, and here for the exact same reasons.

If you are using docker run

docker run --ipc=host

If you are using docker compose

version: '2'
services:
  foobarbazservice:
    ipc: host ## this line right here

In the future we are working on a more permanent fix such as described in #349 - either to automatically recover from these crashes, or mostly prevent them up from by nuking the renderer process in between spec files.

[jheijkoop]

I seem to be getting this problem too, because Chrome is running out of shared memory (/dev/shm). By default docker starts images with a 64M /dev/shm (try running df -h in you instance). To change this you can supply docker with an extra argument: docker run --shm-size 512M my-image. Because we are working with mesos/marathon I had to do this via the "docker": { "parameters": [{"key": "shm-size", "value": "512M"}], ...} (https://mesosphere.github.io/marathon/docs/native-docker.html) in the json configuration when creating the app/instances.

[einomi]

I had the same issue with Codeship CI. Just needed to change config in my codeship-services.yml file, according to documentation https://documentation.codeship.com/pro/continuous-integration/browser-testing/#chrome-crashing

[jennifer-shehane]

I wonder if this is related to our other issue with a Codeship CI run failing.#328 Unfortunately, we don't have a pro account, and the codeship-services.yml is only available on pro.

[egucciar]

any advice on how to solve for this in CircleCI? I do not have a services section in my config.yml, only a jobs.

[mitar]

I think that since this issue has been made there is now a better fix for the problem by asking Chrome not to use /dev/shm. I opened #3633 for more details about this.

[wralitera]

hello @brian-mann . How do you set up the --ipc=host into the gitlab-ci.yml?

[ccorcos]

Anyone know how to add --ipc=host to CircleCI? It looks like they call docker run somewhere out of our control...

[mitar]

You cannot. And this is why this workaoround does not really work.

[ccorcos]

@maximilianschmitt seems to think you can: https://gitter.im/cypress-io/cypress/archives/2018/10/17

[ccorcos]

Actually, a build just finished and it crashed pretty quickly

[ccorcos]

Looks like there's plenty of shared memory.

df -h /dev/shm
Filesystem      Size  Used Avail Use% Mounted on
shm              30G  8.0K   30G   1% /dev/shm

[adamlewkowicz]

I had the same issue yet I have changed my image from cypress/base to cypress/browsers:node11.13.0-chrome73 and it now works without crashes on gitlab-ci.

image: cypress/browsers:node11.13.0-chrome73

[rwralitera]

I will test this image: cypress/browsers:node11.13.0-chrome73 right now. Thanks @alk831

[rwralitera]

I tested and it failed:

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 2: 0x7fa824226887 [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 3: 0x7fa823d95a57 [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 4: 0x7fa823d959d5 [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 5: v8::internal::Factory::NewStruct(v8::internal::InstanceType) [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 6: 
<--- Last few GCs --->

[425:0x2659cbff8000]  1853805 ms: Mark-sweep 2057.7 (2176.7) -> 2057.7 (2154.2) MB, 2096.2 / 0.0 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 2096 ms) last resort 
[425:0x2659cbff8000]  1856036 ms: Mark-sweep 2057.7 (2154.2) -> 2057.7 (2154.2) MB, 2230.4 / 0.0 ms  last resort 


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x162eb08ad681 <JSObject>
    1: bindContainer(aka bindContainer) [/builds/wynd-qa/BO/node_modules/typescript/lib/typescript.js:~22229] [pc=0x2ef0ae593de3](this=0x2ae892502311 <undefined>,node=0x7235b368b9 <NodeObject map = 0x3516afdfd7d1>,containerFlags=45)
    2: bind(aka bind) [/builds/wynd-qa/BO/node_modules/typescript/lib/typescript.js:~23556] [pc=0x2ef0ae55cdac](this=0x2ae892502311 <undefined>,node=0x7235b368b9 <No...

v8::internal::Factory::NewTuple3(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 7: 0x7fa823b407a1 [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 8: 0x7fa823b4347b [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 9: 0x7fa823b431bc [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
10: 0x7fa823b47b5f [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
11: 0x2ef0add043fd

Timed out waiting for the browser to connect. Retrying...

The following error was thrown by a plugin. We've stopped running your tests because a plugin crashed.

[adamlewkowicz]

@bentrole It crashed only once for me with this image and now it works properly. I would try to split large tests into smaller ones and run without recording until there is no fix for this.

[rwralitera]

@alk831 I already splited my tests because of that but this bug is there since 2016, If there is no fixes until now, I think that they will not correct that! ;-(

[ccorcos]

@brian-mann can we please reopen this issue? If we cannot scale out our testing library, I don't think we can use this service :/

[jackbravo]

What about when running on github actions, like https://github.com/cypress-io/github-action ? How do you enable ipc=host?

[mattvb91]

Started experiencing this again on github actions upgrading from 5.3 to 6.2

[vincentpalita]

Version 6.2.1 seems to fix that for me.

[mattvb91]

Unfortunately 6.2.1 still crashing https://github.com/corejam/corejam/runs/1650892998?check_suite_focus=true#step:9:290 staying on 5.3 for now. Maybe someone else can chime in a version number when it started happening for them between 5.3 & 6.2 as I dont have the time to manually go through each version / build the associated docker images

[mattvb91]

https://github.com/corejam/corejam/runs/1784204472?check_suite_focus=true#step:9:313

Still no luck on 6.3.0

5.3.0 still stable for us have not had any chrome crashes there

---

Edit crashes on 6.4.0 too: https://github.com/corejam/corejam/runs/1820965122?check_suite_focus=true#step:9:287

[mattvb91]

Anyone looking to enable IPC=host on github actions it looks like it is possible like this:
https://github.com/corejam/corejam/pull/311/files#diff-014228303dff9a1af15f4bbd18401f906380129b10ae2a2c62f8b8be592ff88eR105

You can see it setting the option here: https://github.com/corejam/corejam/runs/1888917637?check_suite_focus=true#step:2:111
/usr/bin/docker create --name 01f73992fe8148a89514be690697b85d_corejamcypress640_aa44b9 --label 5588e4 --workdir /__w/corejam/corejam --network github_network_3bce8c6715944b47a995b42f42973efa --ipc=host ...

More context here: actions/runner-images#2697 (comment)

Just let it run through 3 times on our setup and its finally running it seems!!

Can anyone else confirm? If its working for others ill put in a MR with the above sample for docs

Edit: Have since put on about 10 runs from merge requests and its stable!

[dobriai]

FWIW, in our case adding --shm-size=512m seems to have quieted down the issue. For now. Using cypress@6.4.0.

[TimothyRHuertas]

If you added --ipc=host and you still get the error, try restarting docker 🤦

[cypress-bot]

The code for this is done in cypress-io/cypress#15825, but has yet to be released.
We'll update this issue and reference the changelog when it's released.

[flotwig]

In 7.0.1, Cypress passes disable-dev-shm-usage to Chrome and Electron on startup, which should remove the need to use --ipc=host or increase the /dev/shm size. Once it's released this should be solved.

[cypress-bot]

Released in 7.0.1.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v7.0.1, please open a new issue.