"run" is much slower than "open"

[cake-jnorman]

This seems like a bug - the headless version of the tests run 4x - 5x slower than the headed ones. (cypress run is about 4x-5x slower than the same tests run via cypress open)

Current behavior:

I have a test suite with 77 tests in it currently.

In local dev I have been using the open command, per the docs. The GUI is awesome, thank you for this. The ability to select one tests suite at a time to test on the fly is great. And when I run all tests at once, they run fun fairly quick. All 77 of them run in about 1:15. This is acceptable, especially considering all of the debugging tools that are available in this mode. But cypress open isn't the problem.

We're working on getting a CI in place. In preparation for this, I ran the tests via cypress run as this is how I would expect our CI server to operate. I was (honestly) shocked to find that they were running 4x-5x slower in the headless electron browser. Tests routinely take at least 4:10 to complete in this mode. I've dug through other issues, google, docs, everything I could find and I have not found a way to speed this up (aside from running in parallel... which I don't feel addresses the real problem here).

Desired behavior:

Tests run at last as fast vis cypress run as they do for cypress open.

Steps to reproduce: (app code and test code)

...This may be a little tough. The bug is not a specific 'this blew up' but more a 'this really seems like wrong behavior, at the top level'. But if you feel that I can provide something specific, please ask.

Versions

Cypress 3.1.3
Node 10.12.0
Yarn 1.9.4
MacOS X High Sierra
Chrom 71 (cypress open executing in this browser)

config:

{
  "integrationFolder": "src/app/components/_pages/", // we keep tests with their corresponding component
  "testFiles": "**/*.int.spec.js", // as a result, we had to easily denote cypress tests
  "baseUrl": "http://localhost:3000/",
  "video": false,
  "chromeWebSecurity": false
}

Thank you!

[jennifer-shehane]

Hey @cake-jnorman, to clarify, you ran cypress run locally within your terminal? Could you open up the Activity Monitor program on your system and see what the CPU usage looks like in there while Cypress is running?

[cake-jnorman]

@jennifer-shehane Apologies for the delay.

I did as you asked and ran both Cypress run and Cypress open in my terminal. The results are surprising to me, but do make a little sense when in the context of this problem.

run
CPU between 35% and 70% (mostly between the high 30s and low 40s)
156-160 threads

open
5-40% (mostly in the high single digits)
156-160 threads

This surprises me since open launches the GUI... i would expect these numbers swapped. Buuuuttt... I've not looked into how you guys do things under the hood. :D

[cake-jnorman]

@jennifer-shehane any additional thoughts? can i get you any more information?

[brian-mann]

I don't think this is Cypress specific - in fact in Run Mode we disable certain debugging features that actually have a much bigger performance impact in open mode vs run mode.

I think all of what you're experiencing here is simply the difference between Chromium 59 vs the latest version of Chrome.

If you switched cypress run --browser chrome you would likely see even faster results than cypress open. Can you try that?

[brian-mann]

Also, it doesn't appear that you're comparing apples to apples here. You cannot compare your local machine with one in CI AND run them in two completely different modes AND completely different browsers. Do the same thing everywhere and then you'll see comparable results.

Run Electron locally and compare:

• cypress open <-- select Electron
• cypress run <-- Electron by default

Then run Chrome locally and compare:

• cypress open <-- select Chrome
• cypress run --browser chrome

Then run in CI:

• cypress run <-- Electron by default
• cypress run --browser chrome

Now you can compare each set's performance with the other. Mixing anything else isn't equivalent.

Your CI machine is likely completely underpowered and that's why you're seeing such a large differential. Give it more resources and it'll improve. Please post your results.

[cake-jnorman]

both instances were run on my local machine. nothing different between the two runs other than the command executed. I'll give your suggestions a go and get back to you.

[fr0]

I am seeing this problem as well. Is there a reason that such an old version of Electron is being used? I'm not sure if upgrading will solve this problem but it seems like it is worth a shot.

[cake-jnorman]

i do not think that it is electron per se, unless it has issues in headless mode. we ran both the headless and headed versions of the test framework on electron and the headless ran slower every single time, typically by a factor of 4 or 5. That's right, the headless version of the test suite is slower. i don't get it either. I've not had time to dig in more yet but i hope to get back to gathering some metrics soon.

[agardou]

I had a similar case : headless testing was slower than run --browser chrome but it was due to video recording feature. Try add video: false to your cypress.json and check testing time with both cases.

In my case, I only enable video recording in CI mode, but not for local runs.

[jennifer-shehane]

The video recording and uploading does add time to the end of each test file that is ran when video: true is set. You should see the timing this step took within the stdout of the run. Does this account for the extra time during cypress run?

[andreasmihm]

same issue here, my test suite runs (both on local machine):

• headless -config video=false : 02:21
  -headed (with Electron) : 01:35

[devdelpiero]

Cypress run is much slower than cypress open. Is anyone looking into this? Testcafe doesn't have this problem.

[maple-leaf]

This problem on my situation is mainly caused by close and relaunch chrome browser when run cypress run --browser=chrome, which will take abount 2 minutes, while cypress open and run all tests will take just about half minute.

So far, cypress seems not ready for chrome headless, is there a way to auto trigger run all action when run cypress open, I would like to run this even on CI for time saving while performance is not the problem so far.

[maple-leaf]

After digging into some source codes of cypress, could we introduce an env like autoRunAll for open, and check this env at specs.jsx file when it gui ready, If it's true, then auto trigger run all Specs.

I thinks this can be an option to reduce run time on CI caused by browser relaunching.

[jennifer-shehane]

@cake-jnorman @fr0 @devdelpiero @andreasmihm

Can you confirm if you are comparing the 2 situations below?

• cypress run
• cypress open then clicking 'Run All Tests'

Explanation of why it is slower (this is expected behavior)

cypress run relaunches the browser in between each spec file while 'Run All Tests' button does not. There is an issue to address this discrepancy here: #1586 This would account for the difference in running time.

If this is the case, then we will close this issue and suggest you add your thoughts and 👍 to this issue: #2951

[andreasremdt]

@jennifer-shehane I can also confirm the difference in speed between cypress run and cypress open (Run All Tests).

My impression is the same as @maple-leaf: Cypress relaunches the browser for each spec in the run mode, which adds to the overall time it takes to finish. For me, it didn't make a big difference if I ran the Electron or Chrome browser.

[getkode]

I'm experiencing the same issue, cypress open is way much faster than cypress run. If I'm lucky, the speed is decreased by 2. However, most of the time it ends up with timeout. Actually, I can reproduce the issue with cypress open by starting a screen record when the tests are running. I don't know why but the HTTP requests take more time.

BTW, I'm using a MacBook Pro with macOS 10.14.5.

[jennifer-shehane]

As explained in #2912 (comment), there are some reasons why it may be slower - please read this comment and refer to the other issue if needed.

Outside of this - we will need a reproducible example to prove cypress run is slower than cypress open (not accounting for the extra time used to relaunch the browser in between each spec file.

Currently we have to close this issue as there is not enough information to reproduce the problem. This does not mean that your issue is not happening - it just means that we do not have a path to move forward.

Please comment in this issue with a reproducible example and we will reopen the issue. 🙏

[bsarel]

I have the exact same issue (It wasn't always like that).

cypress open with electron runs lightning-fast while cypress run (default electron) runs extremely slow.

Perhaps it's worth mentioning that my colleagues are not suffering from this issue and that my mac's got 8GB of RAM while they have 16GB. (same dev env setup and cypress suite).

I've also noticed that right after a fresh restart it gets better. So it might be resource related.

I'll try to find anyone else with 8GB of RAM and see if I can isolate this one.

[jennifer-shehane]

The resources available on the system running the tests can most definitely result in a slower performance. This should always be investigated.

[tarponjargon]

yeah something changed. cypress run used to be way faster than cypress open "run all specs" but now it's the opposite.

[sergiomap]

I can also confirm this. Run with cypress run was much more faster in previous versions.

[lugus]

Same for us, much more slower, resulting some tests are not passing with cypress run.

[jennifer-shehane]

@tarponjargon @sergiomap @lugus Please comment in this issue with a reproducible example and we will reopen the issue. Also, show timing diffs from previous versions and current version - you can change Cypress versions quite quickly to prove this.

[tarponjargon]

Yes, we're being quite lazy ;) I will try to put together a test repo

[blumk]

@tarponjargon @lugus @sergiomap Switching to Chrome solved this issue for me.
Chrome runs about 4x faster than Electron on my machine.

cypress run --headless --browser chrome

[lugus]

Thanks for the tip @blumk i will test that after release of #5949

[MattMcFarland]

Right now for me each cypress it() function takes ~40 seconds to complete when running on our CI pipeline (as opposed to ~2 when running local). I think the vms just need to be beefier, but I don't know what specs to use. Can anyone suggest specs for vms running in CI for using electron?

[JZebra]

I have a similar problem. After experimenting with both commands locally, these are the differences that I've noticed:

• cypress run runs test suites ("describe" blocks) one by one
  • runs cypress/support/index.js before each suite
  • runs in Electron browser by default
• cypress open opens a GUI. Clicking "Run all specs" runs all suites
  • runs cypress/support/index.js just once

I called a Cypress command in cypress/support/index.js that reset and seeded the db. This was not an issue in local development, because my workflow for developing tests used the GUI and running specific specs. When I wanted to test the whole suite, the dbSetup() command only ran once. However, it would add ~20 * N seconds to cypress run, where N is the number of test suites.

Is there any built in solution to this? Don't see anything in https://docs.cypress.io/guides/core-concepts/writing-and-organizing-tests.html

[jennifer-shehane]

@JZebra No, we run all specfiles separately during cypress run with the support/index.js file running before each file in order to enable the use of parallelization.

We anticipate removing the 'Run all tests' functionality for several reasons, so recommend moving your workflow to the way cypress run runs. #1586

[VladimirMikulic]

I have the exact same problem. cypress open runs a test in a few seconds but cyress run in a Node.js script takes 80+ seconds! (recording disabled)

[scrumvisualize]

'cypress run' is very slower than previous versions, but 'cypress open' is faster when compared to cypress run. Few things I have tried to see if it makes the run faster: Like by setting the 'video' to false, number of retries to 1, earlier it was 2 and 'numTestsKeptInMemory' to 0, earlier it was 5, but no luck yet, still it is running very slow.
This is affecting my test run, I have got 18 spec files to run and it takes around 1 hour and 2 minutes now.
Before running the test i ensure that only few required resources were running in my system.

 "scripts": {
    "cy:run": "cypress run",
    "cy:open-maintest": "npm run cypress open",
    "cy:test-main": "set CYPRESS_RETRIES=1 && npm run  cypress run --browser electron"
  }
 

System configuration:

OS: Windows 10 Pro
Processor: AMD RYZEN 7 PRO 2.30 GH
16 GB RAM
SSD drive

Cypress configuration:

Cypress version: 4.10.0

cypress.json:
{
"baseUrl": "http://sometesturl.net/auth/login",
"permissionUrl": "http://sometesturl.net/",
"numTestsKeptInMemory": 0,
"chromeWebSecurity": false,
"projectId": "some_id",
"video": false,
"failOnStatusCode": false,
"env":
{
"RETRIES": 1
}
}
I have tried running the tests via npm command in command prompt and docker command in git bash, in both cases it is very slow.
In each spec there are approximately 7-8 tests and the same tests were running without any delay in older version of cypress.

context('UAT Test Cases-High-1', () => {

    beforeEach(() => {
        cy.loadTokens();
        cy.loginRequest();
    })

     it('Test 1', () => {
        // rest of test steps follows here ...
     })
    it('Test 2', () => {
      // rest of test steps follows here... 
     })
......

})

Gitbash:

Docker output:

[guilhermetod]

cypress run --headed

This makes it ok for me. It does prompt electron but it automatically closes when the tests are finished. Tests drop from 50 to 1 second even with the video recording.

Problem is possibly on not getting the native video.

[sanelen]

I am facing the same issue my tests are slower on Run and fail often but on Open the performance is quicker

[Michael-Robson]

Is there an open issue for this? I'm also getting the same issue with 6.0.1. If I run all the specs using cypress open it takes 8 minutes, using cypress run with the same browser it can be take anywhere between 15-20 minutes

[ShawnMercado]

I guess I'll just add another data point, same exact behavior for me. Mac OSX, pretty small test suite (12 spec files), using the Angular Cypress schematic. All the spec files are pretty small, 10/12 less than 50 lines and all of them less than 150 lines.
Browser set to chrome, all that.

Just absolutely UNBEARABLY slow when running in headless mode. Each spec individually takes 20-40 seconds, whereas using the GUI it's more like 5 seconds per spec, 10 at the most. In total it's like 2 minutes with the GUI, but in headless it was like 15 minutes at first. I managed to get it down like six minutes by disabling the videos and using dev build and whatnot, but it still doesn't make any sense

Activity Monitor says the terminal process is using between 1-2 GB (goes up and down a lot) which seems pretty high to me. But I have 32GB, it's nowhere near capacity.

[JessicaSachs]

@ShawnMercado can you give us a reproduction and open a new issue? We're currently investigating various performance issues.

[ShawnMercado]

@ShawnMercado can you give us a reproduction and open a new issue? We're currently investigating various performance issues.

Wish I had time, maybe in a couple weeks. What I'm working on is complicated and under NDA so I'd have to build a full-but-vanilla version to have a good chance of a meaningful reproduction.

Sorry I know that's not helpful :/