New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: Long Cypress verification times #21174
Conversation
…into tbiethman/fix-electron
Thanks for taking the time to open a PR!
|
Test summaryRun details
View run in Cypress Dashboard ➡️ This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard |
@@ -5,7 +5,7 @@ | |||
"private": true, | |||
"main": "index.js", | |||
"scripts": { | |||
"test": "mocha --timeout 30000 test/spec.js" | |||
"test": "mocha --timeout 45000 test/spec.js" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are we doing this because even before the electron bump, we were pretty close to the timeout?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep, I tried to call this out in the PR description. This test has flaked a few times prior to the recent changes, even in develop. 30s was just a little too tight to verify cypress and run the two specs within one mocha test.
Please pull this branch down and run verify locally to sanity check the verify duration.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried, could not reproduce this at all. It passes for me even on 10.0-release.
I am on linux though, not an M1 mac.
I also tried on windows but I can't even get the binary to build on it (gave up).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for tracking this down, I think the comment is sufficient to explain why we're requiring that module as a side-effect, although the behavior is still quite strange in general.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for tracking this down. I tried to reproduce the flake on my machine, but I couldn't. I might need to throttle the CPU to simulate the CI machines.
Code looks good and I'm glad the flake is reduced.
In 10.0-release, verifying Cypress can take a long amount of time, ~18s on average on my Mac. This is due to a few factors:
Recent changes were made to optimize dependency loading during boot, but we seem to have been relying on this require having executed elsewhere during initial load: https://github.com/cypress-io/cypress/pull/20932/files#diff-fd8260e3de42c8cd3e678bac66b13db87507e88b3b4b3919d7463645bc345b28L16
Electron v15.4.0 recently fixed a bug related to process termination and will now wait for the run loop to be idle prior to closing (I think, been a while since my C days).
The combination of these two changes is causing the significant slowdown, likely because loading one of those imports keeps the process active longer than it used to. It’s also the root cause of the binary-system-tests flake we’ve been seeing in 10.0-release builds; one of those tests runs verify and occasionally exceeds its timeout.
Changes made here include:
Adding a require for 'server/lib/project-base' to
server/lib/cypress
. This replaces theserver/lib/open_project
require that was there prior to fix: remove extra bundled electron in 10.0 binary #20932. I was able to track down the slowness to the missing project-base import that was previously imported by open_project; however, the imports within project-base seem to be interrelated, as far as the slowness goes. I added a comment here to validate the need for this after we upgrade to Electron 18, which is in progress in develop.Removing the openProject function defined in
server/lib/cypress
. It is never called at this point and I don't want its presence to cause confusion. If I'm missing a non-obvious place where this is used, please call it out, I can't find it.Adding inline requires for electron within makeDataContext and related files; we don't need to require electron initially.
Updated the mocha timeout for the module-api binary system tests. These tests are exceptionally flakey without these PR changes, but they were occasionally flakey prior to the electron upgrade and even in develop due to verification being somewhat dependent on system resources. 30s was just a little too tight; bumped to 45s, which in all my testing has been more than adequate.
User facing changelog
N/A
Additional details
How has the user experience changed?
PR Tasks
cypress-documentation
?type definitions
?cypress.schema.json
?