Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Long Cypress verification times #21174

Merged
merged 12 commits into from Apr 25, 2022
Merged

Conversation

tbiethman
Copy link
Contributor

@tbiethman tbiethman commented Apr 22, 2022

In 10.0-release, verifying Cypress can take a long amount of time, ~18s on average on my Mac. This is due to a few factors:

The combination of these two changes is causing the significant slowdown, likely because loading one of those imports keeps the process active longer than it used to. It’s also the root cause of the binary-system-tests flake we’ve been seeing in 10.0-release builds; one of those tests runs verify and occasionally exceeds its timeout.

Changes made here include:

  • Adding a require for 'server/lib/project-base' to server/lib/cypress. This replaces the server/lib/open_project require that was there prior to fix: remove extra bundled electron in 10.0 binary #20932. I was able to track down the slowness to the missing project-base import that was previously imported by open_project; however, the imports within project-base seem to be interrelated, as far as the slowness goes. I added a comment here to validate the need for this after we upgrade to Electron 18, which is in progress in develop.

  • Removing the openProject function defined in server/lib/cypress. It is never called at this point and I don't want its presence to cause confusion. If I'm missing a non-obvious place where this is used, please call it out, I can't find it.

  • Adding inline requires for electron within makeDataContext and related files; we don't need to require electron initially.

  • Updated the mocha timeout for the module-api binary system tests. These tests are exceptionally flakey without these PR changes, but they were occasionally flakey prior to the electron upgrade and even in develop due to verification being somewhat dependent on system resources. 30s was just a little too tight; bumped to 45s, which in all my testing has been more than adequate.

User facing changelog

N/A

Additional details

How has the user experience changed?

PR Tasks

  • Have tests been added/updated?
  • Has the original issue (or this PR, if no issue exists) been tagged with a release in ZenHub? (user-facing changes only)
  • Has a PR for user-facing changes been opened in cypress-documentation?
  • Have API changes been updated in the type definitions?
  • Have new configuration options been added to the cypress.schema.json?

@tbiethman tbiethman requested review from a team as code owners April 22, 2022 15:16
@tbiethman tbiethman requested review from jennifer-shehane and removed request for a team April 22, 2022 15:16
@cypress-bot
Copy link
Contributor

cypress-bot bot commented Apr 22, 2022

Thanks for taking the time to open a PR!

@tbiethman tbiethman requested review from tgriesser and removed request for jennifer-shehane April 22, 2022 15:17
@tbiethman tbiethman self-assigned this Apr 22, 2022
@cypress
Copy link

cypress bot commented Apr 22, 2022



Test summary

4761 0 71 0Flakiness 0


Run details

Project cypress
Status Passed
Commit f269d96
Started Apr 25, 2022 4:38 PM
Ended Apr 25, 2022 4:49 PM
Duration 11:23 💡
OS Linux Debian - 10.10
Browser Multiple

View run in Cypress Dashboard ➡️


This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

@flotwig flotwig self-requested a review April 22, 2022 15:34
@@ -5,7 +5,7 @@
"private": true,
"main": "index.js",
"scripts": {
"test": "mocha --timeout 30000 test/spec.js"
"test": "mocha --timeout 45000 test/spec.js"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we doing this because even before the electron bump, we were pretty close to the timeout?

Copy link
Contributor Author

@tbiethman tbiethman Apr 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, I tried to call this out in the PR description. This test has flaked a few times prior to the recent changes, even in develop. 30s was just a little too tight to verify cypress and run the two specs within one mocha test.

Please pull this branch down and run verify locally to sanity check the verify duration.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried, could not reproduce this at all. It passes for me even on 10.0-release.

I am on linux though, not an M1 mac.

I also tried on windows but I can't even get the binary to build on it (gave up).

Copy link
Member

@tgriesser tgriesser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracking this down, I think the comment is sufficient to explain why we're requiring that module as a side-effect, although the behavior is still quite strange in general.

Copy link
Contributor

@lmiller1990 lmiller1990 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracking this down. I tried to reproduce the flake on my machine, but I couldn't. I might need to throttle the CPU to simulate the CI machines.

Code looks good and I'm glad the flake is reduced.

@tgriesser tgriesser merged commit cb5b547 into 10.0-release Apr 25, 2022
@tgriesser tgriesser deleted the tbiethman/fix-electron branch April 25, 2022 17:05
tgriesser added a commit that referenced this pull request Apr 25, 2022
…-config

* 10.0-release:
  fix: Long Cypress verification times  (#21174)
  feat: UI updates to scaffolded files (#21155)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants