Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cypress open hangs for a minute or two before loading #29171

Closed
AlexandreRozier opened this issue Mar 20, 2024 · 15 comments · Fixed by #29452
Closed

Cypress open hangs for a minute or two before loading #29171

AlexandreRozier opened this issue Mar 20, 2024 · 15 comments · Fixed by #29452
Labels

Comments

@AlexandreRozier
Copy link

AlexandreRozier commented Mar 20, 2024

Current behavior

Hi !
For some time now, cypress open takes a couple minutes to start when running npx cypress open
Here are the steps we've encountered:

1.Cypress browser window is blank
image

  1. After a minute or so, the cypress loading logo appears and cypress works normally
    image

Desired behavior

npx cypress open opens cypress window quickly

Test code to reproduce

bug-repro.zip

I dumped the output of DEBUG=* npx cypress open --config-file cypress/cypress.local.config.ts --browser edge in this file :
out.log

It seems that the app stops being stuck shortly after the following logs appear :

2024-03-20T08:16:30.243Z cypress:data-context:sources:GitDataSource Refreshing git data
2024-03-20T08:16:30.243Z simple-git:task:rev-parse:4 Adding task to the queue, commands = [ 'rev-parse', '--show-toplevel' ]
2024-03-20T08:16:30.244Z simple-git:scheduler Scheduling id=4
2024-03-20T08:16:30.244Z simple-git:scheduler Attempting id=4
2024-03-20T08:16:30.244Z simple-git:task:rev-parse:4 Starting task
2024-03-20T08:16:30.244Z simple-git [GitExecutor] [SPAWN] git [ 'rev-parse', '--show-toplevel' ]
2024-03-20T08:16:30.244Z simple-git:task:rev-parse:4 [SPAWN] git [ 'rev-parse', '--show-toplevel' ]
2024-03-20T08:16:30.244Z simple-git:task:rev-parse:4 [SPAWN] {
  cwd: '/home/rozierale/Work/Rte/Naza/alz/naza-front',
  env: undefined,
  windowsHide: true
}

The "execution time" associated with the command cypress:data-context:sources:GitDataSource Refreshing git data was of 1minute (pretty long), so I dug a bit in cypress source code (GitDataSource.ts) , but it seems that the "1min" delay is due to this part :

# GitDataSource.ts -> refreshAllGitData()
this.#intervalTimer = setTimeout(() => {
          this.#refreshAllGitData()
        }, SIXTY_SECONDS)

Note that we don't have any delay with cypress run, and this piece of code is ran only in open mode, that's why I thought this was relevant !

if (config.isRunMode) {
      this.#verifyGitRepo().catch(() => {
        // Empty catch for no-floating-promises rule
      })
    } else {
      this.#refreshAllGitData()
    }

By downgrading to cypress 12.5.0, this issue disappears and cypress start normally. Here are the logs of a successful execution :
working.log

Interestingly, these logs do not contain the lines that appear when cypress stops being stuck (see above), so they must be unrelated to the issue....

And finally, cypress version 13.4.0 also has this issue (the only other I tried so far)

Sorry for the lack of other clues, I hope you'll have the time to take a look ! And thanks again for Cypress, we're using it everyday :)

Cypress Version

13.6.4

Node version

v16.15.0

Operating System

Ubuntu 20.04.5 LTS Release

Debug Logs

see linked files.

Other

No response

@AlexGuironnetRTE
Copy link
Contributor

AlexGuironnetRTE commented Mar 20, 2024

Hello,
I'm facing the same problem. Here is my two cents:

Cypress versions Result
12.5.0 Launches quickly
12.6.0 Launches quickly
13.0.0 Launches quickly
13.1.0 Launches quickly
13.2.0 Crashes with seemingly unrelated error about dependencies (Error: Cannot find module 'bluebird'), didn't investigate further Hangs
13.3.0 Hangs

@jennifer-shehane
Copy link
Member

@AlexandreRozier Thanks for the investigation. That sixty seconds is suspicious. This was introduced in 10.0, so it's interesting that @AlexGuironnetRTE mentions this behavior showing up in an older version. I wonder if it's a combination of something recent with this code.

@jennifer-shehane jennifer-shehane added the stage: needs investigating Someone from Cypress needs to look at this label Mar 20, 2024
@AlexGuironnetRTE
Copy link
Contributor

Thanks for the reply. Turns out my issue with 13.2.0 was the same as #27813. Clearing the cache and reinstalling as suggested made the error go away. And then.... cypress open hangs.

So that means whatever might interact with the function from 10.0 as you said would have been introduced with 13.2.0...
Unfortunately, to my untrained eye there is nothing suspicious either in the release notes or in the added commits.
Could dependency updates like the node or electron version have something to do with it?

@AlexGuironnetRTE
Copy link
Contributor

I also tried from home to rule out any company proxy timeout or anything of this kind, and I have the same behaviour.

@AlexandreRozier
Copy link
Author

AlexandreRozier commented Mar 21, 2024

Thks for the reply ! I don't think refreshAllGitData is the culprit since it waits for 3 promises to resolve :

  • loadAndWatchCurrentBranch -> returns OK (Watcher initialized)
  • loadGitHashes -> returns OK (Calling onGitLogChange: callback defined ...)
  • loadBulkGitInfo -> never called, no mention of debugVerbose(checking %d files, absolutePaths.length) in my logs
    So none of these things seem to hang

@AlexandreRozier
Copy link
Author

AlexandreRozier commented Mar 21, 2024

@jennifer-shehane I'm adding a bug repro to this issue , hopefully this will help !

bug-repro.zip

Steps :

  1. (optional) rm -rf ~/.npm/_npx/ // clear npx cache
  2. npm i
  3. npx cypress open
    It should hang for a while :)
    Also changing cypress version to 12.5.0 removes the launch delay on this repro.

@Kaiden42
Copy link

Kaiden42 commented Apr 5, 2024

We are facing the same issue with Cypress 13+. Cypress 12.17.4 launches quickly. Cypress starts hanging for us at 13.0.0 on the loading screen. At least we are getting the "loading animation" instantly 😉.

We have tested 13.0.0, 13.1.0, 13.2.0 and 13.7.2. All are hanging for several minutes.

Our system setup:
We are running Cypress on a virtual remote Linux system with Node v20.10.0. This system is encapsulated in a closed network environment (no internet connection). The Cypress Gui is forwarded to our local machine using x11.

In the logs I can see a lot of timeouts after 4 minutes appearing right at the moment where the "Welcome to Cypress" page appears.

Debug log
...
2024-04-05T13:50:24.140Z cypress:network:agent addRequest called { isHttps: true, href: 'https://download.cypress.io/desktop.json' }
2024-04-05T13:50:24.140Z cypress:network:connect beginning getAddress { hostname: 'download.cypress.io', port: 80 }
2024-04-05T13:50:24.141Z cypress-verbose:server:util:file read succeeded or failed for /home/user/.config/Cypress/cy/production/projects/dashboard-ui-6e167f91a60716f164cd7eb56aa17a5b/state.json
2024-04-05T13:50:24.141Z cypress-verbose:server:util:file attempt to unlock /home/user/.config/Cypress/cy/production/projects/dashboard-ui-6e167f91a60716f164cd7eb56aa17a5b/state.json
2024-04-05T13:50:24.141Z cypress-verbose:server:util:file unlock succeeded or failed for /home/user/.config/Cypress/cy/production/projects/dashboard-ui-6e167f91a60716f164cd7eb56aa17a5b/state.json
2024-04-05T13:50:24.141Z cypress-verbose:server:util:file attempt to get lock on /home/user/.config/Cypress/cy/production/projects/dashboard-ui-6e167f91a60716f164cd7eb56aa17a5b/state.json
2024-04-05T13:50:24.142Z cypress-verbose:server:util:file getting lock succeeded or failed for /home/user/.config/Cypress/cy/production/projects/dashboard-ui-6e167f91a60716f164cd7eb56aa17a5b/state.json
2024-04-05T13:50:24.142Z cypress-verbose:server:util:file write /home/user/.config/Cypress/cy/production/projects/dashboard-ui-6e167f91a60716f164cd7eb56aa17a5b/state.json
2024-04-05T13:50:24.142Z cypress-verbose:server:util:file write succeeded or failed for /home/user/.config/Cypress/cy/production/projects/dashboard-ui-6e167f91a60716f164cd7eb56aa17a5b/state.json
2024-04-05T13:50:24.142Z cypress-verbose:server:util:file attempt to unlock /home/user/.config/Cypress/cy/production/projects/dashboard-ui-6e167f91a60716f164cd7eb56aa17a5b/state.json
2024-04-05T13:50:24.143Z cypress-verbose:server:util:file unlock succeeded or failed for /home/user/.config/Cypress/cy/production/projects/dashboard-ui-6e167f91a60716f164cd7eb56aa17a5b/state.json
2024-04-05T13:50:24.147Z cypress:network:connect got addresses { hostname: 'download.cypress.io', port: 80, addresses: [ { address: '104.22.11.239', family: 4 }, { address: '172.67.25.250', family: 4 }, { address: '104.22.10.239', family: 4 }, { address: '2606:4700:10::6816:bef', family: 6 }, { address: '2606:4700:10::ac43:19fa', family: 6 }, { address: '2606:4700:10::6816:aef', family: 6 } ] }
...
2024-04-05T13:52:32.869Z cypress:network:agent got family { family: undefined, href: 'https://registry.npmjs.org/cypress' }
2024-04-05T13:52:32.880Z cypress:network:agent got family { family: undefined, href: 'https://download.cypress.io/desktop.json' }
2024-04-05T13:52:37.912Z cypress-verbose:packherd:trace Module._load "util"
2024-04-05T13:52:37.913Z cypress-verbose:server:util:process_profiler all Cypress-launched 
...
2024-04-05T13:54:43.941Z cypress:data-context:sources:VersionsDataSource Error fetching https://download.cypress.io/desktop.json: g [FetchError]: request to https://download.cypress.io/desktop.json failed, reason: connect ETIMEDOUT 104.22.10.239:443 at ClientRequest.<anonymous> (<embedded>:2049:1240895) at ClientRequest.emit (node:events:514:28) at TLSSocket.socketErrorListener (node:_http_client:501:9) at TLSSocket.emit (node:events:514:28) at emitErrorNT (node:internal/streams/destroy:151:8) at emitErrorCloseNT (node:internal/streams/destroy:116:3) at process.processTicksAndRejections (node:internal/process/task_queues:82:21) { type: 'system', errno: 'ETIMEDOUT', code: 'ETIMEDOUT' }
2024-04-05T13:54:43.942Z cypress:data-context:sources:VersionsDataSource Error fetching 'https://registry.npmjs.org/cypress' g [FetchError]: request to https://registry.npmjs.org/cypress failed, reason: connect ETIMEDOUT 104.16.24.34:443
    at ClientRequest.<anonymous> (<embedded>:2049:1240895)
    at ClientRequest.emit (node:events:514:28)
    at TLSSocket.socketErrorListener (node:_http_client:501:9)
    at TLSSocket.emit (node:events:514:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  type: 'system',
  errno: 'ETIMEDOUT',
  code: 'ETIMEDOUT'
}
2024-04-05T13:54:43.944Z cypress-verbose:packherd:trace Loaded "/home/user/.cache/Cypress/13.7.2/Cypress/resources/app/packages/graphql/node_modules/debug/src/index.js" (cache | exports) -> moduleCache: 662, exportsCache: 3680, loaded: 677
2024-04-05T13:54:43.944Z cypress-verbose:graphql:fields query HeaderBar_HeaderBarQuery took 259879ms to resolve ["versions"]
...

@jennifer-shehane
Copy link
Member

@AlexandreRozier I'm not able to run this repo, it errors without access to https://devin-depot.rte-france.com/ Could you update it?

@bjowes
Copy link
Contributor

bjowes commented Apr 20, 2024

Due to issues reported to my Cypress plugin cypress-ntlm-auth, I looked into slow/hanging startups. These are my findings:

When started with cypress open behind a corporate proxy that performs HTTPS decryption, the startup stalls/hangs when fetching data from download.cypress.io and registry.npmjs.org. If there is no proxy or the proxy lets HTTPS pass through without decryption, it works fine. If you use cypress run, it works fine.

Looking into the source, I found that the above requests are not performed on cypress run, that is why this mode still works fine.

Figuring out why the issue appears behind a corporate proxy took quite some digging...
When a request to a HTTPS site is sent through a proxy, this starts by creating a tunneling socket to the target. Many corporate proxies fake this by acting as a MITM, to be able to analyze the unencrypted traffic. In that case, the tunneling socket terminates in the proxy. Then, when the actual request is made, the proxy inspects the host header of the request to route the request to the actual target. The issue here is that the port of the host header is incorrect. For both the targets above, they use https on the standard port 443. But the host header states port 80, which makes the proxy send the request to https://download.cypress.io:80 instead of https://dowload.cypress.io. Naturally this fails, and apparently there is some error handling missing in cypress when those requests are made since it just stalls instead of fallback to defaults (the requests are non-critical to running cypress).

Why does the host header become incorrect?

Cypress uses the package cross-fetch for the requests, which in turn uses node-fetch since these are originating from a Node process (and not from inside the browser). In the fetch call, a custom agent is passed in. This is the CombinedAgent (cypress/packages/network/lib/agent.ts). When Node creates the host header, it tries to determine the port to use, using the following sources:

  1. port specified in the request options (this only occurs for non-standard ports)
  2. defaultPort specified in the request options (this value is not passed)
  3. defaultPort as specified by the agent (for CominedAgent this value is not set)
  4. when none of the above are set, use port 80

After this, the addRequest method of the agent is called. In CombinedAgent, the port is adjusted to the correct value (there are comments in the code that the port will be 80 when it should be 443). However, since this occurs after the host header is created, this does not help. The host header is still incorrect, leading to incorrect proxy behavior when a MITM proxy is in use.

How can it be fixed

The simple solution of passing along either port or defaultPort in the options to the fetch call does not work. node-fetch only respects a few options - port and defaultPort are both ignored. Another option is to assign the defaultPort value of the agent. This is respected correctly. However, the CombinedAgent is used for both HTTP and HTTPS requests and it is a shared instance - setting the defaultPort to 443 would render incorrect host headers for HTTP requests instead. I see the following possible solutions:

  1. Persuade the node-fetch maintainers to respect the port (or defaultPort) options. This might be difficult, since this is only required due to the odd use case with the CombinedAgent. An alternative would be to create a fork of node-fetch and add the required support (handle the options or detect the situation by inspecting the URL protocol part).

  2. Refactor CombinedAgent into the standard Node way - separate agents for HTTP and HTTPS.

  3. Create a shadow instance of CombinedAgent with the same instances of http.Agent and https.Agent, while adding a defaultPort property with value 443. When making calls to fetch, determine which agent instance to pass along based on the protocol of the URL. I have successfully tested this option locally.

Which solution seems most suitable? Option 3 would only require a limited effort, but I cannot determine if this could cause issues in other calls to fetch.

I would be happy to help out, but option 2 requires someone highly familiar with the different use cases of the agent.

There are likely many other issues reported that are related to this issue. For instance communication with the cypress cloud services may be affected. Hence I think there is great value to resolve this.

@jennifer-shehane
Copy link
Member

Hey @bjowes, thanks for sharing your detailed investigation. I'll see if the team can look at this. In the meantime we're always welcome to accept PRs for fixes.

@bjowes
Copy link
Contributor

bjowes commented Apr 25, 2024 via email

@cacieprins
Copy link
Contributor

@bjowes Thanks so much for the detailed investigation!

We already use patch-package to patch @benmalka/foxdriver and tsconfig-paths in @packages/server (and other packages elsewhere), so it's an established pattern in this repository.

Refactoring to the more standard pattern is probably the more sustainable approach.

We would be fine with either of these, though I have a slight preference towards the latter :)

@AlexandreRozier
Copy link
Author

Thanks bjowes for your input ! I can confirm we're also having the same issue : cypress open is fast when we're working at home, but slow when we're on-site. So it seems we're having the same problem :)

@jennifer-shehane I'm adding a simpler repro archive below; I recreated one from your cypress-test-tiny repo.

bugrepro-cyrepo.zip

@bjowes
Copy link
Contributor

bjowes commented May 1, 2024

Ok @cacieprins - PR prepared.
Didn't know about patch-package before, that is an excellent tool to avoid forking other packages! I chose that approach since the patch was relatively simple to implement. I'm sure there was plenty of technical thought behind CombinedAgent, didn't want to replace it without more background knowledge.

jennifer-shehane added a commit that referenced this issue May 7, 2024
* Patch node-fetch to set defaultPort based on protocol

* unit test that proper host headers are sent with fetch

* changelog

* make patch more robust

---------

Co-authored-by: Jennifer Shehane <jennifer@cypress.io>
@cypress-bot
Copy link
Contributor

cypress-bot bot commented May 8, 2024

Released in 13.9.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v13.9.0, please open a new issue.

@cypress-bot cypress-bot bot removed the stage: needs investigating Someone from Cypress needs to look at this label May 8, 2024
@cypress-bot cypress-bot bot locked as resolved and limited conversation to collaborators May 8, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants