Cypress runMode retry shows duplicate tests and getting stuck during run #14423
Comments
davidhu2000
changed the title
|
We recently released a fix affecting retries. But this has also been happening in Cypress 5.4.0 right? You mention:
I see you're already setting ipc=host like mentioned here #350 (comment) |
jennifer-shehane
added
the
stage: needs information
|
yeah, it was happening on 5.4.0. I saw the fix in 6.2.1 so I updated and it is still happening.
we are running on a suite of about 300 tests. Altho I don't think this is a requirement, I'm really not sure when this happens.
It's random where it fails. Some times it happens on the first test, some times happening after a few suites. I don't see any pattern on when it happens.
I did a search for In our plugins file, we have this (if this is helpful. on('before:browser:launch', (browser, launchOptions) => {
if (browser.name === 'chrome') {
launchOptions.args.push('--disable-dev-shm-usage');
}
return launchOptions;
});
So i'm using the command during my local testing. I'm not sure if we are using that on our jenkins server. Although I don't think having I tried running the same thing on one of our older jenkins servers
That seems to work fine. The one we are having trouble with is
|
|
I'm not really sure how much more we can investigate without a way to reproduce it. Is it possible to narrow down the issue anymore? Commenting out suites/tests? |
|
I'll see if i can come up with a reproducible example to help track this down. |
|
same problem here, If you managed to solve it, I'll be more than happy if you share the solution . |
|
My two cents here is that there could be an improvement to the component tests where cypress waits for the bundler to finish before trying to load in the test suites, or before it tries to run the first test in the detected suites, as anecdotally there seems to be a bug in the retry logic that has some cascading errors. Additionally, running the same setup in cypress 9 with component tests didn't have the same issue, so there might have been a regression in the major version update to 10. Some context that may help reproducibility: Cypress Version: While I'm referencing a convenience image here, I find it just as reproducible with a project specific official node image as well and referencing cypress as a a vue-cli plugin runtime as well. The biggest factor I see here is that docker's shared volume performance right now is considerably slower than running purely in docker, or purely on the host machine. So by mounting in the working dir as a volume for convenience, you can force a build that takes over the general 30s timeout and reproduce the duplicated tests on the first test suite. As a side note, I also see occasionally that in this situation, there is also a fatal reporter, or fatal runtime error because of the erroneous concurrent test suite executions. For example: Also an example of the error thrown by the reporter with duplicated tests: |
|
This issue has not had any activity in 180 days. Cypress evolves quickly and the reported behavior should be tested on the latest version of Cypress to verify the behavior is still occurring. It will be closed in 14 days if no updates are provided. |
I am truly stumped with this issue. We occasionally have an issue where our site doesn't load during a few tests.
We would see this error
So we thought maybe adding retry would help with the issue.
so we added this into cypress.json.
Current behavior
We would see an output like this, where the
(Attempt 1 of 3)is duplicated. Then the test gets completely stuck. I left a run overnight, and 10 hours later, it still didn't move forward.Desired behavior
Running retry locally works fine. We would see
Attempt 1then 2 and 3.Test code to reproduce
The test is stuck when we are running on jenkins inside of docker images build from
cypress/included:6.2.1.This tricky part is that this error only happens sporadically on random tests. I tried to add
DEBUG=cypress:*but that added so much log I couldn't load it all. If there is any specific I can output for the debug variable, please let me know.Toward the end of the output, we just see the profiler over and over again.
Running the command can sometimes reproduce the issue running against a local server. looking at my application logs locally, it does look like the tests actually finished. It is just not continuing. I suspect somehow due to the duplicate test output? It doesn't just happen on attempt 1 either, I have seen it happen on attempt 2 as well.
The screenshot just shows a gray screen (which is not part of our application. If a page doesn't load, it should show a white background.)
What's interesting is the screenshot names that is generated off of one test
Even though our tests do not have an afterEach. The screenshot with the fully name has the gray blank page, while the after each screenshot has a white background.
We tried increasing timeout to 120 seconds, and still see the same issue.
I think the way to reproduce to run cypress in docker headless chrome/electron for a large test suite.
We are using Rails 5.1.4, ruby 2.4.9. With a react frontend, so the page loads, then load the javascript parts to render.
Versions
6.2.1
5.4.0
The text was updated successfully, but these errors were encountered: