chore: update browser internal images to chrome 124 and firefox 125 #29391

AtofStryker · 2024-04-23T19:59:05Z

Closes Flake: errors ui assertion failures #28942

Additional details

Bumps chrome to version 124 from 118 and firefox from 115 to 125 for docker based tests inside the repo.

A few other things needed to happen in order to get chrome 124 working within the repo. This is outlined in this comment and thread below

Steps to test

How has the user experience changed?

PR Tasks

Have tests been added/updated?
Has a PR for user-facing changes been opened in cypress-documentation?
Have API changes been updated in the type definitions?

cypress · 2024-04-23T21:07:38Z

7 failed and 4 flaky tests on run #55240 ↗︎

7	286	7	22	4

Details:

chore: update documentation in response middleware
Project: cypress	Commit: `96ee05cfe5`
Status: Failed	Duration: 12:22 💡
Started: Apr 30, 2024 4:03 PM	Ended: Apr 30, 2024 4:16 PM

cypress\e2e\top-nav-launchpad.cy.ts • 0 failed tests • launchpad-e2e

View Output

Test		Artifacts

cypress\e2e\config-files-error-handling.cy.ts • 0 failed tests • launchpad-e2e

View Output

Test		Artifacts

cypress\e2e\open-mode.cy.ts • 0 failed tests • launchpad-e2e

View Output

Test		Artifacts

cypress\e2e\global-mode.cy.ts • 0 failed tests • launchpad-e2e

View Output

Test		Artifacts

cypress\e2e\error-handling.cy.ts • 0 failed tests • launchpad-e2e

View Output

Test		Artifacts

The first 5 failed specs are shown, see all 54 specs in Cypress Cloud.

cypress\e2e\project-setup.cy.ts • 2 flaky tests • launchpad-e2e

View Output

Test		Artifacts
... > can skip setup CT testing for an E2E project		`Test Replay` `Screenshots`
... > skips the setup steps when choosing component tests to run		`Test Replay` `Screenshots`

cypress\e2e\scaffold-component-testing.cy.ts • 1 flaky test • launchpad-e2e

View Output

Test		Artifacts
scaffolding component testing > vuecli4vue2 > scaffolds component testing for Vue CLI 4 w/ Vue 2 project		`Test Replay` `Screenshots`

cypress\e2e\migration.cy.ts • 1 flaky test • launchpad-e2e

View Output

Test		Artifacts
Full migration flow for each project > completes journey for migration-e2e-plugins-modify-config		`Test Replay` `Screenshots`

Review all test suite changes for PR #29391 ↗︎

jennifer-shehane · 2024-04-25T15:46:25Z

@AtofStryker This PR is failing with Firefox 115 in CI, but passes locally in Firefox 125. Curious to get the Firefox update in to see if it passes the tests to unblock this: #29224

AtofStryker · 2024-04-26T13:13:06Z

Currently having issues inside the run-app-integration-tests job where the job seems to continuously fail on the same spots. Investigating, I can reproduce this in docker when running on an Ubuntu 22 amd64 machine via

downloading and bashing into the docker image/container from the cypress directory (attaches via workspace)

docker run --rm -it -v $PWD:/e2e -w /e2e --entrypoint bash cypress/browsers-internal:node18.17.1-chrome124-ff125

Now that we are in the docker container, run the e2e jobs but just these two specs. This seems to reproduce every time.

cd packages/app
yarn cypress:run:e2e -b chrome --spec=./cypress/e2e/runner/reporter-ct-mount-hover.cy.ts,./cypress/e2e/runner/reporter-ct-vite.errors.cy.ts

AtofStryker · 2024-04-26T13:48:33Z

The issue looks related to the chrome update from 118 to 124. I am trying to narrow down to when the issue started occurring.

124 (124.0.6367.91-1) -> has problem
123 (123.0.6312.122-1) -> has problem
122 (122.0.6261.128-1) -> has problem
121 (121.0.6167.184-1) -> has problem
120 (120.0.6099.224-1) -> has problem
119 (119.0.6045.199-1) -> has problem
118 (118.0.5993.117-1) -> works
119 (119.0.6045.105-1) (closest revision to 118.0.5993.117-1) -> has problem

The issue looks identical to #28942 which we could not exactly figure out the cause of, but the flake likely existed in windows due to our windows jobs always installing chrome latest stable. The flake for these jobs seemed to get bad right around late October of 2023, which coincides with the Chrome 119 release date.

AtofStryker · 2024-04-26T14:46:09Z

yarn cypress:run:e2e -b chrome --spec=./cypress/e2e/runner/reporter-ct-mount-hover.cy.ts,./cypress/e2e/runner/reporter-ct-vite.errors.cy.ts --headed

reproduces the error every time, regardless of environment. You can add a fey .onlys to speed up the feedback loop with the result being the same. However, running just the spec by itself is fine, like this:

yarn cypress:run:e2e -b chrome --spec=./cypress/e2e/runner/reporter-ct-vite.errors.cy.ts --headed

My guess is there is something going on in the cri-client tab creation code that has changed here.

AtofStryker · 2024-04-26T17:42:22Z

I think I figured out what is going on. So when rehydraying the page with our inner cypress, we request to to not bucket our injected code into a keyed origin agent cluster via the Origin-Agent-Cluster=?0 header on the response. However, traffic to that proxy URL has already gone through without that header being set, and Chrome 119 and above wants the header configured for all requests on that origin. Because we don't, the request gets bucketed anyway, when it shouldn't, and doesn't allow us to set document.domain, which causes all sorts of issues

To work around this, we can either set chromeWebSecurity to false or we can set the response header on the proxy target for cypress in cypress tests. I have elected to do the later which is implemented in 324475e.

This fix also looks to be fixing the windows job failures we saw in #28942, which I will re open and link here, which enables recording again for the windows cypress in cypress jobs 🎉

…run ci]

… 124 and is ultimately out of our control since the options passed into the CLI are STILL not respected [run ci]

packages/proxy/test/unit/http/response-middleware.spec.ts

…date_internal_browser_images

packages/proxy/lib/http/response-middleware.ts

mschile · 2024-04-30T15:00:41Z

packages/proxy/lib/http/response-middleware.ts

+      // behavior starting in Chrome 119. The new behavior works like the following:
+      //     If a page did not request an origin-keyed agent cluster, chrome will place it in one
+      //     anyway because a previous request went through without the Origin-Agent-Cluster=false header set.
+      //     At time of writing, documentation detailing this behavior has not been found.


I believe the documentation is here:

The browser will ensure that all pages from a given origin are either origin-keyed or they are not. This means that:

If the first page from an origin does not set the header, then no other pages from that origin will be origin-keyed, even if those other pages do set the header.

If the first page from an origin sets the header and is made origin-keyed, then all other pages from that origin will be origin-keyed whether they ask for it or not.

referenced the docs and replaced the text in 96ee05c

packages/proxy/lib/http/response-middleware.ts

…ypress-io#29391) * chore: update browser internal images to chrome 124 and firefox 125 [run ci] * update screenshot sizes for chrome as they have changed in the chrome 124 and is ultimately out of our control since the options passed into the CLI are STILL not respected [run ci] * fix cypress-in-cypress tests for chrome 119 and up * remove windows app integration code added in 29198 * run windows build [run ci] * fix some failing system tests [run ci] * update non proxied png [run ci] * fix system tests failing [run ci] * fix afterEach [run ci] * chore: update documentation in response middleware

cypress-bot · 2024-05-08T13:17:33Z

Released in 13.9.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v13.9.0, please open a new issue.

AtofStryker marked this pull request as draft April 23, 2024 19:59

AtofStryker force-pushed the chore/update_internal_browser_images branch from 63be1e3 to b9a95eb Compare April 23, 2024 20:38

AtofStryker force-pushed the chore/update_internal_browser_images branch from e44d048 to 9366673 Compare April 24, 2024 15:35

jennifer-shehane mentioned this pull request Apr 25, 2024

fix: #29093 for elements with length in rem #29224

Open

AtofStryker force-pushed the chore/update_internal_browser_images branch 2 times, most recently from 3df3b9f to 1bc9b3c Compare April 26, 2024 17:34

AtofStryker added 8 commits April 26, 2024 17:45

chore: update browser internal images to chrome 124 and firefox 125 […

a617d42

…run ci]

update screenshot sizes for chrome as they have changed in the chrome…

6ae7347

… 124 and is ultimately out of our control since the options passed into the CLI are STILL not respected [run ci]

fix cypress-in-cypress tests for chrome 119 and up

e82d776

remove windows app integration code added in 29198

fbc0b9b

run windows build [run ci]

308c2ad

fix some failing system tests [run ci]

ec818c8

update non proxied png [run ci]

fdc7cbc

fix system tests failing [run ci]

98cfed6

AtofStryker force-pushed the chore/update_internal_browser_images branch from 5957a09 to 98cfed6 Compare April 26, 2024 21:45

AtofStryker commented Apr 26, 2024

View reviewed changes

packages/proxy/test/unit/http/response-middleware.spec.ts Outdated Show resolved Hide resolved

fix afterEach [run ci]

e2a1636

AtofStryker changed the title ~~chore: update browser internal images to chrome 124 and firefox 125 [run ci]~~ chore: update browser internal images to chrome 124 and firefox 125 Apr 29, 2024

AtofStryker mentioned this pull request Apr 29, 2024

chore: add internal docker images for chrome 124 and firefox 125 cypress-io/cypress-docker-images#1040

Merged

AtofStryker marked this pull request as ready for review April 29, 2024 19:12

cacieprins approved these changes Apr 29, 2024

View reviewed changes

Merge branch 'develop' of github.com:cypress-io/cypress into chore/up…

0d9a3c1

…date_internal_browser_images

mschile reviewed Apr 30, 2024

View reviewed changes

packages/proxy/lib/http/response-middleware.ts Outdated Show resolved Hide resolved

mschile reviewed Apr 30, 2024

View reviewed changes

packages/proxy/lib/http/response-middleware.ts Outdated Show resolved Hide resolved

chore: update documentation in response middleware

96ee05c

AtofStryker requested a review from mschile April 30, 2024 15:34

mschile approved these changes Apr 30, 2024

View reviewed changes

AtofStryker merged commit 6937190 into develop Apr 30, 2024
86 of 93 checks passed

AtofStryker deleted the chore/update_internal_browser_images branch April 30, 2024 18:16

cypress-bot bot locked as resolved and limited conversation to collaborators May 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: update browser internal images to chrome 124 and firefox 125 #29391

chore: update browser internal images to chrome 124 and firefox 125 #29391

AtofStryker commented Apr 23, 2024 •

edited

cypress bot commented Apr 23, 2024 •

edited

jennifer-shehane commented Apr 25, 2024

AtofStryker commented Apr 26, 2024

AtofStryker commented Apr 26, 2024

AtofStryker commented Apr 26, 2024

AtofStryker commented Apr 26, 2024

mschile Apr 30, 2024

AtofStryker Apr 30, 2024

cypress-bot bot commented May 8, 2024

chore: update browser internal images to chrome 124 and firefox 125 #29391

chore: update browser internal images to chrome 124 and firefox 125 #29391

Conversation

AtofStryker commented Apr 23, 2024 • edited

Additional details

Steps to test

How has the user experience changed?

PR Tasks

cypress bot commented Apr 23, 2024 • edited

7 failed and 4 flaky tests on run #55240 ↗︎

Review all test suite changes for PR #29391 ↗︎

jennifer-shehane commented Apr 25, 2024

AtofStryker commented Apr 26, 2024

AtofStryker commented Apr 26, 2024

AtofStryker commented Apr 26, 2024

AtofStryker commented Apr 26, 2024

mschile Apr 30, 2024

Choose a reason for hiding this comment

AtofStryker Apr 30, 2024

Choose a reason for hiding this comment

cypress-bot bot commented May 8, 2024

AtofStryker commented Apr 23, 2024 •

edited

cypress bot commented Apr 23, 2024 •

edited