Full Network Layer Stubbing / fetch support

[brian-mann]

Pull Request (in progress): #4176

What users want

• To be able to stub any kind of HTTP request including window.fetch
  • note: temporary polyfill available polyfill window.fetch  #7710 v4.9.0
• To blacklist certain domains (google analytics, new relic, intercom)
• To have more control over dynamically routing / stubbing requests and responses
• To more easily stub graphQL calls

What we do now

• We modify the host XMLHttpRequest object which limits us to only stubbing XHR's from the main application frame.

What we need to do

• Throw away the entire existing system for stubbing
• Move stubbing into the internal Cypress network proxy layer (outside of the browser)
• Synchronize the state of the browser tests with the backend (so its aware what needs to be stubbed / routed)
• Remove cy.server and instead just use cy.route
• Expand the API's for more advanced configuration
• Accept blacklistDomains option for cypress.json
• As each request comes in, if we expose callback functions we'll need to constantly communicate with the driver for guidance on what to do - whether to proceed - halt, etc. This can get pretty complicated and might have significant performance issues. We'll need to experiment and we may not be able to provide callback functions anymore.

Things to consider

• onRequest and onResponse currently provide the raw level xhr object. This would no longer be possible. We'd have to serialize the request / response into basic objects. No method calling would be available.
• The onError callback could no longer exist
• Additionally we hook into the onerror events for XHR's and will automatically fail the test if this is called. This would no longer be possible.
• Blacklisting https cannot be dynamic. It would have to be set in cypress.json or environment variables. The reason is that for each blacklisted domain that goes over https, we would have to route that traffic to a self signed certificate. Browser freak out if traffic to https gets assigned one certificate, and then within the same session it gets another cert applied. What this means is that before spawning the browser we have to isolate that traffic and know ahead of time not to route it to the actual server. This enables us to send a single cert to the browser.
• We could enable non https hosts to be dynamically blacklisted, but these API's would be async and likely difficult for users to understand / use. We might be able to bake these into cy.route but I'm not sure there's much of a use case here of only dynamically blacklisting non https traffic.
• We will no longer be able to tell the difference between an XHR and any other kind of http request. We'd likely have to change the page events to say REQ and REQ STUB instead of XHR and XHR STUB.
• It would be very difficult to provide accurate onResponse callback functions. It's possible for the internal server to know when an HTTP request has been completed, but it's not possible to know whether the browser has actually finished processing it. This may or may not be a problem.
• How would we handle stubbing binary / streaming responses?
• How would we handle stubbing things like images? (This is likely pretty simple)

Bonus Points

• It would likely be possible to stub Websocket traffic in the same manner
• The problem with this is that most Websocket libraries (ie socket.io) add their own "encoding" pattern on top of each websocket message frame. The ergonomics around stubbing this would be odd. The problem is this is library (and even version of library) specific. This may be a better use case for cy.stub, or perhaps a 3rd party extension

[brian-mann]

Related to:

• [Feature] blacklistHosts should return failed or similar #5084
• Allow modify html content before rendering cy.visit #4211
• Prevent Cypress from resolving stubs with cy.route #4163
• Add response alias support (res.alias) #3083
• Allowing setting Chrome bandwidth and latency setting #2716
• Support mocking of iframe document response #2818
• Ability to test bad internet connection by allowing cy.route never to return a response #1700
• Prevent/Stub network request #1370
• Add ability to wait for script to load (like cy.script('js') + cy.wait()) #1296
• Allow reusing cy.server() in several tests and hooks #1183
• Not native behavior of XMLHttpRequest inside test enviroment #1080
• Partial route mock #976
• window.top goes to Cypress instead of immediate parent iframe #969
• Remove cy.server() boilerplate #439
• Add blacklisting of requests made to specific hosts #442
• Allow dynamically matching requests with a function #387
• Support lazy evaluation for routes #259
• bi-directional WebSocket support and/or message queue/pools #199
• Enable stubbing all kinds of HTTP requests (not just XHR) #192
• Modify headers and status code but not stubbing response #190
• Return conditional data based an XHR's request details #180
• Check for background traffic #121
• implement window.fetch capturing/mocking #95
• Allow silencing of cy.server() in Command Log #79
• Large XHR response objects can exceed the maximum header size. #76

[ericadamski]

@brian-mann I would be very willing to help out with this issue. This is a big blocker for use when testing our app.

[brian-mann]

@ericadamski could you go into more detail about this? What exactly is a big blocker?

[ericadamski]

@brian-mann making network requests (using a fetch polyfill) and asserting on those requests is important to our integration testing. Currently we are just spying on the window.fetch and doing some simple assertions on the params request. It would be ideal to provide the same functionality as XHRs to completely omit the server and mock responses when needed, understanding that we could stub the fetch as well, but this means stubbing each individual request with a tailored response.

[brian-mann]

You can do this right now using stubs. Albeit it's certainly not as nice as what we did for XHR stubbing, but it is possible and we have several examples of making this work. This is even in a new getting started video we're releasing later this week.

https://github.com/cypress-io/cypress-example-recipes/blob/master/cypress/integration/spy_stub_clock_spec.js

[ericadamski]

Yes we realize it can be done, albeit we would rather wait (or work toward) a more XHR style stubbing, it is just super nice.

[brian-mann]

Yeah it is. Okay just making sure we're on the same page.

The proposal here does a decent job outlining what needs to be done but it will be a significant amount of work.

Likely the next steps just need to be formalizing a plan for the API's themselves. There have been many requests for additional features and we should start there.

[ericadamski]

That sounds good.

Is it worth creating separate issues for some of the requested features for I know almost nothing about graphQL queries 😜

[brian-mann]

They mostly already are other issues. We need to factor in all of them and come up with a set of API's that unifies those different use cases and desired experiences.

[ericadamski]

Right OK, I see what you are saying.

[ericadamski]

@brian-mann side note, has it been thought of moving away from coffeescript and into es6?

[paulpruteanu]

Do you consider this feature in the next release? I reckon it would be highly valuable, for instance when

• authentication/ other actions are performed at script level (<script src/> instead of XHR)
• needing to bypass other scripts/ pixel images which takes a lot of time to load

[brian-mann]

Yeah blacklisting specific requests is part of this feature roadmap

[bahmutov]

Temporary experimental window.fetch polyfill available in v4.9.0

[jennifer-shehane]

The experimentalFetchPolyfill configuration option has been released as part of 4.9.0. When this option is true, Cypress will automatically replace window.fetch with a polyfill that Cypress can spy on and stub.

You should be able to remove any of the following code regarding fetch support:

Cypress.on('window:before:load', (win) => {	
  delete win.fetch	
})	

or

cy.visit('/', {	
  onBeforeLoad (win) {	
    delete win.fetch	
  },	
})	

And replace it with the below in your configuration file (cypress.json)

{
  "experimentalFetchPolyfill": true
}

You can see a full example of this in use here: https://github.com/cypress-io/cypress-example-recipes/tree/master/examples/stubbing-spying__window-fetch

If you encounter any issues with the new experimentalFetchPolyfill, please open a new issue with a reproducible example.

[alexkrolick]

Is it possible to stub network calls in a way that that doesn't care about whether the app uses fetch or XHR? Seems like in most cases that is an implementation detail we don't want to enshrine in a test.

[ngbrown]

I was using xhook for stubbing and unfetch to polyfill fetches for Cypress. It doesn't appear that xhook is compatible with experimentalFetchPolyfill.

Using xhook with experimentalFetchPolyfill resulted in every network request failing. Once I removed xhook and just used experimentalFetchPolyfill, only the tests that relied on a stubbed response failed. So still waiting for #4176.

Looks good otherwise.

[jennifer-shehane]

@alexkrolick We're working on bigger changes as part of #4176 to improve the experience behind network requests in general. The current workaround is meant to be helpful for users starting out so they don't have to dive through issues to figure out how to get fetch working.

[bahmutov]

@ngbrown could you please provide a small reproducible example showing how experimentalFetchPolyfill fails?

[samtsai]

I was using xhook for stubbing and unfetch to polyfill fetches for Cypress. It doesn't appear that xhook is compatible with experimentalFetchPolyfill.

Using xhook with experimentalFetchPolyfill resulted in every network request failing. Once I removed xhook and just used experimentalFetchPolyfill, only the tests that relied on a stubbed response failed. So still waiting for #4176.

Looks good otherwise.

cc @bahmutov

xhook replaces fetch as well and changes the interface, you can support both if you properly restore native/polyfilled-fetch after enabling xhook, something like this below:

return cy
    .readFile("node_modules/xhook/dist/xhook.min.js", { log: false })
    .as("xhook")
    .then((xhook) => {
      Cypress.on("window:before:load", (win) => {
        // save original fetch (which is probably Cypress's polyfill)
        const originalFetch = win.fetch
        win.eval(xhook)
        win.fetch = originalFetch // make sure we restore `fetch` to original
        win.xhook.before((req) => {
          let url = req.url
          if (url.indexOf("someapi") > -1 && req.body) {
            // do stuff
          }
        })
      })
    })

[samtsai]

I was using xhook for stubbing and unfetch to polyfill fetches for Cypress. It doesn't appear that xhook is compatible with experimentalFetchPolyfill.
Using xhook with experimentalFetchPolyfill resulted in every network request failing. Once I removed xhook and just used experimentalFetchPolyfill, only the tests that relied on a stubbed response failed. So still waiting for #4176.
Looks good otherwise.

cc @bahmutov

xhook replaces fetch as well and changes the interface, you can support both if you properly restore native/polyfilled-fetch after enabling xhook, something like this below:

return cy
    .readFile("node_modules/xhook/dist/xhook.min.js", { log: false })
    .as("xhook")
    .then((xhook) => {
      Cypress.on("window:before:load", (win) => {
        // save original fetch (which is probably Cypress's polyfill)
        const originalFetch = win.fetch
        win.eval(xhook)
        win.fetch = originalFetch // make sure we restore `fetch` to original
        win.xhook.before((req) => {
          let url = req.url
          if (url.indexOf("someapi") > -1 && req.body) {
            // do stuff
          }
        })
      })
    })

So this turns out not to be as straightforward as I thought. I finally have the two working in my project but it required me to patch xhook if anyone is interested in I can share.

[joshribakoff]

It would be very difficult to provide accurate onResponse callback functions. It's possible for the internal server to know when an HTTP request has been completed, but it's not possible to know whether the browser has actually finished processing it. This may or may not be a problem.

You can correlate it by hashing content length or other things like the URL & use https://developers.google.com/web/tools/chrome-devtools/network/resource-loading to have Cypress "see" in the network tab if the network request finished.

To play devil's advocate, it could also be argued people should not test in this way, because a normal user would not see if a network request has finished, and tests should closely match how a user behaves (maybe waiting for some element on the page to update in response to the network request finishing instead).

[cypress-bot]

Released in 5.1.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v5.1.0, please open a new issue.

[jennifer-shehane]

The features requested in this issue are now possible as part of cy.route2().

This is currently experimental and requires being enabled by passing "experimentalNetworkStubbing": true through your Cypress configuration. This will eventually be merged in as part of our standard API.

Please see the cy.route2() docs for full details: https://on.cypress.io/route2

If you encounter any issues or unexpected behavior while using cy.route2() we encourage you to open a new issue so that we can work out all the issues before public release. Thanks!