Add integration tests for extension functionality (get Puppeteer to intercept web worker requests)

[felixfbecker]

Extensions are one of our most important features to test. The way we would test this is by serving a mock extension that registers very simple providers.

Unfortunately Puppeteer currently has a severe limitation: It can't intercept requests triggered from web workers, if request interception is on for their parent page they will just hang. This means extensions can't be easily loaded in integration tests. Neither importScripts() (which we use to load extension bundles) nor fetch() requests can be intercepted.

Issues in Puppeteer:

• Request interceptions cause web workers network requests to hang puppeteer/puppeteer#4208
• Support Network requests in Workers puppeteer/puppeteer#2781 (open since Jan 2018)

It seems there is a possible workaround by using the CDP through Puppeteer directly: puppeteer/puppeteer#2781 (comment)

This would be crucial to solve for an extensive integration test suite.

[davejrt]

Dear all,

This is your release captain speaking. 🚂🚂🚂

Branch cut for the 3.20 release is scheduled for tomorrow.

Is this issue / PR going to make it in time? Please change the milestone accordingly.
When in doubt, reach out!

Thank you

[marekweb]

I identified these issues with the WIP adapter:

• The request patterns don't intercept all requests properly, and I'm not sure they're intercept responses at all. This may explain why it's not working with Polly.
• Error when capturing the body of some responses. I think this is caused specifically by redirect responses, because they get paused in a different way (at a different stage?) than other requests.
• Error when page gets closed while we're still intercepting. This error may be safely ignored, but it's better if we ignore it explicitly.

Next steps:

I started writing a modified WIP that uses the CDP protocol directly instead of using the puppeteer interceptor library. With this approach I think I can solve all of the issue above. I'll push my work to this PR.

[marekweb]

The last comment above was specific to #13870. I posted a further update on that PR.