Request interceptions cause web workers network requests to hang

[Schniz]

When setting a request interception to a page, the page's workers can no longer make network request, because otherwise they will stop responding. both importScripts and fetch.

It is critical to mention that this behavior does work on native CDP

Steps to reproduce

I've created a repo with a script that does the following, and also shows that a "native" implementation using CDP alone does work:

• Serve a static site with worker.js that makes a network call of some sort
• Create a new browser page with interceptions:

  await page.setRequestInterception(true);
  await page.on('request', request => request.continue);

• See how it causes the worker to stop responding!

Tell us about your environment:

• Puppeteer version: 1.13.0
• Platform / OS version: OSX
• URLs (if applicable):
• Node.js version: v10.15.3

What steps will reproduce the problem?

Please include code that reproduces the issue.

1. Clone https://github.com/Schniz/puppeteer-workers-bug
2. npm i
3. node puppeteer.js

What is the expected result?

The following works:
--------------------
1
{
  "userId": 1,
  "id": 1,
  "title": "delectus aut autem",
  "completed": false
}
finished

But when intercepted, it doesn't!
--------------------------
1

Although it does work with CDP directly:
----------------------------------------
http://localhost:5000/
http://localhost:5000/worker.js
https://jsonplaceholder.typicode.com/todos/1
1
{
  "userId": 1,
  "id": 2,
  "title": "quis ut nam facilis et officia qui",
  "completed": false
}
finished

What happens instead?

The following works:
--------------------
1
{
  "userId": 1,
  "id": 1,
  "title": "delectus aut autem",
  "completed": false
}
finished

But when intercepted, it doesn't!
--------------------------
1

Although it does work with CDP directly:
----------------------------------------
1
{
  "userId": 1,
  "id": 2,
  "title": "quis ut nam facilis et officia qui",
  "completed": false
}
finished

[Schniz]

I'm closing because it's a dupe of #2781

[aslushnikov]

Let me reopen this - looks like a bug to me. Should be addressed with the work we do to support #2548.

[yurynix]

It seems that Network.requestWillBeSent is not fired from CDP for the XHR in the worker...

this hack “fixes” it:

--- a/lib/NetworkManager.js
+++ b/lib/NetworkManager.js
@@ -187,6 +187,7 @@ class NetworkManager extends EventEmitter {
       this._requestIdToRequestWillBeSentEvent.delete(requestId);
     } else {
       this._requestHashToInterceptionIds.set(requestHash, event.interceptionId);
+      this._onRequest(event, null);
     }
   }
 
@@ -306,7 +307,7 @@ class Request {
     this._failureText = null;
 
     this._url = event.request.url;
-    this._resourceType = event.type.toLowerCase();
+    this._resourceType = (event.type || event.resourceType).toLowerCase();
     this._method = event.request.method;
     this._postData = event.request.postData;
     this._headers = {};

Not sure what a proper fix to that issue though.

[SilurianYang]

I did it! When the desired effect is achieved, remove the intercept

    await page.setRequestInterception(true)
    page.on('request', async function jsRequest(res) {
        if (/modules\/cube-loader\/cube-loader\.min\.js(?=\?.*)/.test(res.url())) {
            const jsCont = getExtendsJs('src/extendsJS/cube-loader.min.js');
            await res.respond({
                status: 200,
                contentType: 'application/javascript; charset=utf-8',
                body: jsCont
            });
            page.removeListener('request',jsRequest);
            await page.setRequestInterception(false)
            return false
        }
        res.continue();
    });

[foolip]

I seem to have hit this issue as well, and the workaround in #4208 (comment) worked. Thanks @SilurianYang!

[timothywangdev]

I'm having the same issue, is there a plan to fix the issue?

[timothywangdev]

It seems that Network.requestWillBeSent is not fired from CDP for the XHR in the worker...

this hack “fixes” it:

--- a/lib/NetworkManager.js
+++ b/lib/NetworkManager.js
@@ -187,6 +187,7 @@ class NetworkManager extends EventEmitter {
       this._requestIdToRequestWillBeSentEvent.delete(requestId);
     } else {
       this._requestHashToInterceptionIds.set(requestHash, event.interceptionId);
+      this._onRequest(event, null);
     }
   }
 
@@ -306,7 +307,7 @@ class Request {
     this._failureText = null;
 
     this._url = event.request.url;
-    this._resourceType = event.type.toLowerCase();
+    this._resourceType = (event.type || event.resourceType).toLowerCase();
     this._method = event.request.method;
     this._postData = event.request.postData;
     this._headers = {};

Not sure what a proper fix to that issue though.

this approach does not seem to work in 2.0 though

[felixfbecker]

Having the same issue, importScript() is just handling and not triggering the request interceptor. The workaround in #4208 (comment) doesn't solve the problem if you need to intercept the requests that happen inside the worker.

[stale]

We're marking this issue as unconfirmed because it has not had recent activity and we weren't able to confirm it yet. It will be closed if no further activity occurs within the next 30 days.

[srstsavage]

Still hitting this in 15.4.0

[joonamo]

I'm hitting this as well. I worked around the issue by inlining my worker script as base64 data url using Parcel's bundle inlining. Also, my worker doesn't need to send any requests. It's not pretty or production friendly, but thankfully my page is only used for testing.

[smithki]

I've run into this problem even when my web worker does not send network requests. Removing request interception resolves the problem, though it's not ideal.

[stale]

We're marking this issue as unconfirmed because it has not had recent activity and we weren't able to confirm it yet. It will be closed if no further activity occurs within the next 30 days.

[sidvishnoi]

This issue still persists.

[stale]

We're marking this issue as unconfirmed because it has not had recent activity and we weren't able to confirm it yet. It will be closed if no further activity occurs within the next 30 days.

[sidvishnoi]

Is there an upstream bug for this? The issue still exists in v19

[OrKoN]

@sidvishnoi I don't think there is an upstream bug. I am not quite sure it's an upstream bug at this point: perhaps the Network.requestWillBeSent is dispatched on the worker target while the Network.requestIntercepted events are dispatched on the page target. I'd be great if someone could dig into it: I will see if I find time.

[OrKoN]

It appears that either something in Puppeteer fixed it or more likely upstream changes. I am not able to reproduce it with 22.6.3