Support Network requests in Workers

[aslushnikov]

WebWorkers can fetch data; we should surface traffic from web workers in puppeteer.

This is blocked on #2548 since nested targets are broken with request interception. See #2717 (comment) for details.

[rylyade1]

I know this issue is blocked on #2548 but is there a time frame for this? If not could, does anyone have any suggestion on a workaround?

[aslushnikov]

I know this issue is blocked on #2548 but is there a time frame for this? If not could, does anyone have any suggestion on a workaround?

@rylyade1: there's no ETA. Regarding the workaround, you can attach raw devtools protocol session with target.createCDPSession() and enable network domain manually there. It should give you some events.

[wclr]

@aslushnikov

Regarding the workaround, you can attach raw devtools protocol session with target.createCDPSession() and enable network domain manually there. It should give you some events.

Could you elaborate on this: enable network domain manually there.?

[JBoudMS]

@aslushnikov

Regarding the workaround, you can attach raw devtools protocol session with target.createCDPSession() and enable network domain manually there. It should give you some events.

Could you elaborate on this: enable network domain manually there.?

Not sure if it would still be useful 3 months later, but in case anybody else finds it useful...

This is based on https://jarrodoverson.com/post/using-chrome-devtools-protocol-with-puppeteer-737a1300bac0/ but updated to use Fetch as Network has been deprecated. It is working with "puppeteer": "^1.18.1"

/**
 * Captures all traffic including from Web Workers, does something with it, and continues the request
 * @param page The page/tab to capture from.
 */
const interceptAllTrafficForPageUsingFetch = async (page) => {
    if (page) {
        const client = await page.target().createCDPSession();
        //see: https://chromedevtools.github.io/devtools-protocol/tot/Fetch#method-enable
        await client.send('Fetch.enable', {
            //see: https://chromedevtools.github.io/devtools-protocol/tot/Fetch#type-RequestPattern
            patterns: [{
                urlPattern: '*',
                requestStage: 'Response'
            }]
        });
        //see: https://chromedevtools.github.io/devtools-protocol/tot/Fetch#event-requestPaused
        await client.on('Fetch.requestPaused', async ({ requestId, request, responseHeaders }) => {
            const { url } = request;
            console.log(`Intercepted ${request.url}`)
            //see: https://chromedevtools.github.io/devtools-protocol/tot/Fetch#method-getResponseBody
            const responseBody = await client.send('Fetch.getResponseBody', {
                requestId
            });

            //Do something like saving the response to disk

            //see: https://chromedevtools.github.io/devtools-protocol/tot/Fetch#method-continueRequest
            await client.send('Fetch.continueRequest', {
                requestId
            })
        })
    }
}
/**
 * Launches a puppeteer controlled Chrome and intercepts all traffic.
 */
(async () => {
     const browser = await launch({
        headless: false,
        defaultViewport: VIEWPORT,
    });
    const page = (await browser.pages())[0];

    await interceptAllTrafficForPageUsingFetch(page);
    browser.on('targetcreated', async (target) => {
        const page = await target.page();
        await interceptAllTrafficForPageUsingFetch(page);
    });

   //Normal code like navigation, closing browser session, etc.
})

[mccolljr]

It's now October 2019, I take it this is still not enabled? Is there an ETA?

[tomardern]

Hi all, I'm getting the same issue here, we can't intercept network requests when they are being called from a service worker.

However, using page.setRequestInterception(true) does block the request, we just can't manipulate it or do anything with it:

page.on('request', interceptedRequest => {
 // Nothing shown here for network requests via service workers
})

For now, in our case, we can disable service workers for our page. To do this,

I've added the following to our setup script:

await page.evaluateOnNewDocument(`window.Worker = undefined;`);

[anjackson]

The code above didn't seem to work for me, as there was no page associated with a ServiceWorker target. Managing the Targets directly did work, like this:

/**
 * Captures all traffic including from Web Workers, does something with it, and continues the request
 * @param target The page/tab/worker target to capture from.
 */
const interceptAllTrafficForPageUsingFetch = async (target) => {
    if (target) {
        const client = await target.createCDPSession();
        //see: https://chromedevtools.github.io/devtools-protocol/tot/Fetch#method-enable
        await client.send('Fetch.enable', {
            //see: https://chromedevtools.github.io/devtools-protocol/tot/Fetch#type-RequestPattern
            patterns: [{
                urlPattern: '*',
                requestStage: 'Response',
            }]
        });
        //see: https://chromedevtools.github.io/devtools-protocol/tot/Fetch#event-requestPaused
        await client.on('Fetch.requestPaused', async ({ requestId, request }) => {
            const { url } = request;
            console.log(`Intercepted ${request.url}`);

            //Do something like saving the response to disk

            //see: https://chromedevtools.github.io/devtools-protocol/tot/Fetch#method-continueRequest
            await client.send('Fetch.continueRequest', {
                requestId
            })
        })
    }
}
/**
 * Launches a puppeteer controlled Chrome and intercepts all traffic.
 */
const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  const target = await page.target();
  await interceptAllTrafficForPageUsingFetch(target);
  browser.on('targetcreated', async (target) => {
      await interceptAllTrafficForPageUsingFetch(target);
  });

...

[akorajac]

any update on this?

[Francoyy]

Following this thread regarding extensions testing. Our extension is now using Chrome's Manifest V3 architecture which uses service workers instead of background scripts.
I want to monitor the requests made by my service worker, but it doesn't seem to work as it did previously with background scripts.
await serviceWorkerTarget.setRequestInterception(true);
throws the error TypeError: serviceWorkerTarget.setRequestInterception is not a function

[CodingMeSwiftly]

Any update on this?

[martiL]

Any update on this?

[agugut-nexgen]

Hello People Any update on this?

[LastPlayerTR]

Any update?

[sandstrom]

A popular pattern for securely storing credentials in single page apps is to use a web worker. For example, see this guide:

https://engineering.mercari.com/en/blog/entry/20220930-building-secure-apps-using-web-workers/

Unfortunately, a lot of testing frameworks rely on Puppeteer, and writing tests for this code is hard since support is lacking in Puppeteer.

[elmaester]

Please provide a way to intercept extension / service worker v3 requests!