[Bug]: "Waiting for target frame x failed" when crawling a lot of websites concurrently

[stevenwdv]

Bug description

Steps to reproduce the problem:

1. Do a lot at the same time (opening a lot of browsers and visiting pages etc.), causing Node to lag.
2. Observe an unhandledRejection like the following:

TimeoutError: Waiting for target frame 38DAB13DFEC9AD09315770D838D9F8C9 failed
    at Timeout._onTimeout (path\to\my_package\node_modules\puppeteer\src\util\DeferredPromise.ts:36:14)
    at listOnTimeout (node:internal/timers:564:17)
    at processTimers (node:internal/timers:507:7)

This is caused by the promise from FrameManager.ts:153 not being fulfilled within 5 seconds (hardcoded in DeferredPromise.ts). If you add if (isRejected) console.log('resolved anyway') to resolve: in DeferredPromise.ts and set process.on('uncaughtException', console.error) then you can see that the promise will be resolved anyway later on.

I think the timeout should probably be configurable in some way (or removed entirely, but it probably serves a purpose).

Puppeteer version

16.2.0

Node.js version

18.7.0

npm version

8.17.0

What operating system are you seeing the problem on?

Windows

Relevant log output

No response

[stevenwdv]

Update: I sometimes have this issue even when I change the timeout to 30 seconds while doing just 5 crawls concurrently, and I can see that the promise would actually be fulfilled eventually.

[ivibe]

The worst part - there is no way to catch this error. If it happens node crashes.

[stevenwdv]

@ivibe Except process.on('uncaughtException', console.error) that is, but yeah that's a bit evil.
--unhandled-rejections=warn may also work

[ivibe]

@stevenwdv yep, there're workarounds, but I meant traditional try ... catch block :)

[stevenwdv]

@OrKoN I just checked and unfortunately the issue is not fixed in puppeteer 17.1.0. This seems to be a regression compared to 17.0.0. I can reproduce it with the snippet from #8838.

TimeoutError: Waiting for target frame 94B6342A551DB27D1F1D5A185A000C34 failed
    at Timeout._onTimeout (file:///C:/dev/puppeteer-test/node_modules/puppeteer/lib/esm/puppeteer/util/DeferredPromise.js:26:22)
    at listOnTimeout (node:internal/timers:564:17)
    at process.processTimers (node:internal/timers:507:7)

Can this issue be reopened?

[OrKoN]

Sorry, looks like a regression slipped in the @jrandolf's refactoring #8863

[NoelToy]

Hi there am using puppeteer version 17.1.0 and Node 18.10.0. Am facing the exact same issue when I am doing a web scrapping concurrently. Is there any fix or workaround for this? It would be so helpful if it can be address as we are facing the issue in a production system.

[stevenwdv]

@NoelToy
Workaround: Reduce concurrency, only scrape a couple of sites in parallel.
Fix: Already released, just upgrade puppeteer to v17.1.1 or higher, you're on a really old version.

[NoelToy]

Thanks, I have already changed the version to 17.1.1 and the issue got resolved. Thanks once again.