feat(api): implement Page.waitForNetworkIdle()

[tjenkinson]

Which will wait for there to be no network requests in progress during the idleTime before resolving.

One case where this is useful is if you want to take a screenshot after adding a css class that results in more assets being downloaded (background images etc).

refs #3083

TODO

• The tests currently use a public url, because for some reason when using "/digits/*.png" the RequestFinished/RequestFailed events aren't fired, and the request isn't removed from _requestIdToRequest, even though the fetch() call resolves. Any idea why? Is this a chromium bug?
• Is firefox support needed too?

[stevezau]

any status on this?

[davispuh]

I'm also interested in this. Super useful feature when want to do some action after everything has finished loading.

[davispuh]

Just found this https://github.com/jtassin/pending-xhr-puppeteer looks exactly what's needed.

[alvis]

I'm trying to implement something similar and also realise that sometimes the RequestFinished/RequestFailed events aren't fired. It seems like to be that some document type requests get cancelled before it's fulfilled and in this case no event is emitted.

[jschfflr]

@tjenkinson Thanks for your contribution! Could you resolve the merge conflicts and verify if local urls might be working now? Happy to get this landed :)

[tjenkinson]

@jschfflr I brought it back up to date, but it still passes locally with the remote urls but fails with the digit ones :/

The tests currently use a public url, because for some reason when using "/digits/*.png" the RequestFinished/RequestFailed events aren't fired, and the request isn't removed from _requestIdToRequest, even though the fetch() call resolves. Any idea why? Is this a chromium bug?

I'll have a look if this is still the cause later

[tjenkinson]

Fixed it :)

But I think there may be a memory leak/bug.

Currently

puppeteer/src/common/NetworkManager.ts

Lines 423 to 434 in 8d1a697

	_forgetRequest(request: HTTPRequest, events: boolean): void {
	const requestId = request._requestId;
	const interceptionId = request._interceptionId;
	this._requestIdToRequest.delete(requestId);
	this._attemptedAuthentications.delete(interceptionId);
	if (events) {
	this._requestIdToRequestWillBeSentEvent.delete(requestId);
	this._requestIdToRequestPausedEvent.delete(requestId);
	}
	}

is not called if the response is from the cache, meaning anything from the cache stays in

puppeteer/src/common/NetworkManager.ts

Line 111 in 8d1a697

_requestIdToRequest = new Map<string, HTTPRequest>();

I wonder if there should be a _forgetRequest call in

puppeteer/src/common/NetworkManager.ts

Lines 414 to 421 in 8d1a697

	_onResponseReceived(event: Protocol.Network.ResponseReceivedEvent): void {
	const request = this._requestIdToRequest.get(event.requestId);
	// FileUpload sends a response without a matching request.
	if (!request) return;
	const response = new HTTPResponse(this._client, request, event.response);
	request._response = response;
	this.emit(NetworkManagerEmittedEvents.Response, response);
	}

?

[jschfflr]

Thanks for updating the commit!
It does indeed look like a memory leak. But calling _forgetRequest in _onResponseReceived might cause problems because there's also the ResponseReceivedExtraInfo event that might arrive before or after ResponseReceived.
Can you file an issue for that and assign it to me as there's currently some work being done to improve support for ResponseReceivedExtraInfo to unblock rolling the latest Chromium.

[tjenkinson]

No problem, it's nice to finally get this added :)

Here's the ticket: #7560. I don't have permission to assign it to you.

Also noticed you turned auto merge on, but it's not approved yet

[jschfflr]

@tjenkinson I just saw this test flake on the mac bot:
https://github.com/puppeteer/puppeteer/runs/3585119647?check_suite_focus=true

Could you take a look into why that happened and address the issue? Thanks!

[tjenkinson]

That's weird. I wonder if it lagged in a few places and it went over idleTime 🤔

Opened #7564 to see if that helps

[jschfflr]

Yeah I think the bots are not very powerful so timing might be problematic in some places. But it looks like the fix solved it :)