Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Headless mode consumes too much? #9934

Closed
2 tasks
Tonyx97 opened this issue Mar 28, 2023 · 2 comments
Closed
2 tasks

[Bug]: Headless mode consumes too much? #9934

Tonyx97 opened this issue Mar 28, 2023 · 2 comments

Comments

@Tonyx97
Copy link

Tonyx97 commented Mar 28, 2023

Bug expectation

Hi, I'm assuming headless mode removes all the rendering completely and so on so all it does is execute the js scripts and fill the HTML even though it's not being shown of course. I'm scraping websites, the problem is when I run everything in headless mode, if I open 30 pages (tabs) in the same browser context with headless = false then it will run smoothly, decent CPU usage. If I do the same but in headless it will use a lot of CPU, reaching 100% depending on the website you scrap. Now, the thing is that the refresh rate of javascript rendering and the HTML updates for each single tab seems to be the same in headless and non-headless both so why is it consuming so much in headless? Is there any option I don't know about to reduce the usage of each tab or something? Is there a way to "idle" (make it run less often?) the headless browser the same way the non-headless does? Thanks.

Bug behavior

  • Flaky
  • PDF

Minimal, reproducible example

const puppeteer = require("puppeteer");

async function main()
{
	const _browser = await puppeteer.launch({
		defaultViewport: null,
		headless: false,	// change to true to verify it consumes more
		ignoreHTTPSErrors: true,
		ignoreDefaultArgs: ["--enable-automation"],
		args:
		[
			'--kiosks',
     		'--disable-accelerated-2d-canvas',
			'--disable-backgrounding-occluded-windows',
			'--disable-renderer-backgrounding',
			'--disable-canvas-aa',
			'--disable-2d-canvas-clip-aa',
			'--disable-gl-drawing-for-tests',
			'--disable-dev-shm-usage', 
			'--disable-gpu',
			'--no-zygote', 
			'--use-gl=desktop', 
			'--hide-scrollbars',
			'--mute-audio',
			'--no-first-run',
			'--disable-infobars',
			'--disable-breakpad', 
			'--no-sandbox',
			'--disable-setuid-sandbox'
		],
	});
	
	let pages = [];

	for (let i = 0; i < 30; ++i)
	{
		let page = await _browser.newPage();

		await page.goto('https://store.steampowered.com/charts/mostplayed'); // use any website as an example
		
		pages.push(page);
	}

	while (true)
	{
		let promises = [];

		for (const page of pages)
		{
			async function parsePage()
			{
				await page.waitForNetworkIdle(); // this is optional
				
				let content = await page.content();
	
				console.log(content.length);
			};

			promises.push(parsePage());
			
			// do stuff with page
		}

		await Promise.all(promises);
	}
}

main();

Error string

no error

Puppeteer configuration

{
defaultViewport: null,
		headless: false,
		ignoreHTTPSErrors: true,
		ignoreDefaultArgs: ["--enable-automation"],
		args:
		[
			'--kiosks',
     		        '--disable-accelerated-2d-canvas',
			'--disable-backgrounding-occluded-windows',
			'--disable-renderer-backgrounding',
			'--disable-canvas-aa',
			'--disable-2d-canvas-clip-aa',
			'--disable-gl-drawing-for-tests',
			'--disable-dev-shm-usage', 
			'--disable-gpu',
			'--no-zygote', 
			'--use-gl=desktop', 
			'--hide-scrollbars',
			'--mute-audio',
			'--no-first-run',
			'--disable-infobars',
			'--disable-breakpad', 
			'--no-sandbox',
			'--disable-setuid-sandbox'
		]
}

Puppeteer version

v19.8.0

Node version

v18.15.0

Package manager

npm

Package manager version

9.5.0

Operating system

Windows

@github-actions
Copy link

github-actions bot commented Mar 28, 2023

This issue has an outdated Puppeteer version: 19.8.0. Please verify your issue on the latest 19.8.1 version. Then update the form accordingly.


Analyzer run

@OrKoN
Copy link
Collaborator

OrKoN commented Mar 29, 2023

You can try using the new headless mode: https://developer.chrome.com/articles/new-headless/ You should get the same behaviour as with the headful mode. As for the CPU consumption, it might have smth to do with the fact that you disable GPU (that might be ignored in headful). In any case, if you have a bug report about the headless performance, please open an issue at crbug.com as Puppeteer is only a client to the browser binary.

@OrKoN OrKoN closed this as not planned Won't fix, can't repro, duplicate, stale Mar 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants