[Bug]: Chromium download over http proxy fails

[fairbanksg]

Bug description

Steps to reproduce the problem:

1. Have a proxy server setup, and set HTTP_PROXY and HTTPS_PROXY environment variables to point to it, both as http:// urls.
2. Install puppeteer
3. Download of Chromium during install fails with ERR_INVALID_PROTOCOL

It appears that this is happening because we are proxying HTTPS over an HTTP proxy.

Puppeteer version

5.5.0

Node.js version

16.9.1

npm version

7.21.1 (but we use yarn, version 1.22.5)

What operating system are you seeing the problem on?

Linux

Relevant log output

2021-09-23T19:25:57.805Z puppeteer:fetcher Downloading binary from https://storage.googleapis.com/chromium-browser-snapshots/Linux_x64/818858/chrome-linux.zip
ERROR: Failed to set up Chromium r818858! Set "PUPPETEER_SKIP_DOWNLOAD" env variable to skip download.
TypeError [ERR_INVALID_PROTOCOL]: Protocol "https:" not supported. Expected "http:"
    at new NodeError (node:internal/errors:371:5)
    at new ClientRequest (node:_http_client:158:11)
    at Object.request (node:https:353:10)
    at httpRequest (<REDACTED>/node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserFetcher.js:488:17)
    at downloadFile (<REDACTED>/node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserFetcher.js:357:21)
    at BrowserFetcher.download (<REDACTED>/node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserFetcher.js:239:19)
    at async downloadBrowser (<REDACTED>/node_modules/puppeteer/lib/cjs/puppeteer/node/install.js:48:5) {
  code: 'ERR_INVALID_PROTOCOL'
}

[fairbanksg]

puppeteer/src/node/BrowserFetcher.ts

Lines 608 to 611 in f2e1927

	const request =
	options.protocol === 'https:'
	? https.request(options, requestCallback)
	: http.request(options, requestCallback);

In our setup, the download url is https, and the proxy url is http. This line chooses whether to use the http or https module based on the protocol of the download url, not the proxy url, so it is choosing https but passing a http url.

[fairbanksg]

Actually it looks like this is a bug in https-proxy-agent
TooTallNate/proxy-agents#126

[jschfflr]

@fairbanksg Thanks for reporting this problem. We are in the process of improving our proxy support (
#6274) and I hope this will fix the problem.

[wjq990112]

Same problem.

[daemon110282]

Same problem.

[dragonfist453]

Same problem.

[dnltsk]

Same problem. Using npm 8.11.0.

[guellaeq]

Same problem. Using node 16.5.0. npm 7.19.1.

[willyboy]

Same

[abozas]

+1 for node 16 and 18

[Nexyll]

Same problem. Using node 16.14.2 and npm 8.5.0

[anthony-hull]

I worked around this by adding this to my package.json:

  "overrides": {
    "agent-base": "^6.0.2"
  }

[stale]

We're marking this issue as unconfirmed because it has not had recent activity and we weren't able to confirm it yet. It will be closed if no further activity occurs within the next 30 days.

[CoaxVex]

We're marking this issue as unconfirmed because it has not had recent activity and we weren't able to confirm it yet. It will be closed if no further activity occurs within the next 30 days.

Confirming that this is still a problem.

[noproxy]

This line need to be fixed:

options.agent = new ProxyAgent(proxyOptions); options.rejectUnauthorized = false;

A HttpsProxyAgent is required if the https.request is used.

See more here: https://stackoverflow.com/a/71482233

@jschfflr

[boindil]

+1

would be neat if this could be fixed.

[stale]

We're marking this issue as unconfirmed because it has not had recent activity and we weren't able to confirm it yet. It will be closed if no further activity occurs within the next 30 days.

[OrKoN]

I am unable to reproduce, it seems to be downloading fine with a proxy server I tried. The https-proxy-agent has been fixed.
If someone experiences this issue, please share the proxy setup for troubleshooting.

[icy0307]

@OrKoN
node v16.14.2
pnpm 8.3.1
pnpm config set https-proxy ANY_HTTP_PROXY

and this would reproduce this problem

[simin75simin]

if you have direct connection to Chinese npm mirrors, turn off vpn and do:
npm install cnpm -g --registry=https://registry.npmmirror.com
cnpm install puppeteer
windows 11, node 20.12.2, npm 10.5.0