Using a proxy per page with Puppeteer

[Nicolas-vrcc]

Hello,

I need to use multiple proxy and change let's say every 1000 page.goto() requests.
Is there a way to do this with puppeteer/chromium ?

Thank you !

[tomgallagher]

#1861

[Nicolas-vrcc]

@tomgallagher Yeah but there's no answer

[tomgallagher]

Not to changing them on the fly, but if you only want to do it every 1000 requests, just count and then browser.close() and relaunch the browser with new proxy, as per this.

So try:

puppeteer.launch({ args: ['--proxy-server="socks5://myproxy:8080'] });

or some variant. I agree the docs for this are non-existent.

[Nicolas-vrcc]

@tomgallagher Thanks bro

[tomgallagher]

No problem. You might as well tell everyone what worked if you have a moment. There's a more detailed discussion of proxies at #678.

[ebidel]

Dupe of #678

[pariola]

@tomgallagher Thanks for the answer but seems to only work without the quotes "

[coolicer]

I also need it.

[yi-ge]

I need it, too.

[thangtv611]

Thanks, I need too.

[remyzane]

I also need it.

[xiyuan-fengyu]

try this PuppeteerUtil.useProxy

[yi-ge]

@xiyuan-fengyu use http-proxy-agent to tackle Puppeteer proxy, this is a good idea! thank you.

[xiyuan-fengyu]

@xiyuan-fengyu use http-proxy-agent to tackle Puppeteer proxy, this is a good idea! thank you.

you are welcome

[gajus]

You can use https://github.com/gajus/puppeteer-proxy to set proxy either for entire page or for specific requests only, e.g.

import puppeteer from 'puppeteer';
import {
  createPageProxy,
} from 'puppeteer-proxy';

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  const pageProxy = createPageProxy({
    page,
    proxyUrl: 'http://127.0.0.1:3000',
  });

  await page.setRequestInterception(true);

  page.once('request', async (request) => {
    await pageProxy.proxyRequest(request);
  });

  await page.goto('https://example.com');
})();

To skip proxy simply call request.continue() conditionally.

Using puppeteer-proxy Page can have multiple proxies.

[princePeterHansen]

Hi Nicolas,

I have also been looking for a proxy solution when using Puppeteer. One of the easiest solutions I found is using API proxy services.

The basic example will look like this:

import puppeteer from 'puppeteer';

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  const url = 'https://proxybot.io/api/......?url=https://example.com';

  await page.goto(url);
})();

Every page will be opened through random proxy 😉

[Cuadrix]

It's possible with puppeteer-page-proxy.
It supports setting a proxy for an entire page, or if you like, it can set a different proxy for each request.

First install it:

npm i puppeteer-page-proxy

Then require it:

const useProxy = require('puppeteer-page-proxy');

Using it is easy;
Set proxy for an entire page:

await useProxy(page, 'http://127.0.0.1:8000');

If you want a different proxy for each request,then you can simply do this:

await page.setRequestInterception(true);
page.on('request', req => {
    useProxy(req, 'socks5://127.0.0.1:9000');
});

Then if you want to be sure that your page's IP has changed, you can look it up;

const data = await useProxy.lookup(page);
console.log(data.ip);

It supports http, https, socks4 and socks5 proxies, and it also supports authentication if that is needed:

const proxy = 'http://login:pass@127.0.0.1:8000'

Repository:
https://github.com/Cuadrix/puppeteer-page-proxy

[huanshiwushuang]

You can use https://github.com/gajus/puppeteer-proxy to set proxy either for entire page or for specific requests only, e.g.

import puppeteer from 'puppeteer';
import {
  createPageProxy,
} from 'puppeteer-proxy';

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  const pageProxy = createPageProxy({
    page,
    proxyUrl: 'http://127.0.0.1:3000',
  });

  await page.setRequestInterception(true);

  page.once('request', async (request) => {
    await pageProxy.proxyRequest(request);
  });

  await page.goto('https://example.com');
})();

To skip proxy simply call request.continue() conditionally.

Using puppeteer-proxy Page can have multiple proxies.

——————————————————————————————————————————————
But this library cannot capture the first document request of the page opened by page.click
Perhaps it is because Fetch.requestPaused does not support capture?

[AneeshQuonain]

@Cuadrix when I try to request a https site it gives me invalid browser request.
if I use proxy on https://example.com, the response will be invalid browser request and for http url its working fine. Can you please give me any solution for this