Error retrieving data from .com

[adnanhassan23]

Thank you for this great package! when i make more than 5 requests getting this error Error retrieving data from .com. Google is pretty hard on scrapers. After some research, I found that this issue can be fixed with proxies. This Proxy package https://www.npmjs.com/package/simple-proxies can be used with google-index ? This package https://github.com/christophebe/serp also using simple-proxies and https://www.scraperapi.com

[rodion-arr]

Hi @adnanhassan23, the package does not do requests from Node.js directly - it uses headless browser. So I assume if you need proxy - it should be configured on network level, not code

[adnanhassan23]

Hi @rodion-arr Thank you for your response. According to this comment puppeteer/puppeteer#1948 (comment) i changed request URL

 const scraperApiUrl = 'http://api.scraperapi.com?api_key=3c0481387e80c18778d20145c095e9&url='
    const googleDomain = 'https://google.com/search?hl=en-US&rls=en&q='
    const requestUrl = `${scraperApiUrl}${googleDomain}site:${site}` 

Is this good practise? Actually i want to use multiple proxies. Can you Please elaborate on configured on network level?

[rodion-arr]

@adnanhassan23, it's OK to pass request URL like you've shown.
Regarding "network level" - I meant to setup proxy on server level and left the code part unchanged, e.g. setup http_proxy env variable.