Puppeteer bot detection with out-of-process iframes by user-agent and screen size

[ukrexpo]

Real User-Agent is detectable with <iframe> when using "puppeteer-extra-plugin-stealth". To prevent it use "'--disable-features=site-per-process'" argument for puppeteer.launch(). See explanation: puppeteer/puppeteer#2548

Real screen size is also detectable with iframe, to prevent it I used preload() script.

OS: Windows 10
"puppeteer": 3.0.4
"puppeteer-extra": 3.1.9
"puppeteer-extra-plugin-stealth": 2.4.9

Code to reproduce:

const puppeteer = require('puppeteer');
const puppeteerExtraPluginStealth = require('puppeteer-extra-plugin-stealth');
const puppeteerExtraPluginUserAgentOverride = require('puppeteer-extra-plugin-stealth/evasions/user-agent-override');
const {PuppeteerExtra} = require('puppeteer-extra');

function preload(device) {
  Object.defineProperty(navigator, 'platform', {
    value: device.platform,
    writable: true,
  });
  Object.defineProperty(navigator, 'userAgent', {
    value: device.userAgent,
    writable: true,
  });
  Object.defineProperty(screen, 'height', {
    value: device.viewport.height,
    writable: true,
  });
  Object.defineProperty(screen, 'width', {
    value: device.viewport.width,
    writable: true,
  });
  Object.defineProperty(window, 'devicePixelRatio', {
    value: device.viewport.deviceScaleFactor,
    writable: true,
  });
}

const device = {
  userAgent: 'Mozilla/5.0 (Macintosh)',
  viewport: {
    width: 1200,
    height: 800,
    deviceScaleFactor: 1,
    isMobile: false,
    hasTouch: false,
    isLandscape: true,
  },
  locale: 'en-US,en;q=0.9',
  platform: 'Macintosh',
};

(async () => {
  try {
    const pptr = new PuppeteerExtra(puppeteer);
    const pluginStealth = puppeteerExtraPluginStealth();
    pluginStealth.enabledEvasions.delete('user-agent-override'); // Remove this specific stealth plugin from the default set
    pptr.use(pluginStealth);

    const pluginUserAgentOverride = puppeteerExtraPluginUserAgentOverride({
      userAgent: device.userAgent,
      locale: device.locale,
      platform: device.platform,
    });
    pptr.use(pluginUserAgentOverride);

    const browser = await pptr.launch({
      args: [
        '--disable-features=site-per-process',
        `--window-size=${device.viewport.width},${device.viewport.height}`,
      ],
      headless: false,
      defaultViewport: device.viewport,
    });
    const page = await browser.newPage();
    await page.evaluateOnNewDocument(preload, device);
    await page.goto('https://codepen.io/ukrexpo/pen/JjYverG');
  } catch (err) {
    console.error(err);
  }
})();

[ruimarinho]

@ukrexpo shouldn't platform be MacIntel instead?

[brdeav39]

Awesome! This just fixed a problem I was encountering on a particular site. Thanks @ukrexpo.

[kensnyder]

I also discovered that puppeteer reports a screen.colorDepth and screen.pixelDepth of 30 instead of the 24 I see on other desktop browsers (Windows and Mac). It sounds like there are a few devices that support 30-bit color, but theoretically the depth should match up with graphics card vendor reported by WebGL.

Additionally a plugin like this could consider changing window.outerHeight and screen.availHeight to account for an OS dock height and browser tabs + omnibar height. It gives away puppeteer quickly if the window.innerHeight doesn't make sense with the values for screen.height and screen.availHeight.

[momala454]

careful also with

document.documentElement.clientWidth
document.documentElement.clientHeight

it looks like chrome headless doesn't account for scrollbar size

regarding window.innerHeight / screen.availHeight
maybe be carefull with css media queries which could detect the real size ?

[berstend]

I think in more recent Chrome versions this might be required: --flag-switches-begin --disable-site-isolation-trials --flag-switches-end

[flashjames]

Also make sure you're using a recent puppeteer version, in puppeteer@1.8.0 @berstend answer crashes chromium

[berstend]

Also make sure you're using a recent puppeteer version, in puppeteer@1.8.0 @berstend answer crashes chromium

pptr 1.8.0 has been released 3 years ago and uses chrome v70 😅 but happy to hear our efforts for backwards compatibility are appreciated :-)

[berstend]

Opportunity for someone to get involved (#439):

Would be great if someone can make a proper test case to confirm failing behavior + fixed behavior when the two launch args are used. Also it'd be worthwhile to document negative side-effects - if they're neglectable we can make an evasion to enable that as the default.

[ya-mouse]

The workaround is broken since puppeteer@12.0.0, the last working version is @11.0.0.
At least on Linux host it doesn't work. Running workaround reveals the real platform details/UA.

[berstend]

Pretty sure this issue is not applicable anymore, closing as there's no substantive discussion either

[onlynfp]

@ukrexpo How can I use multiple devices from your code?