[Bug]: ElementHandle.click() in cross-origin iframes is off target in non-headless mode #7849

Mr0grog · 2021-12-15T21:23:27Z

Bug description

This is most likely an issue in Chromium, but I figured I’d file here first in case folks here know whether it’s actually intended behavior that Puppeteer needs to deal with or if I’m misunderstanding the issue.

In Puppeteer 12.0.0 (Chromium 97.0.4692.0), the DOM.getContentQuads and DOM.getBoxModel protocol methods (and maybe others) seem to behave differently in headless vs. non-headless mode: in headless mode; they return coordinates relative to the main frame’s viewport (as they used to in both modes), but in non-headless mode, they return coordinates relative to the frame the object they are called with is in.

This leads to a situation where click() works in headless mode, but not in non-headless mode (because it’s clicking in the wrong place), and where methods like clickablePoint() or boundingBox() return incorrect coordinates in non-headless mode.

Steps to reproduce the problem:

Load a page structured like:

<html>
  <body>
    <iframe id="frame" width="300" height="300" style="position: absolute; left: 50px, top: 50px;">
      # contentDocument
        <html>
          <body>
            <div id="inner-content" style="position: absolute; left: 50px, top: 50px;">Hello</div>
          </body>
        </html>
    </iframe>
  </body>
</html>

Try to get the clickable point of #inner-content:

const frameElement = await page.$("#frame");
const frame = await frameElement.contentFrame();
const innerContent = await frame.$("#inner-content");
const clickPoint = await innerContent.clickablePoint({ x: 0, y: 0 });
console.log("Point:", clickpoint);

Doing the above in headless mode and non-headless mode should log the same coordinate, but does not.

Puppeteer version

12.0.0

Node.js version

16.13.0

npm version

8.1.3

What operating system are you seeing the problem on?

macOS

Relevant log output

No response

The text was updated successfully, but these errors were encountered:

Mr0grog · 2021-12-15T21:29:12Z

I was able to work around this by calculating the correct position using .evaluate(), but this is kind of complicated:

const getBoundsInFrame = (node) => node.getBoundingClientRect().toJSON();
const innerBox = await innerContent.evaluate(getBoundsInFrame);
const frameBox = await frameElement.evaluate(getBoundsInFrame);
const clickPoint = { x: frameBox.x + innerBox.x, y: frameBox.y + innerBox.y };
page.click("body", { offset: clickPoint });

(Obviously there’s some work in figuring out what frame(s) to check in order to make that generic.)

OrKoN · 2021-12-22T07:55:36Z

I am unable to reproduce it. Was it an out of process iframe?

OrKoN · 2021-12-22T07:57:41Z

This is my script. Do you have an exact script that reproduces the problem?

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({
    headless: true, // or false
  });
  const page = await browser.newPage();
  await page.goto('bug-click.html');
  const frameElement = await page.$("#frame");
  const frame = await frameElement.contentFrame();
  const innerContent = await frame.$("#inner-content");
  const clickPoint = await innerContent.clickablePoint({ x: 0, y: 0 });
  console.log("Point:", clickPoint);
  await browser.close();
})();

<html>
  <body>
    <iframe id="frame" width="300" height="300" style="position: absolute; left: 50px, top: 50px;">
    </iframe>
    <script>
    	document.getElementById('frame').src = "data:text/html;charset=utf-8," + escape(`<html>
          <body>
            <div id="inner-content" style="position: absolute; left: 50px; top: 50px;">Hello</div>
          </body>
        </html>`);
    </script>
  </body>
</html>

Mr0grog · 2021-12-22T17:07:28Z

Was it an out of process iframe?

The iframe where I encountered this is a different domain, so I think it would normally be, but the script I was dealing with uses --disable-features=site-per-process, which I was thinking should prevent it from being out of process. It’s possible I might be misunderstanding that, though.

Do you have an exact script that reproduces the problem?

Not one that is straightforward (I hit this in a reasonably complex legacy tool I’m now maintaining that deals with Airtable). I’ll try to create a complete test case sometime in the next week or so.

Mr0grog · 2021-12-23T16:39:55Z

Yep, if the frame is from a different origin (and probably therefore a separate process), that triggers the bug. Here’s a modified version of your script that works:

<!-- bug-click.html -->
<html>
  <body>
    <iframe id="frame" width="300" height="300" style="position: absolute; left: 50px; top: 50px;" src="http://example.com/">
    </iframe>
  </body>
</html>

// bug-click.js
const puppeteer = require('puppeteer');

async function getClickablePoint(headless) {
  const browser = await puppeteer.launch({ headless });
  const page = await browser.newPage();
  await page.goto(`file://${__dirname}/bug-click.html`);
  const frameElement = await page.$("#frame");
  const frame = await frameElement.contentFrame();
  const innerContent = await frame.$("a");
  const clickPoint = await innerContent.clickablePoint({ x: 0, y: 0 });
  await browser.close();
  return clickPoint;
}

async function testBug() {
  const clickableHeadless = await getClickablePoint(true);
  const clickableVisible = await getClickablePoint(false);
  console.log("These two values should be the same:")
  console.log("Clickable Point (Headless):", clickableHeadless);
  console.log("Clickable Point (Visible): ", clickableVisible);
}

testBug();

And I see this output:

$ node pptr-test/bug-click.js
These two values should be the same:
Clickable Point (Headless): { x: 84, y: 326.875 }
Clickable Point (Visible):  { x: 32, y: 274.875 }

It also doesn’t appear to matter if args: ['--disable-features=site-per-process'] was passed to puppeteer.launch.

OrKoN · 2022-01-14T09:04:09Z

So my current understanding is:

headless is likely to have the iframe in the same process therefore the bounds received from the renderer are correct.
OOPIF iframes support was only added in v11 so the headful case would not work before v11. Unfortunately, we didn't account for the click position calculations.

Issues: #7849

Mr0grog · 2022-01-14T17:02:29Z

OOPIF iframes support was only added in v11 so the headful case would not work before v11. Unfortunately, we didn't account for the click position calculations.

Oh, interesting! I thought I had tried this in 11 (I worked my way back from 13 to see when it broke), but I may need to check again. Thanks for helping me understand and clarify the issue! Looking forward to seeing that PR land. 😄 🙏

…7900) Issues: #7849

Mr0grog · 2022-01-17T18:38:07Z

The values for clickablePoint, boundingBox, and boxModel are still off in v13.1.0, and it appears to be because the borders of the iframe aren’t accounted for. From the above script:

$ node bug-click.js
These two values should be the same:
Clickable Point (Headless): { x: 84, y: 332.875 }
Clickable Point (Visible):  { x: 82, y: 330.875 }

OrKoN · 2022-01-18T07:29:07Z

@Mr0grog thanks for double checking. I opened a CL to switch to using content box that should account for paddings/margins/borders properly.

Mr0grog · 2022-01-18T19:29:54Z

No problem. Just tried 13.1.1 with that fix, and it’s working great for me — thanks so much!

Mr0grog added the bug label Dec 15, 2021

Mr0grog changed the title ~~[Bug]: ElementHandle.click() in an iframe is off target in non-headless mode~~ [Bug]: ElementHandle.click() in cross-origin iframes is off target in non-headless mode Jan 3, 2022

OrKoN mentioned this issue Jan 14, 2022

fix: correctly compute clickable points for elements inside OOPIFs #7900

Merged

OrKoN added a commit that referenced this issue Jan 14, 2022

fix: correctly compute clickable points for elements inside OOPIFs

90e2853

Issues: #7849

OrKoN added a commit that referenced this issue Jan 14, 2022

fix: correctly compute clickable points for elements inside OOPIFs

72720e2

Issues: #7849

OrKoN added a commit that referenced this issue Jan 17, 2022

fix: correctly compute clickable points for elements inside OOPIFs (#…

486bbe0

…7900) Issues: #7849

OrKoN closed this as completed Jan 17, 2022

OrKoN mentioned this issue Jan 18, 2022

fix: use content box for OOPIF offset calculations #7911

Merged

This was referenced May 30, 2022

chore(main): release 1.0.0 #8407

Closed

chore(main): release 1.0.0 #8411

Closed

This was referenced May 30, 2022

chore(main): release 1.0.0 #8414

Closed

chore(main): release 1.0.0 #8418

Closed

chore(main): release puppeteer 14.2.0 #8420

Closed

This was referenced May 30, 2022

chore(main): release 1.0.0 #8425

Closed

chore(main): release 1.0.0 #8427

Closed

This was referenced Jun 13, 2022

chore(main): release 15.0.0 #8513

Closed

chore(main): release 15.0.0 #8514

Closed

chore(main): release 15.0.0 #8516

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: ElementHandle.click() in cross-origin iframes is off target in non-headless mode #7849

[Bug]: ElementHandle.click() in cross-origin iframes is off target in non-headless mode #7849

Mr0grog commented Dec 15, 2021 •

edited

Mr0grog commented Dec 15, 2021 •

edited

OrKoN commented Dec 22, 2021

OrKoN commented Dec 22, 2021

Mr0grog commented Dec 22, 2021

Mr0grog commented Dec 23, 2021

OrKoN commented Jan 14, 2022

Mr0grog commented Jan 14, 2022

Mr0grog commented Jan 17, 2022

OrKoN commented Jan 18, 2022

Mr0grog commented Jan 18, 2022

[Bug]: ElementHandle.click() in cross-origin iframes is off target in non-headless mode #7849

[Bug]: ElementHandle.click() in cross-origin iframes is off target in non-headless mode #7849

Comments

Mr0grog commented Dec 15, 2021 • edited

Bug description

Puppeteer version

Node.js version

npm version

What operating system are you seeing the problem on?

Relevant log output

Mr0grog commented Dec 15, 2021 • edited

OrKoN commented Dec 22, 2021

OrKoN commented Dec 22, 2021

Mr0grog commented Dec 22, 2021

Mr0grog commented Dec 23, 2021

OrKoN commented Jan 14, 2022

Mr0grog commented Jan 14, 2022

Mr0grog commented Jan 17, 2022

OrKoN commented Jan 18, 2022

Mr0grog commented Jan 18, 2022

Mr0grog commented Dec 15, 2021 •

edited

Mr0grog commented Dec 15, 2021 •

edited