Unreliable font loading causes difference between test and reference

[giorgiosironi]

We are experiencing intermittent test failures due to font not loading for some scenarios.

Using BackstopJS 6.2.2, running in Docker with:

"dockerCommandTemplate": "docker run --rm -i --user $(id -u):$(id -g) --network sciety-test_default --mount type=bind,source=\"{cwd}\",target=/src backstopjs/backstopjs:{version} {backstopCommand} {args}",

Examples:

Researching previous issues lead us to code to wait for font loading:
#1298 (comment)
#1303 (comment)
but due to the age of these issues we don't know how to implement this in onReady with the Puppeteer API, can anyone advise?

[weseze]

We are also experiencing this issue on 100+ websites with large testscanario's.
We are testing local websites vs production websites to find visual regressions between updates.

Anything from 50-95% of tests fail. The fails are random. Every time we run the tests, the result is different.

These tests used to run fine until we updated our entire npm stack. Mainly because the chromium we were using didn't support modern CSS (things like grid) and those tests failed.

Tried almost every suggestion I could find here in this issue and related issues. Nothing has worked so far.

1/
Tried reverting to older version of backstopJS (mainly 5.0/5.1): this didn't work because of a dependency hell on old libraries that I couldn't get resolved. I was able to install a 5.2 release, but that didn't help.

2/
Tried updating everything to latest versions, remaking our backstop configs from the latest versions examples: did not work

3/
Tried adding JS to check if fonts are fully loaded before creating the screenshots: did not work

4/
Added delays up to 10 seconds before taking the screenshots: did not work

5/
Limit the async capture limit to 1: did not work

6/
Tried al sort of CSS injections for font smoothing, kerning, aliasing: nothing worked. Fonts just started looking ugly, but still the same small differences

7/
Tried to use 2 testscenario's with only the testUrl, not the referenceUrl and comparing those: did not work

I have run out of things to try...
As of right now, backstopJS testing has become unusable for us and we are investigating other methods.
Any help to get it working again would be greatly appreciated, because switching our entire testsuite to a new system is a big project...

[giorgiosironi]

For reference, we tried (from dgrebb@bf413fb):

await page.waitForFunction(() => document.fonts.ready);

added to onReady. Still see intermittent failures of the same kind.

[dgrebb]

Thanks @giorgiosironi - this adds the same into the runner itself, and I've seen some very minimal improvements.

Out of curiosity, and it sounds like the answer is no, could this be a CSP issue? The fonts are there sometimes but not always, correct?