Screenshot Testing

[rwaskiewicz]

I'd like to open up a discussion regarding screenshot testing & Stencil.

Stencil has screenshot functionality that is currently marked as 'experimental'. While this functionality has been a part of Stencil for quite some time now, it's never graduated beyond that experimental phase.

To the best of the Stencil team's knowledge, this feature is seldom used. Assuming this is the case, there are a few possible explanations for this (which are not mutually exclusive of one another):

• teams are hesitant to adopt a feature marked as experimental
• there are other solutions in the JavaScript ecosystem that fit their needs better
• existing bugs that prevent full adoption

Alternatively, the claim that this feature not getting much usage could turn out to be completely false.

In the spirit of this forum/discussion around testing packages, I'd like to open up some dialogue around screenshot testing. We'd like to try to better understand a few things around screenshot testing in your Stencil projects:

• Are you using screenshot testing from Stencil? A third party solution?
• What do you like/dislike about your current screenshot testing?
• Would you rather not use screenshot testing? Why?

[johnjenkins]

We're using jest-image-snapshot within Stencil's core jest/puppeteer tests atm.
It works ok within the constraints of pupeteer. I hate that I have to run them in a docker container to match the CI environment but that's just the nature of the beast(?).
We plan to move to playwright for browser-based tests (including snapshots) soon.

[danyball]

Why you cant just run the tool without a container?

[johnjenkins]

Because you'll get inconsistencies between operating systems (e.g. Mac Dev machines / Linux CI machines) with things like font rendering.
Font rendering sounds like it would only cause a very small % difference, but in order to accurately / usefully track the small image that a component produces you need the tolerance turned low.

[danyball]

Are you using screenshot testing from Stencil? A third party solution?
What do you like/dislike about your current screenshot testing?
Would you rather not use screenshot testing? Why?

We do not use it yet. But adding a simple solution would be great to check any visual changes at our components. Any screenshot testing logic is better than nothing to get these changes. Ok it is not pixel perfect, but setting a threshold is good enough for this. We dont need AI or anything else.