Visual tests - did any one manage to do screenshot on DOM item always the same?

[Pieras2]

Hi,

I wanted to implement simple tests to check basic parts of my application (some headers, sidebars etc.)
No matter which plugin I take, no matter how I configure cypress the results differs among runs :(

Did anyone manage to have repetitive test results?
I use Cypress 7.1.0, tried plugins cypress-image-snapshot, and cypress-image-diff-js and I lean to statemenet that there's something wrong with Cypress.

I tried via cypress open / run, 1920x1080, 1280x720 and also default Cypress viewport size.
Always some picture is shifted down couple of lines, or some pixels to the left.

[bahmutov]

Can you provide any more details? Like the repo you are trying to run in? In general, recreating a screenshot outside a container that is controlled, like a Docker container is very flakey and produces different images. I even had to switch to Docker container to create repeatable images, especially between the local machine and the CI machine, see the playlist of videos https://www.youtube.com/playlist?list=PLP9o9QNnQuAYhotnIDEUQNXuvXL7ZmlyZ

[Pieras2]

I will write tommorow being at work. In general I was launching Cypress on my laptop only. The tested app is a cloud application. I will show some parts of UI of tested application and the Cypress screenshots made on defined page elements. For daily automation I work with Azure Devops.

[Pieras2]

Below are my configs and simple test that just logs in and take screenshots of couple of elements on page.
As mentioned I test cloud app. and at first I run tests on my desk, then launching it on Azure DevOps but the problem is yet on my desk.

The pictures presented have overpainted parts.

Today I reverted all my code and restested:

• if I set viewport 1920x1080 it was ok (however once failed with some item difference 0.039%)
• if I removed viewport settings from cypress.json I got such a diff (1 fail) - the top of the picture looks as if some GUI part of a different element on page which is above it)
  
• test run again and fail was on a different element - I manually check that when I grab browser window to resize it the white stripe appears on the right and the case might be that when cypress switches view it does the screenshots too fast, maybe if it was possible to introduce some delay between the moment when Cypress resizes view and does a screenshot
  

Please look at the video ... I think it might be the case, resizing screen intoduces some white parts which normally are not present. I am almost sure that this resize make failure because screenshot is made to fast. Look at the right vertical edge of the window.
https://youtu.be/QfX2KA6lZRA

Unfortunately results once are pass, other time such a picture "shift" is noticed and random failures happen :(

I also tried this https://docs.cypress.io/api/plugins/browser-launch-api#Set-screen-size-when-running-headless (with resolution as in example). It would be a pitty if I could not do test with various resolutions because then I cannot make UI tests if my app is displayed on a smaller screen where some automatic CSS adjustments are made in the app. Anyway always the failure is as on the screenshots.

plugins.index.js

/**
 * @type {Cypress.PluginConfig}
 */
  const {
  addMatchImageSnapshotPlugin,
} = require('cypress-image-snapshot/plugin');

module.exports = (on, config) => {
  addMatchImageSnapshotPlugin(on, config);
  on('task', {
    failed: require('cypress-failed-log/src/failed')(),
  });
};

support/commands/commands.ts

import { addMatchImageSnapshotCommand } from 'cypress-image-snapshot/command';
import 'cypress-wait-until';

addMatchImageSnapshotCommand({
  customSnapshotsDir: "cypress/snapshots/" + Cypress.browser.name,
  capture: 'viewport'
});

support/commands/index.js

import '../pages/loginPage'

import 'cypress-xpath'
import 'cypress-failed-log'
import 'cypress-mochawesome-reporter/register'

Cypress.on('uncaught:exception', (err) => {
    console.log(err);
    return false
});

cypress.json

    "defaultCommandTimeout": 60000,
    "pageLoadTimeout": 120000,
    "screenshotOnRunFailure": true,
    "trashAssetsBeforeRuns": true,
    "viewportWidth": 1920,
    "viewportHeight": 1080,
    "integrationFolder": "cypress/tests",
    "video": false,
    "env": {
        "testEnv": "dev"
    },
    "reporter": "junit",
    "reporterOptions": {
        "mochaFile": "results/test-results.xml",
        "toConsole": true
    }
}

Simple test:

describe('UI Test - Main Page', () => {

        beforeEach(() => {
            cy
                .login();
            }),

        it ('One Header', () => {
            cy
                .get(oneHeader)
                .matchImageSnapshot('oneHeader', {blackout: uiExcluded});
        }),

        it('Main Page - toolbar', () => {
            cy
                .get(mainPageToolbar)
                .matchImageSnapshot('main-page-toolbar');
        }),

        it('Main Page - archives header', () => {
            cy
                .get(mainPageArchivesHeader)
                .matchImageSnapshot('main-page-archives-header');
        }),

        it('Left Navbar', () => {
            cy
                .get(leftNavbar)
                .matchImageSnapshot('left-navbar');
            cy
                .get(mainPageToolbar)
                .matchImageSnapshot('main-page-toolbar');
        }),

        it('Right Sidebar - Recent uploads', () => {
            cy
                .get(recentUploadsIcon)
                .click()
                .get(rightSidebar)
                .matchImageSnapshot('right-sidebar-recent-uploads');
        }),

        it('Right Sidebar - Filters', () => {
            cy
                .get(filtersIcon)
                .click()
                .get('.right-side-bar')
                .matchImageSnapshot('right-sidebar-filters')
        }),

        it('Detail view', () => {
            cy
                .get(detailViewButton)
                .click()
                .get(leftSidebar)
                .matchImageSnapshot('detail-view-left-sidebar')
                .get(detailViewToolbar)
                .matchImageSnapshot('detail-view-toolbar')
                .get(detailViewContent)
                .matchImageSnapshot('detail-view-content');
        }),

        it('Map fields', () => {
            cy
                .get(mapFieldsButton)
                .click()
                .get(leftSidebar)
                .matchImageSnapshot('map-fields-left-sidebar')
                .get(mapFieldsToolbar)
                .matchImageSnapshot('map-fields-toolbar')
                .get(mapFieldsContent)
                .matchImageSnapshot('map-fields-content');
        }),

        it('3D preparation view', () => {
            cy
                .get(orientationViewButton)
                .click()
                .get(leftSidebar)
                .matchImageSnapshot('3d-preparation-left-sidebar')
                .get(orientationViewToolbar)
                .matchImageSnapshot('3d-preparation-toolbar')
                .get(orientationViewContent)
                .matchImageSnapshot('3d-preparation-content')
                .get(orientationViewFooter)
                .matchImageSnapshot('3d-preparation-footer')
                .get(orientationViewRightNavbar)
                .matchImageSnapshot('3d-preparation-right-navbar');
        });
});