Snapshot generation doesn't seem to be working correctly (first observed Mar 23)

[mskaterin1]

Expected behavior

• Snapshots should record updated changes consistently
• If snapshots folder is deleted, running wtr --update-snapshots should generate a new folder with the snapshots in them

Actual Behavior

• Snapshots are not updated consistently. Sometimes they only generate snapshots for a few test cases in a test suite but not all test casses
• Sometimes new snapshot folder is not generated when running command even if it's deleted

Both me and my coworkers are able to reproduce this issue. Wondering if we are the only ones experiencing this issue or there's a newly introduced bug?

Additional context

We cannot share the source codes due to intellectual property rights. But snapshot generation used to be working and we first observed this issue on Mar 23. There were no changes made in between

Here are the related package versions we are currently using:

"@web/test-runner": "^0.15.1",
"@web/test-runner-chrome": "^0.11.0",
"@open-wc/semantic-dom-diff": "^0.19.7",
"@open-wc/testing": "^3.1.6"

We tried updating them all to the latest versions of the packages and that still did not fix the issue.

Thanks!

[Westbrook]

I've not encountered this while running thousands of VRTs a day. Could you share more about your usage?

[mskaterin1]

It's worthwhile to mention that the tests still register as passed. But if we actually inspect the snapshots, they are not updated correctly and are often generated for only a few of all the test cases that have snapshots

The usage is standard Lit + Typescript starter, just adding snapshot testing:

import { expect, html, fixture, fixtureCleanup } from '@open-wc/testing';
import { spy } from 'sinon';
import './OurComponent';
import { OurComponent } from './OurComponent';

describe('OurComponent', () => {
    afterEach(fixtureCleanup);

    it('should render with default props', async () => {
        const component: OurComponent = await fixture(html`<our-component></our-component>`);
        expect(component).shadowDom.to.equalSnapshot();
    });
});