Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use original baseline for rebuilds (new build for the same commit) #313

Merged
merged 4 commits into from Apr 9, 2021

Conversation

ghengeveld
Copy link
Member

@ghengeveld ghengeveld commented Mar 25, 2021

If the last build was on the same branch and for the same commit, we're dealing with a rebuild (e.g. restarted CI job). In case this build was passed/accepted, we can skip the build altogether, because it shouldn't lead to any changes. This of course assumes flaky stories won't get accepted.

@tmeasday
Copy link
Member

tmeasday commented Mar 25, 2021

@ghengeveld I thought about this a bit more and I realised we had neglected to think about the more common situation where this happens.

So we've been thinking about people re-running CI jobs due to flakes.

However, when you have Chromatic setup without provider integration, you use chromatic without the --exit-zero-on-changes flag, and the process for approving snapshots looks like so:

  1. Push a commit with changes
  2. Chromatic runs, and fails CI due to those changes. You get a red PR.
  3. You go into Chromatic and approve those changes. Nothing happens on the PR as you aren't integrated.
  4. You re-run the CI job, which this time passes as the new baselines are from the same commit that you are running.

So with that in mind, two things:

  1. This change is going to mess things up for folks doing the above, with or without --onlyChanged, because the baseline in step 4 won't be updated to the build from step 2.
  2. If someone is using --onlyChanged, without this change it's quite cool actually -- we use the build from the same commit as the most recent ancestor in step 4, which, given those tests have been approved (and are the baselines) means we preserve every story and set it automatically to PASSED, which means the build passes without any captures taken. Which is correct!

(PS we should ensure we add tests in the index for step 2, both with and without approving the changes in step 1).


So I think probably the correct solution here is a "recapture" button in the UI that is smart enough to just rerun the captures in the current build (not the preserved stories). WDYT?

@ghengeveld
Copy link
Member Author

@tmeasday I updated this according to what we discussed.

Copy link
Member

@tmeasday tmeasday left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @ghengeveld

@ghengeveld ghengeveld changed the title Don't use last build on branch as baseline if for the same commit Use original baseline for rebuilds (new build for the same commit) Apr 9, 2021
@ghengeveld ghengeveld merged commit a4b3e38 into next Apr 9, 2021
@ghengeveld ghengeveld deleted the ignore-rebuilt-baseline branch April 9, 2021 12:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants