-
Notifications
You must be signed in to change notification settings - Fork 13.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore(visual-regression): add script to update ground truths #29204
Open
thetaPC
wants to merge
15
commits into
main
Choose a base branch
from
visual-regression-script
base: main
Could not load branches
Branch not found: {{ refName }}
Could not load tags
Nothing to show
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
d4726ec
chore(visual-regression): add script to update ground truths
thetaPC 9293b69
Merge branch 'main' of github.com:ionic-team/ionic-framework into vis…
thetaPC 49b8ddf
refactor(visual-regression): add check for uncommitted changes
thetaPC 2b2edc6
Merge branch 'main' of github.com:ionic-team/ionic-framework into vis…
thetaPC e0fd061
refactor(visual-regression): move build to outside if
thetaPC baf41c0
Merge branch 'main' of github.com:ionic-team/ionic-framework into vis…
thetaPC 0e9f069
refactor(visual-regression): set default branch
thetaPC d151260
refactor(visual-regression): move file to scripts
thetaPC d5e7364
Merge branch 'main' of github.com:ionic-team/ionic-framework into vis…
thetaPC 076c0b8
refactor(visual-regressions): only check img files
thetaPC 8f477aa
refactor(visual-regressions): use clack library
thetaPC 7690a5d
refactor(visual-regressions): add info on how to open report
thetaPC 3efff0d
refactor(visual-regressions): use spawn to display report quit key
thetaPC f528236
fix(visual-regressions): remove unused line
thetaPC 28c2aa5
fix(visual-regressions): fetch base branch to sync local and remote
thetaPC File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,260 @@ | ||
// The purpose of this script is to provide a way run the E2E tests | ||
// without having the developer to manually run multiple commands based | ||
// on the desired end result. | ||
// E.g. update the local ground truths for a specific component or | ||
// open the Playwright report after running the E2E tests. | ||
|
||
import { | ||
intro, | ||
outro, | ||
confirm, | ||
spinner, | ||
isCancel, | ||
cancel, | ||
text, | ||
log, | ||
} from '@clack/prompts'; | ||
import { exec, spawn } from 'child_process'; | ||
import fs from 'node:fs'; | ||
import { setTimeout as sleep } from 'node:timers/promises'; | ||
import util from 'node:util'; | ||
import color from 'picocolors'; | ||
|
||
async function main() { | ||
const execAsync = util.promisify(exec); | ||
const cleanUpFiles = async () => { | ||
// Clean up the local ground truths. | ||
const cleanUp = spinner(); | ||
|
||
// Inform the user that the local ground truths are being cleaned up. | ||
cleanUp.start('Restoring local ground truths'); | ||
|
||
// Reset the local ground truths. | ||
await execAsync('git reset -- src/**/*-linux.png').catch((error) => { | ||
cleanUp.stop('Failed to reset local ground truths'); | ||
console.error(error); | ||
return process.exit(0); | ||
}); | ||
|
||
// Restore the local ground truths. | ||
await execAsync('git restore -- src/**/*-linux.png').catch((error) => { | ||
cleanUp.stop('Failed to restore local ground truths'); | ||
console.error(error); | ||
return process.exit(0); | ||
}); | ||
|
||
// Inform the user that the local ground truths have been cleaned up. | ||
cleanUp.stop('Local ground truths have been restored to their original state in order to avoid committing them.'); | ||
}; | ||
|
||
intro(color.inverse(' Update Local Ground Truths')); | ||
|
||
// Ask user for the component name they want to test. | ||
const componentValue = await text({ | ||
message: 'Enter the component or path you want to test (e.g. chip, src/components/chip)', | ||
placeholder: 'Empty for all components', | ||
}); | ||
|
||
// User cancelled the operation with `Ctrl+C` or `CMD+C`. | ||
if (isCancel(componentValue)) { | ||
cancel('Operation cancelled'); | ||
return process.exit(0); | ||
} | ||
|
||
// Ask user if they want to update their local ground truths. | ||
const shouldUpdateTruths = await confirm({ | ||
message: 'Do you want to update your local ground truths?', | ||
}); | ||
|
||
// User cancelled the operation with `Ctrl+C` or `CMD+C`. | ||
if (isCancel(shouldUpdateTruths)) { | ||
cancel('Operation cancelled'); | ||
return process.exit(0); | ||
} | ||
|
||
if (shouldUpdateTruths) { | ||
const defaultBaseBranch = 'main'; | ||
|
||
// Ask user for the base branch. | ||
let baseBranch = await text({ | ||
message: 'Enter the base branch name:', | ||
placeholder: `default: ${defaultBaseBranch}`, | ||
}) | ||
|
||
// User cancelled the operation with `Ctrl+C` or `CMD+C`. | ||
if (isCancel(baseBranch)) { | ||
cancel('Operation cancelled'); | ||
return process.exit(0); | ||
} | ||
|
||
// User didn't provide a base branch. | ||
if (!baseBranch) { | ||
baseBranch = defaultBaseBranch; | ||
} | ||
|
||
/** | ||
* The provided base branch needs to be fetched. | ||
* This ensures that the local base branch is up-to-date with the | ||
* remote base branch. Otherwise, there might be errors stating that | ||
* certain files don't exist in the local base branch. | ||
*/ | ||
const fetchBaseBranch = spinner(); | ||
|
||
// Inform the user that the base branch is being fetched. | ||
fetchBaseBranch.start(`Fetching "${baseBranch}" to have the latest changes`); | ||
|
||
// Fetch the base branch. | ||
await execAsync(`git fetch origin ${baseBranch}`).catch((error) => { | ||
fetchBaseBranch.stop(`Failed to fetch "${baseBranch}"`); | ||
console.error(error); | ||
return process.exit(0); | ||
}); | ||
|
||
// Inform the user that the base branch has been fetched. | ||
fetchBaseBranch.stop(`Fetched "${baseBranch}"`); | ||
|
||
|
||
const updateGroundTruth = spinner(); | ||
|
||
// Inform the user that the local ground truths are being updated. | ||
updateGroundTruth.start('Updating local ground truths'); | ||
|
||
// Check if user provided an existing file or directory. | ||
const isValidLocation = fs.existsSync(componentValue); | ||
|
||
// User provided an existing file or directory. | ||
if (isValidLocation) { | ||
const stats = fs.statSync(componentValue); | ||
|
||
// User provided a file as the component. | ||
// ex: `componentValue` = `src/components/chip/test/basic/chip.e2e.ts` | ||
if (stats.isFile()) { | ||
// Update the local ground truths for the provided path. | ||
await execAsync(`git checkout origin/${baseBranch} -- ${componentValue}-snapshots/*-linux.png`).catch((error) => { | ||
updateGroundTruth.stop('Failed to update local ground truths'); | ||
console.error(error); | ||
return process.exit(0); | ||
}); | ||
} | ||
|
||
// User provided a directory as the component. | ||
// ex: `componentValue` = `src/components/chip` | ||
if (stats.isDirectory()) { | ||
// Update the local ground truths for the provided directory. | ||
await execAsync(`git checkout origin/${baseBranch} -- ${componentValue}/test/*/*.e2e.ts-snapshots/*-linux.png`).catch((error) => { | ||
updateGroundTruth.stop('Failed to update local ground truths'); | ||
console.error(error); | ||
return process.exit(0); | ||
}); | ||
} | ||
} | ||
// User provided a component name as the component. | ||
// ex: `componentValue` = `chip` | ||
else if (componentValue) { | ||
// Update the local ground truths for the provided component. | ||
await execAsync(`git checkout origin/${baseBranch} -- src/components/${componentValue}/test/*/${componentValue}.e2e.ts-snapshots/*-linux.png`).catch((error) => { | ||
updateGroundTruth.stop('Failed to update local ground truths'); | ||
console.error(error); | ||
return process.exit(0); | ||
}); | ||
} | ||
// User provided an empty string. | ||
else { | ||
// Update the local ground truths for all components. | ||
await execAsync(`git checkout origin/${baseBranch} -- src/components/*/test/*/*.e2e.ts-snapshots/*-linux.png`).catch((error) => { | ||
updateGroundTruth.stop('Failed to update local ground truths'); | ||
console.error(error); | ||
return process.exit(0); | ||
}); | ||
} | ||
|
||
// Inform the user that the local ground truths have been updated. | ||
updateGroundTruth.stop('Updated local ground truths'); | ||
} | ||
|
||
const buildCore = spinner(); | ||
|
||
// Inform the user that the core is being built. | ||
buildCore.start('Building core'); | ||
|
||
/** | ||
* Build core | ||
* Otherwise, the uncommitted changes will not be reflected in the tests because: | ||
* - popping the stash doesn't trigger a re-render even if `npm start` is running | ||
* - app is not running the `npm start` command | ||
*/ | ||
await execAsync('npm run build').catch((error) => { | ||
// Clean up the local ground truths. | ||
cleanUpFiles(); | ||
|
||
buildCore.stop('Failed to build core'); | ||
console.error(error); | ||
return process.exit(0); | ||
}); | ||
|
||
buildCore.stop('Built core'); | ||
|
||
const runE2ETests = spinner(); | ||
|
||
// Inform the user that the E2E tests are being run. | ||
runE2ETests.start('Running E2E tests'); | ||
|
||
// User provided a component value. | ||
if (componentValue) { | ||
await execAsync(`npm run test.e2e.docker.ci ${componentValue}`).catch((error) => { | ||
// Clean up the local ground truths. | ||
cleanUpFiles(); | ||
|
||
runE2ETests.stop('Failed to run E2E tests'); | ||
console.error(error); | ||
return process.exit(0); | ||
}); | ||
} else { | ||
await execAsync('npm run test.e2e.docker.ci').catch((error) => { | ||
// Clean up the local ground truths. | ||
cleanUpFiles(); | ||
|
||
runE2ETests.stop('Failed to run E2E tests'); | ||
console.error(error); | ||
return process.exit(0); | ||
}); | ||
} | ||
|
||
runE2ETests.stop('Ran E2E tests'); | ||
|
||
// Clean up the local ground truths. | ||
await cleanUpFiles(); | ||
|
||
// Ask user if they want to open the Playwright report. | ||
const shouldOpenReport = await confirm({ | ||
message: 'Do you want to open the Playwright report?', | ||
thetaPC marked this conversation as resolved.
Show resolved
Hide resolved
|
||
}); | ||
|
||
// User cancelled the operation with `Ctrl+C` or `CMD+C`. | ||
if (isCancel(shouldOpenReport)) { | ||
cancel('Operation cancelled'); | ||
return process.exit(0); | ||
} | ||
|
||
// User chose to open the Playwright report. | ||
if (shouldOpenReport) { | ||
brandyscarney marked this conversation as resolved.
Show resolved
Hide resolved
|
||
// Use spawn to display the server information and the key to quit the server. | ||
spawn('npx', ['playwright', 'show-report'], { | ||
stdio: 'inherit', | ||
}); | ||
} else { | ||
// Inform the user that the Playwright report can be opened by running the following command. | ||
log.info('If you change your mind, you can open the Playwright report by running the following command:'); | ||
log.info(color.bold('npx playwright show-report')); | ||
} | ||
|
||
if (shouldOpenReport) { | ||
outro("You're all set! Don't forget to quit serving the Playwright report when you're done."); | ||
} else { | ||
outro("You're all set!"); | ||
} | ||
|
||
await sleep(1000); | ||
} | ||
|
||
main().catch(console.error); |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Trying to update the ground truths from a branch that doesn't have some screenshots causes errors:
I'm not sure if this is expected or something we want to catch.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed! 28c2aa5