New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WIP: Enabling test coverage on e2e tests #31787
Changes from all commits
f8ef412
857939e
f4071e4
41c7efb
33cab53
67ef853
372e38a
b11e565
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -67,7 +67,7 @@ async function main() { | |
) { | ||
downloadDistOutput(FILENAME); | ||
timedExecOrDie('gulp update-packages'); | ||
timedExecOrDie('gulp e2e --nobuild --headless --compiled'); | ||
timedExecOrDie('gulp e2e --nobuild --headless --compiled --coverage'); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This should eventually be gated with There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agree. Adding an additional job to the CI matrix isn't that expensive, so we should probably just do all coverage measurements in a parallel job. |
||
} else { | ||
console.log( | ||
`${FILELOGPREFIX} Skipping`, | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Without this, the longer
--coverage
build will timeout on TravisThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As written, this will increase the time taken by all Travis builds because the "test" stage will now have to wait for (up to) a 30 minute "build" stage.
The good news is that we should be able to solve this problem with the move to CircleCI, which I believe allows for more sophisticated matrices (e.g. create a separate independent "coverage" stage, which starts at the beginning and runs in parallel with the "build" and "test" stages, which run serially.)
I expect to make progress on this in the next week or so, will loop you in when the coast is clear to try this out.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for reminding me about this, forgot to note it! My thinking was this would only be viable for something like the Nightly builds, and probably shouldn't be a part of CI during the development day. Even if Circle CI supports splitting the work off, e2e test coverage shouldn't fluctuate massively from one commit to another, and daily data would be sufficient as a metric.