Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(coverage): c8 ignore hints actually work #2755

Merged
merged 1 commit into from Feb 3, 2023

Conversation

AriPerkkio
Copy link
Member

When writing this part of the guide initially I thought c8's ignore hints would not work, as their regexp is very strict and does not allow additional content such as @preseve keywords.

The ignore hints don't work with @vitest/coverage-istanbul since esbuild strips the comments before code is instrumented. Istanbul uses ignore hints to check whether coverage counters should be placed in given code block.

With @vitest/coverage-c8 the coverage filtering is done after the coverage report has been collected. The exclusion is done by comparing results to the actual sources where original comments are in place. v8-to-istanbul used by c8 uses ignore hints to see whether uncovered lines should be marked as covered.

@AriPerkkio AriPerkkio force-pushed the docs/c8-ignore-correction branch 2 times, most recently from 67b0d5c to 515cd2d Compare January 26, 2023 13:17
@AriPerkkio
Copy link
Member Author

The c8 tests seem really unstable now on CI. Randomly failing on both ubuntu-latest, 16 and ubuntu-latest, 18. I'm unable to reproduce this locally.

I'll do some more testing on CI and try to find root cause.

@AriPerkkio AriPerkkio marked this pull request as draft January 26, 2023 13:25
@AriPerkkio
Copy link
Member Author

After rebasing latest main this seems to be working fine. I think #2758 fixed the issue. Before #2758 the worker from earlier test run was interfering the next run, as coverage tests run two test runs in loop. This only happend on heavy CPU load. The #2758 doesn't still terminate workers early enough to stop writing unnecessary V8 reports but seems to help a lot.

With 0.28.3 I'm also seeing some really good performance and stability improvements. I've been testing the test/coverage-test C8 tests in 100x loop to reproduce these CI crashes locally.

async function runTests() {
+ for (const [round] of Array(100).fill(null).entries()) {
+   console.log(`Round #${1 + round}`)
    for (const threads of [true, false]) {
      for (const [directory, config] of configs) {
        await startVitest('test', [directory], {

100x rounds of pnpm test:c8 with version

  • 0.28.3: 2m29.773s, does not crash with this PRs changes
  • 0.28.2, 0.28.1, 0.28.0:
    • Without changes of this PR 3m34.741s
    • With this PR's changes crashes after 3-7 rounds
  • 0.27.3:
    • Crashes even without the changes of this PR after 71 rounds. Even that takes 3m50.894s. So 100x rounds would be somewhere around 4m30s maybe?
    • With changes of this PR crashes after 3 rounds.

@AriPerkkio AriPerkkio marked this pull request as ready for review January 27, 2023 16:44
@sheremet-va sheremet-va merged commit c0f91be into vitest-dev:main Feb 3, 2023
@AriPerkkio AriPerkkio deleted the docs/c8-ignore-correction branch February 3, 2023 09:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants