Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inconsistent coverage levels between runs with "all" #1113

Closed
1 task done
jameswilddev opened this issue May 11, 2019 · 6 comments · Fixed by #1155 · May be fixed by sthamizhselvan/postManCollection#8
Closed
1 task done

Inconsistent coverage levels between runs with "all" #1113

jameswilddev opened this issue May 11, 2019 · 6 comments · Fixed by #1155 · May be fixed by sthamizhselvan/postManCollection#8

Comments

@jameswilddev
Copy link

Link to bug demonstration repository

I've seen a few other issues reporting similar symptoms, but their fixes are things I'm already doing or are not applicable (different tools): #981

Expected Behavior

Repeated runs produce the same result (which is expected to be 100% here).

Observed Behavior

Repeated runs produce 92% or 100% coverage.

Troubleshooting steps

  • still occurrring when I put cache: false in my nyc config

Environment Information

System:
    OS: Linux 4.18 Ubuntu 18.04.2 LTS (Bionic Beaver)
    CPU: (4) x64 Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
    Memory: 557.58 MB / 7.64 GB
  Binaries:
    Node: 11.15.0 - /usr/bin/node
    npm: 6.7.0 - /usr/bin/npm
  npmPackages:
    nyc: ^14.1.1 => 14.1.1 
    ts-node: ^8.1.0 => 8.1.0 
    typescript: ^3.4.5 => 3.4.5 
@jameswilddev jameswilddev changed the title Inconsistent Inconsistent coverage levels between runs May 11, 2019
@jameswilddev
Copy link
Author

Just compared a successful and failed run's json reporter output; the files are identical, but the coverage values in the terminal output still differ.

@jameswilddev
Copy link
Author

Also compared the build output (/dist) of a successful and unsuccessful build; they are again identical.

@jameswilddev
Copy link
Author

I can get 10 consistent passes by changing all to false (NYC config at the bottom of package.json). It seems whatever the bug is, it can be disabled that way. However, I then don't get coverage failures for modules with no coverage at all (as expected).

@jameswilddev jameswilddev changed the title Inconsistent coverage levels between runs Inconsistent coverage levels between runs with "all" May 12, 2019
@elliot-nelson
Copy link

Very interesting!

The issue here appears to be that istanbul is attempting to instrument both your original source typescript file, and your output dist/ javascript version. Whichever "wins" determines the file coverage object being populated. That's a problem because if the wrong one wins, the coverage lines don't match up (one file has ~33 lines, the other has ~35 lines).

For a reporter like text, this ends up just showing incorrect coverage %; for a reporter like html, it actually blows up when it tries to show the report because the annotator can't read the lines that should exist.

I think this might be a configuration conflict between your tsconfig and jasmine.json config, though. Either your nyc+jasmine runtime should be in ts-node, and target only .ts files (no pre-build required); or, your nyc+jasmine should target only .js files, and have no idea that typescript exists at all (not ideal, as you would be losing the TS->JS source mapping). Something is not right if istanbul knows about both the .ts and .js files.

@jameswilddev
Copy link
Author

That makes sense, and explains why all would trigger it if it's actively looking for files to include and picking up ones it shouldn't. Going to have to play and see if I can fix it my end but doubtful.

@bcoe
Copy link
Member

bcoe commented Jun 17, 2019

@elliot-nelson great digging. I've seen this issue for several years, and you're correct it seems to be a race condition between which JSON file is loaded from .nyc_output first.

I would love to better document how to avoid this problem, as it's a frequent complaint for TypeScript users. If TypeScript is being built before running tests, It might also be worth looking at c8 which I've been finding works a bit better for my TypeScript libraries (the disclaimer being it doesn't work if TypeScript is being dynamically built in tests).

coreyfarrell added a commit to coreyfarrell/nyc that referenced this issue Aug 1, 2019
Sometimes the coverage data produced by `nyc --all` is incompatible with
the coverage data produced by actual test runs.  This is generally due
to configuration error but results in inconsistent coverage reports or
in some cases causes `nyc report` to crash.  The workaround is
implemented in istanbul-lib-coverage to drop coverage data associated
with `nyc --all` when coverage data from a test run is found.  This
commit tags the coverage data when appropriate so the coverage merge
logic knows what to do.

Fixes istanbuljs#1113, istanbuljs#1124, istanbuljs#1148
coreyfarrell added a commit that referenced this issue Aug 2, 2019
…#1155)

* fix: Drop coverage produced by `nyc --all` for files that were tested

Sometimes the coverage data produced by `nyc --all` is incompatible with
the coverage data produced by actual test runs.  This is generally due
to configuration error but results in inconsistent coverage reports or
in some cases causes `nyc report` to crash.  The workaround is
implemented in istanbul-lib-coverage to drop coverage data associated
with `nyc --all` when coverage data from a test run is found.  This
commit tags the coverage data when appropriate so the coverage merge
logic knows what to do.

Fixes #1113, #1124, #1148
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants