ReferenceError: cov_${id} is not defined

[HugoKawamata]

Context

At Tanda, we run tests on CircleCI as part of our deployment process. Recently when running with --coverage flag, our tests are only failing on CircleCI, not locally. @builtbyproxy and I spent a whole day walking through the process to identify what may throw the error mentioned below. These are our findings.

Prerequisites

package.json versions

jest 24.8.0
react-native 0.59.9
babel-jest 24.8.0
react: 16.8.3

We touch:

• istanbul-lib-instrument
• babel-plugin-istanbul
• babel-jest
• istanbul-reports

Description

When running tests in our local env, everything passes as intended. However, when deployed to CircleCI, we get an error in the form of ReferenceError: cov_14jyvcatev is not defined. The id that comes after the cov_ changes depending on which file it is referring to. The file referred to by the cov_${id} has (so far) never been the file that is being tested.

This only happens on CircleCI, and when the jest --coverage flag is on.

This also only happens to one test per test run. It's always the same test, and the cov_id is always the same (but does not refer to the test or the tested component that fails). For example, we test component B, and component J's cov_id will be the cause. We have had cases where the two components are completely unrelated.

Generating the cov_id

This cov_${id} is generated in istanbul-lib-instrument/dist/visitor.js using the function genVar. genVar is called in the constructor for the VisitState class, and is assigned to this.varName. A VisitState object is created in the function programVisitor and used in the enter and exit functions, which is the default export for istanbul-lib-instrument/dist/visitor.js

The cov_id's journey

With our programVisitor default export we found it is imported in istanbul-lib-instrument/dist/index.js and exported again as programVisitor.

Then it is imported by babel-plugin-istanbul/lib/index.js and is given types, realPath, and other options. With the result being assigned to this.__dv__:

Then it gets imported and exported by babel-jest/build/index.js and exports it as a part of the transformer (it's the babelIstanbulPlugin part)

We logged transformResult and got a big string with two consistent values of code and map. code looked something like this:

code: 'var cov_14jyvcatev = function () {\n  var path = "/usr/local/repo/src/components/Cards/ShiftAcceptance.js",\n...
// a LOT more lines

This gave us our first reference to the cov_${id} that was being used as a value, thus pointing to a potential ReferenceError: cov_id is not defined.

We searched our entire repo for instances of code being used and found the file istanbul-reports/lib/html/annotator.js running a map on the code variable after it's been split on newlines to generate an array of code lines. The istanbul-reports code looks like this:

When running the same split on the same code string from transformResult (as mentioned above), we got an array looking like this:

Furthermore, when scrolling through we noticed instances of cov_id value being specifically mutated. This is the only place we can find reason for a ReferenceError: cov_id is not defined. (Pictured is the babel-plugin-graphql-tag import example, however the cov_id increments are inserted throughout our code as well.)

istanbul-report generates an interactive directory of coverage reports. In these reports, there is a counter beside each code line which indicates how often it's called in tests. The GUI for this looks like this:

We believe that there is a race condition assigning the cov_${id} which isn't finishing before babel(or istanbul-reports or something?) tries to increment it.

[HugoKawamata]

This issue was posted in the old organisation (gotwarlost/istanbul#922) by mistake first. There has already been discussion with another community member, @robertleeplummerjr, who created a package (npmjs.com/package/istanbul-spy) that seems to hack a fix in by prepending a definition for each cov_id variable to each generated code string/array.

[coreyfarrell]

I believe #481 fixes this bug. This patch is released to istanbul-lib-instrument@4.0.0-alpha.1 and above as well as babel-plugin-istanbul@6.0.0-beta.0. I'm not sure what to suggest right now for jest users experiencing this issue as jest controls the version of istanbul packets you use.

[builtbyproxy]

Awesome thanks @coreyfarrell
We've temporarily patch-packaged jest to hit our istanbul for testing so let us repeat that with this fix and let you know!

We appreciate the quick response!

[coreyfarrell]

See also jestjs/jest#9192 for WIP updating jest to use the new versions of istanbul.

[coreyfarrell]

I believe this issue is resolved through #481, now released to istanbul-lib-instrument 4.0.0. If anyone can reproduce the error with that version please comment and I can reopen this issue.

[HugoKawamata]

Hey @coreyfarrell, the new Jest version with upgraded Istanbul packages has been released, and I've begun attempting to upgrade (which involves un-commenting out all the tests affected by this bug). It seems that this bug has not been fixed in babel-plugin-istanbul@6.0.0 and istanbul-lib-instrument@4.0.0. These versions were added to my yarn.lock file automatically during the install of jest 25.1.0.

babel-plugin-istanbul@^6.0.0:
  version "6.0.0"
  resolved "https://registry.yarnpkg.com/babel-plugin-istanbul/-/babel-plugin-istanbul-6.0.0.tgz#e159ccdc9af95e0b570c75b4573b7c34d671d765"
  integrity sha512-AF55rZXpe7trmEylbaE1Gv54wn6rwU03aptvRoVIGP8YykoSxqdVLV1TfwflBCE/QtHmqtP8SWlTENqbK8GCSQ==
  dependencies:
    "@babel/helper-plugin-utils" "^7.0.0"
    "@istanbuljs/load-nyc-config" "^1.0.0"
    "@istanbuljs/schema" "^0.1.2"
    istanbul-lib-instrument "^4.0.0"
    test-exclude "^6.0.0"

babel-plugin-istanbul@^6.0.0:
  version "6.0.0"
  resolved "https://registry.yarnpkg.com/babel-plugin-istanbul/-/babel-plugin-istanbul-6.0.0.tgz#e159ccdc9af95e0b570c75b4573b7c34d671d765"
  integrity sha512-AF55rZXpe7trmEylbaE1Gv54wn6rwU03aptvRoVIGP8YykoSxqdVLV1TfwflBCE/QtHmqtP8SWlTENqbK8GCSQ==
  dependencies:
    "@babel/helper-plugin-utils" "^7.0.0"
    "@istanbuljs/load-nyc-config" "^1.0.0"
    "@istanbuljs/schema" "^0.1.2"
    istanbul-lib-instrument "^4.0.0"
    test-exclude "^6.0.0"

I think we're going to disable code coverage now, as we'd rather have a functioning test suite than code coverage. Choosing the other path would be rather ironic in my eyes, using a code coverage package that prevents additional tests from being added to the codebase.

[builtbyproxy]

@HugoKawamata Can we re-open this please until it is fixed 👍

[robertleeplummerjr]

In my case GPU.js and Brain.js, we're working with new Function(compiledSource), where the context is lost within compiledSource, or glsl generated from JavaScript, and so I'm not sure the changes from the PR would affect that use case, which would have us still detecting and removing the injected entities.

[coreyfarrell]

I appreciate the trouble everyone is having but without a link to a (minimal) reproduction it's unlikely I'll be able to help.

[HugoKawamata]

Reproducing this is (I assume) almost impossible, or at least would be very time consuming, since

1. it only fails on our CircleCI server
2. we'd have to create a new project since our software is proprietary
3. the bug seems to arbitrarily select which new tests cause it to fail

I don't think I can reasonably provide much more information than what is already in the PR body above. Sorry if this means the issue will go unsolved.

[coreyfarrell]

One thing I just thought of, if anyone is running functions remotely those functions need to be excluded from coverage. For example if using selenium-webdriver to run a function in the browser:

/* istanbul ignore next */
browser.executeScript(() => console.log('this is run in the browser'));

The way this works is browser.executeScript runs Function.toString() on the argument then tells the browser to run that function. If this is being done from a file which is not already excluded by istanbul then you must provide the ignore hint otherwise the stringified function will contain references to istanbul globals which do not exist when the browser evaluates that code.

This is the only tip I can give without seeing a demonstration of the problem. Absolutely nothing wrong with using istanbul to measure test coverage of proprietary software but if you need help then it is your responsibility to provide the necessary information.

[robertleeplummerjr]

End to end steps to reproduce:

1. Install nyc.
2. Install mocha.
3. Create a file, thing.js, and fill it with this:

   function aFunction(a) {
     if (a) {
       return 'there was an a';
     }
     return 'no a';
   }
   module.exports = { aFunction };

4. Create another file, test.js, and fill it with this:

   const { aFunction } = require('./thing');
   console.log(new Function('return ' + aFunction.toString()).toString());
   console.log(new Function('return ' + aFunction.toString())()()); // this will throw

5. Run this as if it was a test:

     nyc mocha test.js

6. See mangled javascript that one cannot remove unless they were to use some fancy regex or a javascript parser, and as well the function exception where cov_2e4w61yxd7 cannot be found in this context:

      function anonymous() {
        function aFunction(a){cov_2e4w61yxd7().f[0]++;cov_2e4w61yxd7().s[0]++;if(a){cov_2e4w61yxd7().b[0][0]++;cov_2e4w61yxd7().s[1]++;return'there was an a';}else{cov_2e4w61yxd7().b[0][1]++;}cov_2e4w61yxd7().s[2]++;return'no a';}
      }

There should be a way to prevent this. Even if it is an inline comment or something.

[coreyfarrell]

@robertleeplummerjr you need to add /* istanbul ignore next */ before function aFunction(a) { in thing.js. If you are using function.toString() and evaluating that in a different context then you must do this.

[robertleeplummerjr]

@robertleeplummerjr you need to add /* istanbul ignore next */

I am not going to fill my source code with a bunch of istanbul comments. I want code coverage, I just don't want it in this exact and specific context. That would be the equivalent of turning off code coverage.

It would be easy to detect calls to new Function(firstArgument) and filter these artifacts just before it is sent in and that would be the one and true way of solving this issue.

[coreyfarrell]

@robertleeplummerjr you need to add /* istanbul ignore next */

I am not going to fill my source code with a bunch of istanbul comments. I want code coverage, I just don't want it in this exact and specific context. That would be the equivalent of turning off code coverage.

It would be easy to detect calls to new Function(firstArgument) and filter these artifacts just before it is sent in and that would be the one and true way of solving this issue.

istanbul does not support turning off coverage "in a specific context". Code is instrumented at the source level so in the example of a specific function it's either instrumented or it's not. Injecting source parsing and manipulation to the runtime so we can de-instrument code being passed to new Function is not practical.

I feel like maybe c8 would be a better fit for your needs as it works without modifying the sources (so aFunction.toString() would be the original source).

[robertleeplummerjr]

Injecting source parsing and manipulation to the runtime so we can de-instrument code being passed to new Function is not practical.

Ok. What we simply had an option to turn on checking if the value exists? Like:

(typeof cov_id !== 'undefined' ? cov_id++ : undefined)

This way when we're using new Function it is assumed the person knows what they are doing, and when we do an end to end test. Too, istanbul won't break everything, and we get away with not inserting tons of comments to remove istanbul. Also, this issue would likely be resolved, at least for the most part.

[robertleeplummerjr]

I feel like maybe c8 would be a better fit for your needs as it works without modifying the sources (so aFunction.toString() would be the original source).

Ty for pointing me to c8!

[coreyfarrell]

Using (typeof cov_id !== 'undefined ? cov_id++ : undefined) would massively increase the output and create additional complexity.

Does c8 solve your issue? IMO istanbul cannot support the use case of not ignoring coverage on a function but then using Function#toString on that function to execute the code in a different context.

[robertleeplummerjr]

Does c8 solve your issue?

For me, it solves the issue.

[coreyfarrell]

I'm going to close this issue as out of scope. istanbuljs is unable to support the use case of evaluating / running Function#toString in a different context.

Using c8 for situations where this is needed is my recommendation.

[timcharper]

In our case, we hit this case in a test-only code path, this simple hack seems to work:

/**
 * Hack to work around coveralls issue which prevents functions from being detached and evaluated in a new bound context.
 *
 * See https://github.com/istanbuljs/istanbuljs/issues/499
 *
 * @param fnSource Javascript function source to evaluate, usually sourced from `Function.toString()`
 * @returns
 */
function removeCovCalls(fnSource: string) {
  const regex = /\bcov_[^;,]+[;,]/g;
  return fnSource.replace(regex, "");
}

[arpanhalder7384]

If you are using jest for your code coverage & babel-plugin-istanbul to instrument your code then disable babel-plugin-Istanbul for your test files [eg: index.test.jsx / index.test.js/ index.test.ts]. You can disable it in babel.config.js where you are configuring your Istanbul.

babel.config.js

"plugins" : [ ["istanbul",{ "exclude" : "*.test.jsx", "include" : "*.jsx" } ] ]