Cannot pass custom watermarks options to coverage reporters

[matejmihalik]

🐛 Bug Report

After the merging of #9572 and closing of #4103 I was excited to finally set my custom watermarks for jest coverage reports. Unfortunately, passing custom watermarks options via jest config doesn't work. The coverage will run, but the watermarks thresholds are the same defaults as they've always been.

Digging through the jest/istanbul codebase, the best explanation I could come up with is this. #9572 added the ability to define custom options for individual reporters. It then takes those options and passes them to report creation:
https://github.com/facebook/jest/blob/2dc87f226e4935005fe61acf8f995d947249a1b6/packages/jest-reporters/src/coverage_reporter.ts#L106-L118
However, notice that right after report is created, it gets executed with custom context. That context defines it's custom watermarks though, which most likely override the watermarks defined by the options set at report creation:
https://github.com/facebook/jest/blob/2dc87f226e4935005fe61acf8f995d947249a1b6/packages/jest-reporters/src/coverage_reporter.ts#L501-L505

As is evident from the code of those default jest watermarks getter, the lower threshold values are taken from the jest default, while the higher threshold is taken from jests coverageThreshold.global option.
https://github.com/facebook/jest/blob/2dc87f226e4935005fe61acf8f995d947249a1b6/packages/jest-reporters/src/get_watermarks.ts#L11-L36

This is not sufficient for me though, for the following reasons:

1. No ability to set the lower watermark threshold at all
2. The higher watermark threshold is tied to the coverageThreshold.global option, however, I don't want to have my report colour-coding tied in with how the jest evaluates the results. While I would love to have a "green" watermark for my coverage only at 100%, that does not mean that I want my jest to fail if my entire codebase is not at 100% (Because currently, it's not 🙂). Ideally, I don't want to use the coverageThreshold.global option at all.

To Reproduce

Steps to reproduce the behaviour: Try setting custom watermark options via jest config:

    coverageReporters: [
        [
            'html',
            {
                watermarks: {
                    statements: [75, 100],
                    branches: [75, 100],
                    functions: [75, 100],
                    lines: [75, 100],
                },
            },
        ],
        [
            'text-summary',
            {
                watermarks: {
                    statements: [75, 100],
                    branches: [75, 100],
                    functions: [75, 100],
                    lines: [75, 100],
                },
            },
        ],
    ],

Expected behavior

Custom watermarks options will apply to generated coverage reports.

envinfo

  System:
    OS: macOS 10.15.3
    CPU: (8) x64 Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
  Binaries:
    Node: 13.11.0 - /usr/local/bin/node
    Yarn: 1.22.4 - /usr/local/bin/yarn
    npm: 6.14.4 - /usr/local/bin/npm
  npmPackages:
    jest: 25.2.4 => 25.2.4

[notoriousmango]

can i look into this issue?

[szellcsaba-ge]

Reproduced as follows:

package.json / jest

    "coverageReporters": [
      [
        "text",
        {
          "watermarks": {
            "statements": [80, 90],
            "branches": [80, 90],
            "functions": [80, 90],
            "lines": [80, 90]
          }
        }
      ]
    ]

Result of jest --coverage:

envinfo

  System:
    OS: Windows 10 10.0.18363
    CPU: (12) x64 Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
    Memory: 4.30 GB / 15.75 GB
  Binaries:
    Node: 12.16.3 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.22.4 - ~\AppData\Roaming\npm\yarn.CMD
    npm: 6.14.5 - C:\Program Files\nodejs\npm.CMD
  npmPackages:
    @angular-builders/jest: ^9.0.1 => 9.0.1
    @types/jest: ^25.2.1 => 25.2.3
    jest: ^25.5.4 => 25.5.4
    jest-html-reporter: ^3.1.0 => 3.1.3
    jest-preset-angular: <8.2.1 => 8.2.0

[szellcsaba-ge]

"html" reporter with the same setting:

[dcwarwick]

So no one has commented on this for 18 months, but I'm still experiencing the same problem. I tried this to set the watermarks so that only 100% coverage would color green:

  coverageReporters: [[ 'html', { watermark: { statements: [80, 100], lines: [80, 100], functions: [80, 100], branches: [80, 100] } } ]]

but it doesn't seem to have any effect.

I did find the following workaround, based on @matejmihalik's notes above -- setting the global coverage threshold does set the upper watermark, but like him I don't actually want my jest to fail whenever anything falls short of 100%, so I did the following:

  coverageThreshold: {
    global: { branches: 100, functions: 100, lines: 100, statements: 100 },
    '**/*.js': { branches: 0, functions: 0, lines: 0, statements: 0 }
  }

This uses the global threshold to set the upper watermark, then the '**/*.js' line effectively nullifies the global threshold so that it doesn't actually have any effect. So I've achieved what I wanted, but it would be better if the coverage reporter settings could be passed through successfully somehow!

[github-actions]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity. Please open a new issue if the issue is still relevant, linking to this one.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.