test: add windows and c++ coverage #35670

bcoe · 2020-10-16T04:51:46Z

Experiment to see if collecting coverage for Windows will "just work".

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

bcoe · 2020-10-18T14:08:00Z

.github/workflows/coverage-linux.yml

@@ -23,14 +21,22 @@ jobs:
          python-version: ${{ env.PYTHON_VERSION }}
      - name: Environment Information
        run: npx envinfo
+      - name: Clone gcovr reporter
+        run: git clone -b 3.4 --depth=1 --single-branch https://github.com/gcovr/gcovr.git
+      - name: Clone patch for gcovr


@mhdawson do you remember why we're running gcovr from a branch and floating a patch? It would be nice to simplify this.

It been long enough that I don't remember the specifics. I assume it failed to run properly with out that patch. It's possible a later version of gcovr may have fixed it.

@mhdawson I've managed to get gcovr@4.2 working, installed directly from pip.

The trick was to continue running from the out/ folder, but to specify a --root pointing to the root project directory.

codecov.yml

bcoe · 2020-10-18T19:48:36Z

@richardlau @Trott @watilde @mhdawson , I think this is ready for review:

coverage is now collected for C++ using gcov, and for Windows.
coverage reports won't show up until both the Windows and linux coverage runs finish.
I've added codecov configuration that removes the graph from the report (the default message on the PR took up a lot of space.).

Why the drop in coverage?

This is the first time we've combined C++ and JavaScript coverage, and our C++ coverage is a bit below the JavaScript thresholds -- I also noticed that codecov.io's numbers are a bit lower than our uploaded reports, I think this is because it treats partial line misses the same as misses.

What's next

Once this has been running for a week or two, and we're happy, I think we should:

remove some of the coverage logic from our Makefile, I'd be happy to do this.
it would be nice to stop floating a patch for gcovr (it looks like we have been for years.).
should we consider redirecting coverage.nodejs.org, to the reports on codecov.io, and remove a section of the website we maintain?

.github/workflows/coverage-windows.yml

mhdawson · 2020-10-19T21:43:03Z

Is it expected that I have to log in to see what under and 266 more ?

It seems to want quite a bit of access to be authorized when I go to do that...

mhdawson · 2020-10-19T21:44:18Z

Does remove some of the coverage logic from our Makefile, mean that you can no longer run coverage locally?

.github/workflows/coverage-linux.yml

.github/workflows/coverage-windows.yml

bcoe · 2020-10-20T15:48:54Z

@mhdawson did you manage to open the reports? perhaps I can open an issue with codecov.io independently of this PR, since I don't believe this behavior is new.

I've noticed myself in the past, that when looking at Node.js' nightly reports, I sometimes get prompted to authenticate -- I wonder if it happens to folks who have a prior session with codecov.io that's expired.

bcoe · 2020-10-20T15:49:45Z

.nycrc

@@ -9,7 +9,7 @@
  "reporter": [
    "html",
    "text",
-    "lcov"
+    "cobertura"


I'm hoping this gives us better partial coverage (I noticed C++ reports branch level coverage).

targos · 2020-10-20T15:50:47Z

@bcoe It also asks me to authenticate, even in an incognito window.

bcoe · 2020-10-20T15:54:54Z

@thomasrockhu sorry to keep bothering you (I think we're on the right track to getting this configured appropriately for the project).

Folks are reporting that codecov.io is prompting them to authenticate to see the public reports:

@bcoe It also asks me to authenticate, even in an incognito window.
Is it expected that I have to log in to see what under and 266 more ?

Ideally there'd never be an authentication step for our users to view detailed reports, as the Node.js project is open source.

Is there a configuration setting I'm missing?

bcoe · 2020-10-20T18:18:02Z

@targos @mhdawson is the authentication issue a blocker, or is it okay if I open a tracking issue on codecov.io?

I seem to not be getting the authentication screen myself, either in an incognito window, or in another browser -- but, I have seen the behavior you're describing before -- I was hoping it was a bug that had been addressed, since I haven't been seeing a login prompt myself.

targos · 2020-10-20T18:55:00Z

It's not a blocker to me

mhdawson · 2020-10-21T16:59:27Z

Thanks for following up on the auth. As long as its not something that will be required long term it's not a blocker for me.

mhdawson

LGTM

nodejs-github-bot · 2020-10-21T23:23:41Z

CI: https://ci.nodejs.org/job/node-test-pull-request/33792/

bcoe · 2020-10-22T02:46:52Z

Landed in 7657f62

Collect Windows and C++ coverage. Configure codecov so that comments are more concise and are only left when coverage varies. PR-URL: #35670 Fixes: #35696 Refs: #35653 Refs: #35646 Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Richard Lau <rlau@redhat.com> Reviewed-By: Jiawen Geng <technicalcute@gmail.com> Reviewed-By: Michael Dawson <midawson@redhat.com>

thomasrockhu · 2020-10-22T13:18:03Z

Hi @bcoe, I noticed you opened this ticket so I'll address the issue there. Users shouldn't be seeing any problems viewing open source reports, so there's definitely an issue. Thanks for raising it up!

Collect Windows and C++ coverage. Configure codecov so that comments are more concise and are only left when coverage varies. PR-URL: #35670 Fixes: #35696 Refs: #35653 Refs: #35646 Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Richard Lau <rlau@redhat.com> Reviewed-By: Jiawen Geng <technicalcute@gmail.com> Reviewed-By: Michael Dawson <midawson@redhat.com>

nodejs-github-bot added the meta Issues and PRs related to the general management of the project. label Oct 16, 2020

bcoe added the wip Issues and PRs that are still a work in progress. label Oct 16, 2020

bcoe changed the title ~~test: windows coverage GitHub action~~ test: add windows and C++ coverage Oct 17, 2020

test: add windows and C++ coverage

df981ea

bcoe force-pushed the windows-coverage-action branch from aaa926e to df981ea Compare October 17, 2020 16:13

build: improve coverage configuration

d91b81d

nodejs deleted a comment from codecov-io Oct 18, 2020

bcoe mentioned this pull request Oct 18, 2020

Coverage report left in comments is "noisy" #35696

Closed

bcoe added 4 commits October 17, 2020 22:04

chore: attempt to build with gcov

656160f

build: ensure gcovr tool exists

9eac50b

chore: tweak codecov.yml file

f869255

multi-line command had issues

f5d6272

nodejs deleted a comment from codecov-io Oct 18, 2020

bcoe commented Oct 18, 2020

View reviewed changes

codecov.yml Show resolved Hide resolved

bcoe added 2 commits October 18, 2020 08:35

chore: delete more output files

40cb785

build: run test-cov rather than test-ci

fc7c05c

bcoe removed the wip Issues and PRs that are still a work in progress. label Oct 18, 2020

bcoe requested review from Trott, mhdawson, richardlau and watilde and removed request for mhdawson October 18, 2020 19:41

bcoe added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Oct 18, 2020

bcoe changed the title ~~test: add windows and C++ coverage~~ test: add windows and c++ coverage Oct 18, 2020

Trott approved these changes Oct 18, 2020

View reviewed changes

richardlau reviewed Oct 19, 2020

View reviewed changes

.github/workflows/coverage-windows.yml Outdated Show resolved Hide resolved

bcoe requested a review from mhdawson October 20, 2020 05:09

bcoe added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Oct 20, 2020

aduh95 approved these changes Oct 20, 2020

View reviewed changes

.github/workflows/coverage-linux.yml Show resolved Hide resolved

.github/workflows/coverage-linux.yml Show resolved Hide resolved

.github/workflows/coverage-windows.yml Show resolved Hide resolved

.github/workflows/coverage-windows.yml Show resolved Hide resolved

richardlau approved these changes Oct 20, 2020

View reviewed changes

gengjiawen approved these changes Oct 20, 2020

View reviewed changes

build: a couple more small tweaks to reports

c646f6d

bcoe commented Oct 20, 2020

View reviewed changes

nodejs deleted a comment from codecov-io Oct 20, 2020

mhdawson approved these changes Oct 21, 2020

View reviewed changes

This comment has been minimized.

Sign in to view

bcoe closed this Oct 22, 2020

bcoe deleted the windows-coverage-action branch October 22, 2020 02:46

richardlau mentioned this pull request Oct 22, 2020

Total regression in codecov coverage report looks suspicious #35759

Open

targos mentioned this pull request Nov 3, 2020

v15.1.0 release proposal #35948

Merged

BethGriggs mentioned this pull request Dec 10, 2020

v14.15.2 proposal #36476

Merged

BethGriggs added the dont-land-on-v14.x label Jan 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: add windows and c++ coverage #35670

test: add windows and c++ coverage #35670

bcoe commented Oct 16, 2020 •

edited

bcoe Oct 18, 2020

mhdawson Oct 19, 2020

bcoe Oct 20, 2020

bcoe commented Oct 18, 2020

mhdawson commented Oct 19, 2020

mhdawson commented Oct 19, 2020

bcoe commented Oct 20, 2020

bcoe Oct 20, 2020

targos commented Oct 20, 2020

bcoe commented Oct 20, 2020 •

edited

bcoe commented Oct 20, 2020

targos commented Oct 20, 2020

mhdawson commented Oct 21, 2020

mhdawson left a comment

This comment has been minimized.

nodejs-github-bot commented Oct 21, 2020

bcoe commented Oct 22, 2020

thomasrockhu commented Oct 22, 2020 •

edited

test: add windows and c++ coverage #35670

test: add windows and c++ coverage #35670

Conversation

bcoe commented Oct 16, 2020 • edited

Checklist

bcoe Oct 18, 2020

Choose a reason for hiding this comment

mhdawson Oct 19, 2020

Choose a reason for hiding this comment

bcoe Oct 20, 2020

Choose a reason for hiding this comment

bcoe commented Oct 18, 2020

Why the drop in coverage?

What's next

mhdawson commented Oct 19, 2020

mhdawson commented Oct 19, 2020

bcoe commented Oct 20, 2020

bcoe Oct 20, 2020

Choose a reason for hiding this comment

targos commented Oct 20, 2020

bcoe commented Oct 20, 2020 • edited

bcoe commented Oct 20, 2020

targos commented Oct 20, 2020

mhdawson commented Oct 21, 2020

mhdawson left a comment

Choose a reason for hiding this comment

This comment has been minimized.

nodejs-github-bot commented Oct 21, 2020

bcoe commented Oct 22, 2020

thomasrockhu commented Oct 22, 2020 • edited

bcoe commented Oct 16, 2020 •

edited

bcoe commented Oct 20, 2020 •

edited

thomasrockhu commented Oct 22, 2020 •

edited