GHA/Codecov: "Unchanged files with check annotations (Beta)" is noisy

[amotl]

Hi there,

as we always try to improve on flakyness on the bits related to the test harness whatsoever, I believe it might make sense to add this here - even if we might want to close it right away because it is not a real issue but just about noise.

Introduction

With minor patchsets like #10821 and #10823 being brought in, we are sometimes observing Codecov warnings which report changes in coverage even when no code changes were made.

@amotl asked:

Just curious why they only recently popped up here?

@mfussenegger stated:

No idea why it is sometimes popping up and sometimes not.

Observations

The noise we are actually looking at looks like this.

Check warning on line 170 in app/src/main/java/org/elasticsearch/bootstrap/BootstrapProxy.java
@codecov codecov / codecov/patch

app/src/main/java/org/elasticsearch/bootstrap/BootstrapProxy.java#L170

Added line #L170 was not covered by tests

Research

So, while the official documentation says:

• https://blog.codecov.io/2020/09/08/announcing-line-by-line-coverage-via-github-checks/
• https://docs.codecov.io/docs/github-checks-beta

and also gives some advice on how to investigate unexpected coverage changes:

• https://docs.codecov.io/docs/unexpected-coverage-changes

I have been able to collect some pointers where others are actually reporting nondeterministic behavior in this space, for both Codecov and GHA:

• "Unchanged files with check annotations" flags distracting lint errors sveltejs/svelte#5026
• "Unchanged files with check annotations" is disruptive and should be configurable actions/toolkit#457
• It should be possible to annotate files that are not in the commit actions/toolkit#470
• https://github.community/t/creating-annotations-for-failed-tests-in-files-not-in-the-commit-changeset/17060
• https://community.codecov.io/t/reported-change-in-coverage-when-no-code-changes-made/1346

Maybe this helps ourselves or others for either confirming or even getting hold of this issue.

With kind regards,
Andreas.

[amotl]

Others reported that those issues about reports from Codecov that lines are being added while they are actually not, might come from the fact that the upload of the coverage report to codecov.io has gone wrong or something. Indeed, when looking at [1] or [2], it says

Missing base report.
Unable to compare commits because the base of the pull request did not upload a coverage report.
Changes found in between b631d66...521faaf (pseudo...base) which prevent comparing this pull request.

[1] https://codecov.io/gh/crate/crate/pull/10821
[2] https://codecov.io/gh/crate/crate/pull/10823

[amotl]

While I don't know if I am on the right track on this, I tried to find similar observations by others and would like to report about it.

Easy fix?

If we are lucky, the problem might be easy to solve by manually uploading the latest commit on master once, like @hisabimbola described at commercetools/nodejs#179 (comment):

The simple fix I did was to manually push a report to the latest commit on master and PRs now [codecov has] a report to compare with.

The canonical documentation of this apparently is [1] and TEAMMATES/teammates#6791 as well as [2] are further references to it.

[1] https://docs.codecov.io/docs/error-reference
[2] https://twitter.com/codecov/status/835856171143684096

Hm?

Other reports about this are codecov/codecov-bash#134 as well as codecov/codecov-bash#137 and unfortunately, they are still open/unresolved. People at [3] and [4] apparently report about similar things.

[3] https://community.codecov.io/t/flag-n-to-set-the-base-report-does-not-work/973
[4] https://community.codecov.io/t/how-to-fix-missing-base-report-on-open-source-project/1559

[amotl]

FWIW, I also took a look back in time. At [1] everything seems to be still all right. However, from there things apparently start to deteriorate gradually and starting from [2], all pull requests processed by Codecov show the "Missing base report" flag more often than not.

While I don't want to blame anything before knowing more details, I wonder if Mergify might be involved to these hiccups in any way? However, I haven't been able to find any traces of people talking about this.

[1] https://app.codecov.io/gh/crate/crate/pulls?page=137&state=merged&order=-pullid
[2] https://app.codecov.io/gh/crate/crate/pulls?page=98&state=merged&order=-pullid

[amotl]

@mfussenegger and me believe that

part of the problem is that we partly do a rebase merge (either via GitHub UI or mergify). Then the hash that gets committed to master changes and Jenkins doesn't run on master for commits but only on PR. That is the reason why the codecov report for the commits in master is missing.

A likely solution would be to set up push on master as a trigger for tests on Jenkins which uploads the reports.

[amotl]

Apparently, https://github.com/crate/jenkins-dsl/pull/879 does not specify any triggers. I am now running the job [1] manually and will reopen this.

[1] https://jenkins.crate.io/job/CrateDB/job/crate_on_master

[amotl]

Hi again,

at crate-pdo, we recently had similar issues where the Scrutinizer service was not able to grok code coverage reports properly because the source checkout job step did not account for enough "depth" of the repository.

We have been able to mitigate it by adjusting the GitHub Actions "checkout" recipe through crate/crate-pdo@1c1693d92.

- name: Acquire sources
  uses: actions/checkout@v2
  with:
    fetch-depth: 2

With kind regards,
Andreas.

[amotl]

At [1], we can see that the "Missing base report" badge is still present on pull request checks more often than not.

[1] https://app.codecov.io/gh/crate/crate/pulls

[mfussenegger]

This hasn't been that much of an issue lately. Closing this