Implement requiredStatusChecks

[rarkins]

This is a:

• Bug report (non-security related)
• Feature request
• I'm not sure which of those it is

The requiredStatusChecks option is not fully implemented - it only works if you set it to null (i.e. NO required status checks).

We should implement the feature fully so that users can specify a list of status checks that are required for automerge to be "green" and we ignore the rest.

[stevenzeck]

Is it possible to see if the base branch has required status checks and go off of those? Our configuration looks like the below, but it always opens a PR since the required status checks we have on master prevent branch automerge.

  "devDependencies": {
    "automerge": true,
    "automergeType": "branch",
    "major": {
      "automerge": false
    }
  }
}

[rarkins]

@stevenzeck can you describe more about what branch protection configuration you have? e.g. does your repo have 5 status checks but only 3 of them are marked as required?

[stevenzeck]

For one of the repos I count 12, where 4 are required. The Renovate dashboard error says 2 of 4 required status checks are expected.

[stevenzeck]

@rarkins do you know of any ways for Renovate to work with GitHub required status checks while continuing to have automergeType of branch?

[rarkins]

I think it’s possible to look up via the commit hash like we did with status checks.

[stevenzeck]

Can you share how to configure our renovate files to do that? Right now we're disabling required status checks against master so that Renovate doesn't run into errors when merging from branches.

[rarkins]

Sorry, I misunderstood your previous question and thought it was about the new check runs.

Right now Renovate just waits for all test to be passed (then it merges the branch), or for any to be failed (it then raises a PR instead), or for for a timeout (default: 25 hours) in which case it also raises a PR.

I guess what you'd prefer is that Renovate can read which status checks you have configured in branch protection and wait only on those before deciding to merge or raise PR?

The alternative if that is too hard is to make you configure the same status checks in the Renovate config's requiredStatusChecks field so we don't need to look them up, although then you would be making "duplicate" settings in your repo, which is never good.

[stevenzeck]

@rarkins thanks for the information. Ultimately when Renovate is configured with automergeType: branch, I want it to wait until the required status checks are completed or until at least one required one fails before attempting to merge or creating a PR. It looks like

renovate/lib/platform/github/index.js

Line 498 in 8f8cffb

const gotString = `repos/${config.repository}/commits/${branchName}/status`;

doesn't return pending statuses perhaps?

[rarkins]

From memory, that query returns all statuses including pending. So Renovate will wait until they are all "passed", or one is "failed", or for the 25 hour pending timeout.

[stevenzeck]

@rarkins I looked into this a bit more and not all checks are present in the combined status response, which is what renovate looks at. For this repo, here are the PR "checks" and which API they show in:

• codeclimate/total-coverage (shows in status API only after it's not in the dark brown status)
• codeclimate/diff-coverage (shows in status API only after it's not in the dark brown status)
• Travis CI - Branch (shows in check-runs API)
• Travis CI - Pull Request (shows in check-runs API)
• Unibeautify - Branch (shows in check-runs API)
• Unibeautify - Pull Request (shows in check-runs API)
• WIP (shows in status API)
• codeclimate (shows in status API)
• continuous-integration/unibeautify/pr (shows in status API)
• continuous-integration/unibeautify/push (shows in status API)

I'm not sure where codeclimate/total-coverage and codeclimate/diff-coverage are. I know Travis CI recently stopped using statuses in favor of checks. But this is definitely part of our problem with automerging.

Edit: #2659 looks to have implemented check-runs into renovate

[stevenzeck]

Here's a good example. The first run of renovate found that @types/jest had an update, created the branch, and checked the status of the branch and it to be pending.

The second run only returned one status when it should have returned six. If you curl it now curl -H "Accept: application/vnd.github.antiope-preview+json" 'https://api.github.com/repos/unibeautify/beautifier-prettier/commits/renovate/jest-23.x/status' you'll see all six.

Edit: @rarkins I think the checks that start with a dark brown circle (like codeclimate/coverage) are not in the status response until they have been reported. When I open a sample PR and run the code immediately after I open it, it does not include the two codeclimate/coverage statuses. When I wait until the statuses are not in the dark brown color and run, they are returned.

[rarkins]

Thanks, that is very helpful. I think we can look for that message like “2 of 4 required status checks are expected” and if so then not raise a PR yet because it’s hopefully a temporary failure and not a permanent one

[stevenzeck]

Might want to consider retrieving the required status checks under the branch protection rules for the repo and branch, then compare that with the status and checks API response and see if they are included in that yet.

[denkristoffer]

We're also experiencing that setting e.g. 3 out of 6 checks to required makes Renovate "give up" trying to automerge, so to speak, even before the status checks have finished running.

However, today it looks like it actually came back later and merged it. That's a first 🙂

[rarkins]

I’ve added extra logic to the branch automerge try/catch so that it shouldn’t force a PR if the error is just because some requires status checks aren’t met. But beware you shouldn’t set checks that are incompatible with a branch automerge, such as required reviews or checks that only fire when there’s a PR.

[rarkins]

@stevenzeck thank you for the detailed logs. You were right - we were rewriting the err object and losing the context. Renovate should now detect automerge failures due to status checks pending and not raise PRs so eagerly. Leaving this issue open though because it's not technically the same thing.

[karfau]

If the current status is that the only option is to ignore all checks it should also be documented more clearly. Te current documentation is a bit hard to understand from my perspective.

If anything else is possible, it's also not well documented what the strings needs to contain to match certain checks.
E.g. on github they are displayed differently debending on the page, there is a workflow name (from yaml or file name), that contains one or more jobs, that can be run against a matrix, and in a PR even the event that triggered the check is part of the "label":

[rarkins]

@karfau confirmed: only null is supported today. A docs-only PR would be welcome - it's in this repo

[rarkins]

BTW I have a suspicion that some people want this to mean "only require certain status checks and ignore if others are incomplete or failing" while others want this to mean "don't merge until at least these status checks are met, and of course only if all others are passing too"

[karfau]

@karfau confirmed: only null is supported today. A docs-only PR would be welcome - it's in this repo

#7926

[glasser]

I'm trying to understand the current behavior of this option. I understand that an actual list doesn't work and the only choices are "default" and "null". For the default behavior, documented as "Currently Renovate's default behavior is to only automerge if every status check has succeeded.", does "every" mean "every status check that starts running on the branch within a period of time", or "every status check that's marked as required"? (I'm particularly curious about GitHub.)

[rarkins]

Renovate won't merge if any status check fails, even if it's not part of the non-required set

[Zyava]

Is it possible to configure renovate to merge branch if no jobs were started at all for the branch since it was created?
In many repos we have some logic built into .gitlab-ci.yml to detect changes and create jobs only for these changes.
For some commit generate by renovate no jobs spawned but we don't wish to set "requiredStatusChecks": null because it means renovate will merge even if job failed, not only when no job was spawned for the commit.

[viceice]

Nope, thats not possible, you can simply add a dummy job, which passes always and does nothing.

[Jolg42]

We have a use-case where this would be amazing

We have optional tests defined in https://github.com/prisma/e2e-tests/blob/7669f82bc665725881581d72cd82777f3c04e9e9/.github/workflows/optional-test.yaml

And all Renovate PRs are currently failing because of of the job of the optional test workflow is failing prisma/ecosystem-tests#1958

Not sure if there was a suggestion to implement the "opposite" that would be easier to maintain in this case to only ignore the optional tests?

[samuelfernandez]

An issue I'm noticing is that when retrieving status checks only one is retrieved instead of waiting for all. Check these logs:

DEBUG: Created Pull Request #507 (repository=ng-easy/platform, branch=renovate-github/nrwl-workspace)
DEBUG: PR is configured for automerge (repository=ng-easy/platform, branch=renovate-github/nrwl-workspace)
DEBUG: resolveBranchStatus(branchName=renovate-github/nrwl-workspace, ignoreTests=false) (repository=ng-easy/platform, branch=renovate-github/nrwl-workspace)
DEBUG: getBranchStatus(renovate-github/nrwl-workspace) (repository=ng-easy/platform, branch=renovate-github/nrwl-workspace)
DEBUG: branch status check result (repository=ng-easy/platform, branch=renovate-github/nrwl-workspace)
       "state": "pending",
       "statuses": []
DEBUG: check runs result (repository=ng-easy/platform, branch=renovate-github/nrwl-workspace)
       "checkRuns": [{"name": "commit-lint", "status": "completed", "conclusion": "success"}]
DEBUG: Branch status green (repository=ng-easy/platform, branch=renovate-github/nrwl-workspace)
DEBUG: Automerging #507 with strategy auto (repository=ng-easy/platform, branch=renovate-github/nrwl-workspace)
DEBUG: mergePr(507, renovate-github/nrwl-workspace) (repository=ng-easy/platform, branch=renovate-github/nrwl-workspace)
DEBUG: mergePr (repository=ng-easy/platform, branch=renovate-github/nrwl-workspace)
       "options": {"body": {"merge_method": "rebase"}},
       "url": "repos/ng-easy/platform/pulls/507/merge"
DEBUG: PR merged (repository=ng-easy/platform, branch=renovate-github/nrwl-workspace)
       "automergeResult": {
         "sha": "013bc8ac041699665f76e4b42779d4a5115cc110",
         "merged": true,
         "message": "Pull Request successfully merged"
       },
       "pr": 507

While you can see here ng-easy/platform#507 that one status check failed

[rarkins]

@samuelfernandez unfortunately that doesn't prove anything. That status check could have been added after. GitHub's UI indicates that Renovate's view at the time was correct too:

Isn't that literally saying "this PR was merged when it had 1 passing check"?

[samuelfernandez]

@rarkins exactly that... It is possible that the other checks from the CI pipeline were added after Renovate run, so it can be an unfortunate timing issue. That update was in "Pending status checks" phase, and in the same cycle Renovate created the PR and immediately merged it before other checks were added. I'm starting to suspect that it is because I have the CI configured to run on PR, but not on branches. Do you think it would be possible to somehow delay a bit the detection of status checks so that they are added?

[rarkins]

Do you have branch protections enabled? That's generally the best approach, because GitHub enforces it. Otherwise we could look into whether we can skip a status check immediately after PR creation, although that would be a different request to this current issue.

[samuelfernandez]

Do you have branch protections enabled? That's generally the best approach, because GitHub enforces it. Otherwise we could look into whether we can skip a status check immediately after PR creation, although that would be a different request to this current issue.

Yes I do. Maybe it is a permissions issue, since Renovate is self-hosted and runs with a personal token, so it still has permissions. Anyway, after configuring CI to run on pushes to Renovate branches now all checks are found in the commit, so it will likely solve the issue https://github.com/ng-easy/platform/pull/512/files