Reduce noise for in-range dependency updates

[gr2m]

follow up to https://mobile.twitter.com/rarkins/status/1330928203343548424

What would you like Renovate to be able to do?

I'm migrating some of the Open Source projects I maintain to Renovate. A major concern is to limit the noise from dependency update pull requests, to reduce our maintenance burden, and more importantly, to not put off potential contributors who start watching the repository.

Did you already have any implementation ideas?

Greenkeeper had a great feature to address that problem: silent check for in-range updates. It would create a branch for the in-range update and await the CI checks. If all checks passed, the branch would simply be deleted. If the build fail, an issue would be created to inform the maintainers about a potential breaking change.

That means pull requests would only be raised for out-of range updates (as well as security updates). A weekly *.lock file update would make sure that there are no problems coming from the combination of multiple in-range updates.

[viceice]

you can use branch automerge feature, so renovate will push a branch with update(s), if ci succeeds and branch is exactly one commit ahead, it will merge and push directly to target branch, so no pr required.

also with default config:base presets renovate treats patch as minor updates, so only one update is proposed.

[gr2m]

the branch auto merge feature does not work when branch protection is enabled

[HonkingGoose]

I've helped @gr2m and the semantic-release project with getting set up with their Renovate bot config as best I could.

Here's their current renovate.json: https://github.com/semantic-release/.github/blob/main/renovate.json file that they use across multiple repositories.

---

Hi @gr2m, nice to see you here. 👋 😄

[rarkins]

@gr2m I observed Gatekeeper's approach and found it caused a lot of unnecessary noise in its own way, especially due to false positives.

One additional problem with implementing it in the Renovate context is that it would require state, and mo' state mo' problems. You wrote:

the branch would simply be deleted

Unfortunately, that's where the simple ends. Because deleted branches disappear (unlike closed PRs, of which there's a record), it means you can't query the repository/project to know what's been done before, so you need to store it somewhere else. Then sometimes state gets out of sync and you end up spamming - their - repo.

You also wrote:

That means pull requests would only be raised for out-of range updates (as well as security updates). A weekly *.lock file update would make sure that there are no problems coming from the combination of multiple in-range updates.

This part can already be done. Renovate by default won't raise PRs for in-range updates unless they're security updates. And if you enable lockFileMaintenance then you can get a weekly lock file refresh. I give a thumbs-up to this approach.

What you won't get from this approach is the continuous testing of new releases and Greenkeeper-style issues created as soon as tests fail. It's my belief:

1. This had too many false positives
2. Almost nobody ever pinned/downgraded anyway

To attempt to add some metrics to my gut feel on this, I checked the greenkeeper repo itself. I count 22 times the bot thought the Greenkeeper repo itself was breaking.

10 of them remain open and un-actioned to this day.

Of the other 12, they were mostly ignored and closed, but sometimes explicitly noted as false positives:

Examples:

---

---

---

---

I could not observe a single one of those 22 instances where the bot's issue caused the team members to make any change to the project dependencies.

In conclusion, I still don't like the idea of this feature. Here's my rationale:

• I'm yet to see a library author have complicated semver ranges that exclude every single broken dependency in a range. I doubt most have the time for that.
• If you're build an app, pin your dependencies anyway
• If you don't want the noise of updating every time there's a new update, put the updates on a schedule (e.g. weekly or monthly) and use Renovate's dependencyDashboard feature for visibility/approval
• Our upcoming Merge Confidence workflows has an aim that open source maintainers like yourself can just get a weekly or monthly bundle of "known good" updates so that your dependency maintenance overhead is minimal and you stop having to worry about this sort of thing most of the time

[gr2m]

mo' state mo' problems

I cannot argue with that :)

This had too many false positives

I don't agree here, it's not Greenkeepers fault. If your CI tests are unreliable, you have a whole set of problems, false positives by Greenkeeper's background checks was just a symptom. The difference would also be that the background check was default behavior for Greenkeeper, but would be opt-in for Renovate.

Almost nobody ever pinned/downgraded anyway

I did it several times and the value was very high in these cases. It's not about the instances of when a warning about a broken test from an in-range CI test resulted in pinning a dependency, but about about the amount of dependents and the impact on them.

---

to summarize, I would appreciate the feature for my own libraries, which usually have very reliable testing and 100% coverage, in order lower the barrier to accept contributions. But if the current architecture does not keep the necessary state then I wouldn't add it just to cover this particular use case.

How about a different approach: renovate could trigger repository dispatch events in case there is an update for a dependency, and leave the handling of that information up to the user? I could build my own action/app that implements the background check and share it with other renovate users, and enable an ecosystem of renovate-powered actions/apps that would use it as a means to act on npm registry events

[rarkins]

I did it several times and the value was very high in these cases. It's not about the instances of when a warning about a broken test from an in-range CI test resulted in pinning a dependency, but about about the amount of dependents and the impact on them.

Do I understand this right? Although you didn't pin or downgrade, it was helpful to be aware that people downstream using a fresh install of your package might be broken for some period? And if so, does that mean that the concern/threat is over once a working release has been published, or the non-working release has been pulled?

If "noise" wasn't a concern then couldn't you satisfy that today using the update-lockfile strategy? i.e. Renovate would bump the version in the lock file each time a new release is out, which triggers a build, and typically would be green and available for merge (including automatic).

Alternatively - a feature not supported today but pretty easy to add - Renovate could create branches to bump lock files and trigger tests, and only raise a PR if they fail. So normal scenario is that you might have 5-10 branches and no PRs but occasionally one branch will be updated and then raised as a PR if testing fails. You could still get a wholesale lock file maintenance PR ~weekly or whenever you want.

Repository dispatch events would have the same state challenge btw - Renovate would need a way to prevent duplicate notifications for the same version.

[gr2m]

Although you didn't pin or downgrade

I did pin until the problem was resolved.

The ideal flow would be:

1. An in-range update breaks CI
2. Create separate branch that pins to previous version
3. Create issue, mention problem with in-range version, add link to create PR from branch which pins version
4. Merge PR that pins version -> release patch version of my lib
5. Next time a release comes out and tests pass, unpin version again to previous range.

Repository dispatch events would have the same state challenge btw - Renovate would need a way to prevent duplicate notifications for the same version.

Yeah that makes sense.

If GitHub's API would return the information if a branch existed for a given name, e.g. by returning 404 Deleted vs 404 Not Found, would that be all the state you'd need to implement that feature (in theory)?

[HonkingGoose]

@rarkins I'm not sure if you actually want to implement this feature or not?

The general feeling I'm getting from your posts is that doing this would lead to "more state more problems", and that you didn't like the idea of it, but you might have changed your mind or found an easier way to solve the problem?

Alternatively - a feature not supported today but pretty easy to add - Renovate could create branches to bump lock files and trigger tests, and only raise a PR if they fail. So normal scenario is that you might have 5-10 branches and no PRs but occasionally one branch will be updated and then raised as a PR if testing fails. You could still get a wholesale lock file maintenance PR ~weekly or whenever you want.

[rarkins]

If it's a valid dependency update problem, we'd like to solve it. Sometimes difficult problems get an elegant solution eventually, once the code base progresses or maybe we just think about it more. I'll leave it open so that it's not forgotten and others can add ideas or sentiment.