Support new "file sync" manager

[rarkins]

What would you like Renovate to be able to do?

Sync repository files with external sources.

Describe the solution you'd like

If a repository contains a file X and that file is "downloaded" from some external source Y (e.g. a website, or other repository, or API) then we would like Renovate to be able to keep it "in sync" with its source. The source would require manual configuration though.

I haven't thought much about the config required. We might call it the file package manager.

It should be possible to support a remote filename different from the local filename.

How to match/sync multiple files in a repo? if we use fileMatch, it would be hard to correlate the files against their sources. I think we need either:

• Ability to define 1:1 mappings between local and remote file, and
• Ability to use parameterised mappings between a list of files and their remote equivalents (e.g. to sync a directory)

What does the PR look like? e.g. PR title, etc. Do we uniquely identify the update based on a hash of the file contents? e.g. Update file backend/Dockerfile (6e2a78). This would allow users to close/ignore a PR but still get a later update.

Describe alternatives you've considered

Typically this would otherwise be done manually, or maybe someone wrote a complex cron script to commit, git status, git commit, etc.

[JamieMagee]

I've actually got some experience with this use case. Scoop and Chocolatey are package managers for Windows, and both support this functionality—Scoop natively, Chocolatey with an extension.

Scoop allows you to specify a URL and a regex or jsonpath. The idea being, that you run the regex against the content of the URL, and that returns a version. The script checkver.ps1 is what gets run, and here's a sample for redis.

au provides similar functionality for Chocolatey. It allows you to run arbitrary scripts to get the version info (probably not the best idea for Renovate). The script Update-Package.ps1 calls the user defined script called au_GetLatest which returns a version (Actually an object with 32/64 bit versions + URLs). Here's a sample that I maintain for the Zulu JDK.

I think the Scoop method of providing a URL + regex/jsonpath is the better way to go. We should also have a special case for GitHub repos, where a user can provide a GitHub URL, and Renovate knows to run against the releases page.

[politician]

We should also have a special case for GitHub repos, where a user can provide a GitHub URL, and Renovate knows to run against the releases page.

Way to go!

@HonkingGoose is there anything else needed for this work to start/be prioritized?

[rarkins]

@politician

1. More work needs to be done on defining how it would work, especially whether it can fit in with our current approaches of managers and datasources. Then it will be moved to status:ready
2. It needs someone to volunteer to develop it. Would you like to be assigned?

[jamietanna]

Hey folks, I'd be interested in picking this up - as we're looking at a way of keeping OpenAPI specifications up-to-date across repos - and wondered what the next steps would need to be to better define this?

[jamietanna]

We had a bit of a discussion about this on the Renovate Slack, and @TWith2Sugars and I are going to start putting some work towards this next week 🙌

[jamietanna]

Tony and I have caught up, and we've come up with the following.

We're thinking that - once we're happy with the below - we'll start on adding the ability to sync files via a GitHub repo, as this fits our usecase, then we can iterate over the rest of the functionality.

Requirements

• New datasource (file)
• New manager (file)
  • For working with github, gitlab, http, etc
• Integrate with hostRules
• Ability to pin to:
  • A branch - i.e. keep me in sync with the latest Git SHA that's currently on this branch
  • A SHA/tag - i.e. if I change the contents of this file, revert it back to this version
• Ability to map from a "source" (remote) location to a "destination" in the current directory
• Introduce a lockfile
  • This would allow a view of how the latest file contents has been resolved, especially without looking into the Git history
  • This would be a custom lockfile (see below) which would be defined through a new JSON Schema

Expected usage

Using a new top-level fileManagers type, which would be added into our renovate.json / config.js:

{
  "fileManagers": [
    {
      "platform": "github",
      "source": "github.com/deliveroo/someapi/openapi.yml",
      "destination": "my/local/openapi.yml",
      // or empty for latest on the default branch
      "constraint": "abcd37f323f",
      // **ALTERNATIVELY**
      "constraint": "commitsha:abcd37f323f",
      "constraint": "branch:staging",
    }
  ]
}

The constraint above is a version to pin to, or the "current" version if omitted. Alternatively could be called versionPinning, version, commitish, if it's a Git repo.

Out of scope

• Being able to specify v7.x, to resolve to i.e. v7.1.2
• Multiple constraints - we've considered that constraints could be possible to have multiple options, but we can't see a usecase where this may be necessary

Inspiration

We've also considered how Paket has done this

[viceice]

i though the idea was to reuse regex managers to custom managers, and add new sub types

[rarkins]

@jamietanna are you proposing that Renovate commits its own "lock file" to the repo to keep track of state for such sync'd files?

I would like to consider using the repository cache for this instead. e.g. we can strongly recommend the repository cache is enabled when the file manager is in use, otherwise it's very inefficient (downloads the same file to check and compare it every run)

[jamietanna]

Using the repository cache sounds like a good shout. The lockfile is more for a user-facing means to present some info about this, but if you think it's overkill / not appropriate we can remove that from the plan

[jamietanna]

@rarkins just to confirm on this, would you agree or disagree with also having a user-facing lockfile?

@viceice is that in reference to #21760 ?

Is there anything that would be worth confirming before we start on this? We're thinking we'll start with the new datasource

[rarkins]

I prefer to avoid a user-facing lock file

[secustor]

viceice is that in reference to #21760 ?

He is referring to #19066

[jamietanna]

Tony and I have just started looking at this, and were looking to start implementing a new datasource, when we spotted GithubTagsDatasource.

Instead of creating a new datasource, we're thinking that a better plan of action may be to start working on adding a new package manager, via docs:

• called file
• only support GithubTagsDatasource for now

We've spotted that there is a git versioning, should we continue to use this, or create a fresh one to avoid re-using the "hack" for submodules? Looking at it a little, we see it may only support git SHAs, so it may not be usable.

We've also spotted the git-submodules manager appears to be using git CLI commands, I assume that trying to do similar here isn't ideal, and we'd prefer to use API calls?

Once we've done this, we'll look at adding in other functionality, such as pinning to a specific branch.

[guyguy333]

Tony and I have just started looking at this, and were looking to start implementing a new datasource, when we spotted GithubTagsDatasource.

Instead of creating a new datasource, we're thinking that a better plan of action may be to start working on adding a new package manager, via docs:

• called file
• only support GithubTagsDatasource for now

We've spotted that there is a git versioning, should we continue to use this, or create a fresh one to avoid re-using the "hack" for submodules? Looking at it a little, we see it may only support git SHAs, so it may not be usable.

We've also spotted the git-submodules manager appears to be using git CLI commands, I assume that trying to do similar here isn't ideal, and we'd prefer to use API calls?

Once we've done this, we'll look at adding in other functionality, such as pinning to a specific branch.

IMO, sounds like a good plan to use GithubTagsDatasource. Please consider GitLab support too (GitlabTagsDatasource is also available), thanks :)

[loosebazooka]

I came across this discussion looking for a way to sync external *.proto files into a repository for code-gen. Is there an update here or is this feature not ready/not planned?

[HonkingGoose]

This used to be an issue but we decided to only have actionable issues. Because we're still figuring out the requirements or best way to do it, we converted it to a discussion.

I don't know if we still plan to do this feature. I'll let a maintainer respond. 😄

By the way, this issue seems a bit related, in that it talks about a more generic type of manager:

• Generic manager that understands json and yaml structure. #15193

[jamietanna]

I've just seen Vendir which we support and thought that this may work for use-cases we've got, but the upstream project only seems to support Helm/Docker, which is fair

Hope to maybe get around to taking a stab at this before long, but if anyone else fancies getting into it, feel free!