Support updating buf.lock

[nightlyone]

What would you like Renovate to be able to do?

I would like to be able to manage the buf.lock .

buf is a new tool and ecosystem registry to manage protobuf dependencies and is really taking off.

versions apparently get deleted from buf build after a certain time, so updating is important.

If you have any ideas on how this should be implemented, please tell us here.

https://docs.buf.build/configuration/v1/buf-lock (and surrounding documentation) describes the process that needs to be done from a users perspective.

It boils down to

• parsing an versioned buf.yaml
• extracting the module names from the list at the deps property
• calling buf mod update to update all or buf mod --only <module> to update only one

optional improvement on top that can improve the user experience, but should really be only options

• verify effects using buf generate which re-generate code from all *.proto files and checks code generation
• verify effects using buf build which uses all *.proto files, including modules updated, and bundles that into a buf image

Is this a feature you are interested in implementing yourself?

No

[github-actions[bot]]

Hi there,

You're asking us to support a new package manager. We need to know some basic information about this package manager first. Please copy/paste the new package manager questionnaire, and fill it out in full.

Once the questionnaire is filled out we'll decide if we want to support this new manager.

Good luck,

The Renovate team

[viceice]

• buf + dependabot Integration bufbuild/buf#1022
• Renovate support for buf.lock bufbuild/buf#1404

[viceice]

Seems to be a littlebit complicated, as there is no official api yet and the registry is only available via protocol buffers.

[trietsch]

@viceice As buf.build now uses their own built connect-web, they support application/json as Content Type, which means that even though they use a gRPC based server, you can talk to it with JSON. Here's an example listing all available tags with cURL for the googleapis/googleapis dependency (probably not the best example, but you could do it like this example below, just listing a single one and reverse the order):

curl 'https://api.buf.build/buf.alpha.registry.v1alpha1.RepositoryTagService/ListRepositoryTags' \
  -H 'content-type: application/json' \
  --data-raw '{"repositoryId":"c84e502e-8a35-4bdc-aae4-a5dbdf9a685c","pageSize":1,"reverse":true}'

{
  "repositoryTags": [
    {
      "id": "55fa2a4f-96ad-4c4e-9d18-3fedb39deba6",
      "createTime": "2022-11-02T15:03:22.295046Z",
      "name": "77bdd3dd26ea10c1b4f1c190e776a2d3fe4b3e8d",
      "commitName": "783e4b5374fa488ab068d08af9658438",
      "author": "bufbot"
    }
  ],
  "nextPageToken": "55fa2a4f-96ad-4c4e-9d18-3fedb39deba6"
}

And the documentation for this endpoint can be found here.

[mfridman]

@trietsch Indeed the backend is based on connect-go and supports the Connect protocol, which means the API will return plain JSON and can be used with any off-the-shelf tool.

Not sure what the status of this issue is, but I can probably help out here as this has come up quite a bit for us.

[jalaziz]

I know this discussion is specifically about buf.lock, but what about remote plugins? Is there an API for fetching the latest versions of remote plugins? Or can we use an existing data source for that?

[mfridman]

Yes, there's an API to fetch the latest Remote Plugins. It can either be done via ListCuratedPlugins (which is what we use on https://buf.build/plugins), or if you know the specific plugin via GetLatestCuratedPlugin.

Example

Assuming you have buf, jq and jo installed, here's a one-liner. (All of this can done via a normal fetch with JSON).

ListCuratedPlugins

jo page_size=250 | buf curl --schema=buf.build/bufbuild/buf --data @- https://buf.build/buf.alpha.registry.v1alpha1.PluginCurationService/ListCuratedPlugins | jq -r '.plugins[] | .owner + "/" + .name + ":" + .version'

Output:

apple/swift:v1.25.2
bufbuild/es:v1.5.0
bufbuild/knit-ts:v0.0.3
[...]

GetLatestCuratedPlugin

jo owner=protocolbuffers name=go | buf curl --schema=buf.build/bufbuild/buf --data @- https://buf.build/buf.alpha.registry.v1alpha1.PluginCurationService/GetLatestCuratedPlugin | jq -r '.plugin.version'

Output:

v1.31.0

[rarkins]

Hi @mfridman that would be great. The order we recommend to approach things is "datasource" first, then "versioning" (if necessary), then "manager".

Here's docs for datasources, and within the same directory there are plenty of examples: https://github.com/renovatebot/renovate/blob/main/lib/modules/datasource/readme.md

[artek-koltun]

Hi @mfridman, are you working on this?

[mfridman]

I'm not, haven't had the cycles to properly implement this.

[bourquep]

Is anyone working on this? If not, I might give it a try.