New datasource.getPackageMeta() function

[rarkins]

What would you like Renovate to be able to do?

Support datasource.getPackageMeta() which is an optional interface to complement datasource.getPackageReleases().

Did you already have any implementation ideas?

Sometimes we can get the list of releases efficiently, but fetching the metadata for the package (e.g. source URL) is not efficient, e.g. a separate and potentially rate-limited API. In such cases we should separate out the getReleases() logic - which might typically have a 10-15 minute cache time - from getPackageMeta(), which could even have a 1 day cache time. Then when doing a lookup we return the merged results of getPackageMeta() with getReleases()

[rarkins]

Example datasources where this could be used right away: crate, rubygems

[rarkins]

@JamieMagee how do you suggest this works with your new class based approach?

[JamieMagee]

Pretty well? If it's abstract, like getReleases, then each datasource has to provide an implementation:

renovate/lib/datasource/datasource.ts

Lines 30 to 32 in f54ff20

	abstract getReleases(
	getReleasesConfig: GetReleasesConfig
	): Promise<ReleaseResult | null>;

To make it optional, like handleSpecificErrors, provide a default implementation in the Datasource class:

renovate/lib/datasource/datasource.ts

Line 37 in f54ff20

handleSpecificErrors(err: HttpError): void {}

[rarkins]

Great. It would be optional, and probably needs a custom cache time per datasource for this, separate from the releases cache time

[rarkins]

@JamieMagee how would you suggest having a different cache and expiry for getPackageMeta() vs getReleases()?

caching is boolean, but I think this can be ok:

renovate/lib/datasource/datasource.ts

Line 16 in ac408e4

caching: boolean | undefined;

And here's an example of the crate datasource using the cache decorator:

renovate/lib/datasource/crate/index.ts

Lines 30 to 36 in ac408e4

	@cache({
	namespace: `datasource-${CrateDatasource.id}`,
	key: ({ registryUrl, lookupName }: GetReleasesConfig) =>
	`${registryUrl}/${lookupName}`,
	cacheable: ({ registryUrl }: GetReleasesConfig) =>
	CrateDatasource.areReleasesCacheable(registryUrl),
	})

Would we need two decorators?

[JamieMagee]

The cache decorator doesn't read the cache property. It has its own cacheable property that takes a function and returns a boolean.

I think it's already generic enough to be reused. The only caveat is that it can only be applied to async functions right now.

(I'm AFK this weekend, so this is from memory)

[rarkins]

@JamieMagee do you think that the approach taken for the crate datasource removes the need for getPackageMeta()? Or we should standardize on a getPackageMeta() with separate caching directive so that it doesn't need to be repeated so much in each manager? The next one would be Rubygems. I'm actually leaning towards scrapping my idea here, although I'm not sure and also feel bad you created a PR.

[JamieMagee]

I think it could still be useful to have if there are more managers that would use it. If it's just rubygems that could probably be handled internally.

Don't feel bad about the PR 😅

[viceice]

I thiunk it's useful. We can extract more metadata handling from datasources to more generalize and long term caching.

[rarkins]

OK. Maybe we can port over the crates logic first