feat(http): Etag support

[zharinov]

Changes

• Integrate ETag support into http API

Context

• Ref: Add back metadata for rubygems.org packages to get release notes / changelog #3670

Documentation (please check one with an [x])

• I have updated the documentation, or
• No documentation update is required

How I've tested my work (please select one)

I have verified these changes via:

• Code inspection only, or
• Newly added/modified unit tests, or
• No unit tests but ran on a real repository, or
• Both unit tests + ran on a real repository

[rarkins]

Does this now return the previous/existing response in the body with a 304?

What's the use case for this?

In npm for example, where etags are already successfully used, we don't cache the raw http body response. We cache the getPkgReleases response.

[zharinov]

Does this now return the previous/existing response in the body with a 304?

In npm for example, where etags are already successfully used, we don't cache the raw http body response. We cache the getPkgReleases response.

I am refactoring this Npm code to make all the data transformation to be performed inside the zod schema. Raw response will be parsed by schema inside http call, so res.body for http.getJson(url, schema) will contain ReleaseResult instead of npm-specific format. Subsequent runs of http.getJson(url, { etagCache }, schema) will bypass schema-driven parsing and return etagCache.data that was previously cached.

It won't protect us from malformed data from cache (since we skip the validation/transform), but type system will require us to have ReleaseResult in the etagCache.data field.

What's the use case for this?

I want to use it for long-term caching of https://rubygems.org/api/v1/versions/$package.json endpoint since we want rubygems datasource to return metadata. Once properly cached (and throttled), I don't see any benefits of dealing with /versions endpoint and the complexities it adds.

[rarkins]

I'm not sure you'll be able to drop /versions because we will be too slow if we stick to rate limits. Are you assuming that 304 won't count towards rate limit?

[viceice]

I think we should move the paginate logic to the getJson function, as it's the only one which will work anyways.

[viceice]

LGTM beside the open discussion

[zharinov]

Please merge this one, it's blocker for me

[renovate-release]

🎉 This PR is included in version 35.101.0 🎉

The release is available on:

• GitHub release
• 35.101.0

Your semantic-release bot 📦🚀