Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cache): Compression for repository cache #15289

Merged
merged 9 commits into from
Apr 24, 2022

Conversation

zharinov
Copy link
Collaborator

Changes

  • Compress repo cache data with snappy algorithm
  • Migrate from previous revisions

Context

Documentation (please check one with an [x])

  • I have updated the documentation, or
  • No documentation update is required

How I've tested my work (please tick one)

I have verified these changes via:

  • Code inspection only, or
  • Newly added/modified unit tests, or
  • No unit tests but ran on a real repository, or
  • Both unit tests + ran on a real repository

@zharinov
Copy link
Collaborator Author

Compression rate is about x4

rarkins
rarkins previously approved these changes Apr 24, 2022
package.json Outdated Show resolved Hide resolved
@viceice
Copy link
Member

viceice commented Apr 24, 2022

what about buildin zlib?
https://nodejs.org/api/zlib.html

@viceice
Copy link
Member

viceice commented Apr 24, 2022

or pure js snappy

https://github.com/zhipeng-jia/snappyjs

@rarkins

This comment was marked as outdated.

@zharinov
Copy link
Collaborator Author

In this case, I think it makes sense to stick to built-in gzip

@zharinov
Copy link
Collaborator Author

zharinov commented Apr 24, 2022

My primary objective was to prefer speed over compression rate, but pure JS implementation is twice slower anyway

@zharinov zharinov marked this pull request as draft April 24, 2022 12:59
@zharinov zharinov marked this pull request as ready for review April 24, 2022 14:49
@zharinov
Copy link
Collaborator Author

For Brotli algorithm, compression rate is x8. Also I've added check that would short-circuit if the data isn't changed.

rarkins
rarkins previously approved these changes Apr 24, 2022
Copy link
Member

@viceice viceice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

otherwise LGTM.

can you check that the new code and also tests are null safe?

lib/util/cache/repository/impl/local.spec.ts Outdated Show resolved Hide resolved
lib/util/cache/repository/impl/local.ts Outdated Show resolved Hide resolved
@rarkins rarkins merged commit b9dc73a into renovatebot:main Apr 24, 2022
@rarkins rarkins deleted the feat/repo-cache-compression branch April 24, 2022 20:15
@renovate-release
Copy link
Collaborator

🎉 This PR is included in version 32.31.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 25, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants