New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
plumbing: format/packfile, prevent large objects from being read into memory completely #330
plumbing: format/packfile, prevent large objects from being read into memory completely #330
Conversation
This PR adds code to prevent large objects from being read into memory from packfiles or the filesystem. Objects greater than 1Mb are now no longer directly stored in the cache or read completely into memory. This PR differs and improves the previous broken go-git#323 by fixing several bugs in the reader and transparently wrapping ReaderAt as a Reader. Signed-off-by: Andrew Thornton <art27@cantab.net>
@hiddeco This should fix the problems! |
} | ||
return r, nil | ||
case plumbing.OFSDeltaObject: | ||
deltaRc, err := asyncReader(p) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This PR differs from the previous #303 here by wrapping the reader in a reader at - so as to prevent problems from the scanner moving on etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was also a bug here in the original #303 that this fixes.
} | ||
|
||
discard := offset - basePos | ||
if basePos > offset { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This line was changed from the original PR where it was if discard < 0 {
discard
here is a uint not an int64 - so checking if it's less than 0 will always fail.
This was the cause of the truncated checkouts.
@radeksimko would you be able to test? |
1a104f5
to
22550b6
Compare
@zeripath I can confirm that our relevant downstream tests (which have go-git in codepath) are passing with |
thanks for checking! On my tests I couldn't see a speed difference but did you see one? |
Thank you for the quick turnaround on the second attempt @zeripath 🥇 As you can see above, I created a PR to lazily let CI run some tests we added to cover some of the previously failing parts. Will probably not get into testing it in detail this week, as we worked around the major issues for now by relying less on go-git. |
@hiddeco have you noticed any problems with this? |
Anyone found any problems? |
I have thus far not found any problems but have not attempted write / push operations yet. |
Good fix,I think it will save me! How to test it? The github.com\go-git\go-git@v4.7.0+incompatible\repository.go:14:2: use of internal package gopkg.in/src-d/go-git.v4/internal/revision not allowed |
I've tested a repository with big blobs (and one big commit data) both packed a unpacked. Everything seems to work ok. |
So do we think we're ready to try again? Should I make it configurable somehow? |
Sorry, I had a major family issue that now is solved. Can you make it configurable the limit and also an option to disable? |
So the only way I can see to make this configurable beyond just the recompiling and replacing the constant with MaxInt at link time using We would need the flags to be available at lines 66 and 67 here: go-git/plumbing/format/packfile/fsobject.go Lines 50 to 69 in fb6894b
which would mean that the signature of Reader would need to change or (better) If That would mean changes for the related functions in We would also need to add changes to If that's acceptable I'll push up changes that do that. |
This commit moves the LargeObjectThreshold into an option set on the ObjectStorage. Unfortunately this means that the LOT has to percolate down the calling tree. Signed-off-by: Andrew Thornton <art27@cantab.net>
Proposed Patch
|
I've pushed it up and changed the first comment to match the new changes. |
I was thinking of an environment variable, to change the threshold. Maybe is the better option? |
The downside of environment variables is that they may grant configuration access to end-users of the library implementation that you do not actually want them to have. |
True, but in this case is something about performance, which I don't think is a bad thing. |
That depends on how your application is utilizing this library, if it is like ours, the performance configuration you want to have is on a per repository basis while the runtime is shared. |
relying on environment variables seems like something that a downstream application should decide |
Following the merging of go-git/go-git#330 we can now add a setting to avoid go-git reading and caching large objects. Signed-off-by: Andrew Thornton <art27@cantab.net>
Following the merging of go-git/go-git#330 we can now add a setting to avoid go-git reading and caching large objects. Signed-off-by: Andrew Thornton <art27@cantab.net>
Following the merging of go-git/go-git#330 we can now add a setting to avoid go-git reading and caching large objects. Signed-off-by: Andrew Thornton <art27@cantab.net>
* master: Update github.com/xanzy/ssh-agent to v0.3.1 Document the push refspec format Add support to push commits per hashes better tests plumbing: gitignore, Read .git/info/exclude file too. plumbing: packp, Add encoding for push-options. Fixes go-git#268. go-git: Add field `Options` to `PushOptions`, wire functionality. git: add --follow-tags option for pushes add tests examples: add find-if-any-tag-point-head Add RemoteURL to {Fetch,Pull,Push}Options plumbing/storer/object: improve grammar Go Doc (go-git#350) plumbing: format/packfile, prevent large objects from being read into memory completely (go-git#330)
… memory completely (go-git#330) This PR adds code to prevent large objects from being read into memory from packfiles or the filesystem. Objects greater than 1Mb are now no longer directly stored in the cache or read completely into memory. This PR differs and improves the previous broken go-git#323 by fixing several bugs in the reader and transparently wrapping ReaderAt as a Reader. Signed-off-by: Andrew Thornton <art27@cantab.net>
[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [cloud.google.com/go/datastore](https://togithub.com/googleapis/google-cloud-go) | require | minor | `v1.8.0` -> `v1.10.0` | | [cloud.google.com/go/pubsub](https://togithub.com/googleapis/google-cloud-go) | require | minor | `v1.24.0` -> `v1.27.1` | | [cloud.google.com/go/storage](https://togithub.com/googleapis/google-cloud-go) | require | minor | `v1.24.0` -> `v1.28.1` | | [github.com/go-git/go-git/v5](https://togithub.com/go-git/go-git) | require | minor | `v5.4.2` -> `v5.5.0` | | [github.com/google/go-cmp](https://togithub.com/google/go-cmp) | require | patch | `v0.5.8` -> `v0.5.9` | | [golang.org/x/sync](https://togithub.com/golang/sync) | require | minor | `v0.0.0-20220601150217-0de741cfad7f` -> `v0.1.0` | | [google.golang.org/api](https://togithub.com/googleapis/google-api-go-client) | require | minor | `v0.91.0` -> `v0.103.0` | --- ### Release Notes <details> <summary>go-git/go-git</summary> ### [`v5.5.0`](https://togithub.com/go-git/go-git/releases/tag/v5.5.0) [Compare Source](https://togithub.com/go-git/go-git/compare/v5.4.2...v5.5.0) #### What's Changed - \*: add collision resistent SHA1 implementation by [@​pjbgf](https://togithub.com/pjbgf) in [go-git/go-git#618 - \*: replace go-homedir with os.UserHomeDir by [@​mvdan](https://togithub.com/mvdan) in [go-git/go-git#535 - Remote: add RemoteURL to {Fetch,Pull,Push}Options by [@​noerw](https://togithub.com/noerw) in [go-git/go-git#375 - Remote: Push, add support to push commits per hashes by [@​tjamet](https://togithub.com/tjamet) in [go-git/go-git#325 - Remote: Push, add ForceWithLease Push Option by [@​john-cai](https://togithub.com/john-cai) in [go-git/go-git#404 - Remote: PushOptions add push-options by [@​S-Bohn](https://togithub.com/S-Bohn) in [go-git/go-git#399 - Remote: Push, add atomic to push options by [@​john-cai](https://togithub.com/john-cai) in [go-git/go-git#406 - Remote: add FollowTags option for pushes by [@​john-cai](https://togithub.com/john-cai) in [go-git/go-git#385 - Worktree: use syscall.Timespec.Unix by [@​tklauser](https://togithub.com/tklauser) in [go-git/go-git#437 - Worktree: Checkout, simplified sparse checkout by [@​john-cai](https://togithub.com/john-cai) in [go-git/go-git#410 - Repository: don't crash accessing invalid pathinfo by [@​muesli](https://togithub.com/muesli) in [go-git/go-git#443 - storage: filesystem, switch from os.SEEK_\* to io.Seek\* by [@​abhinav](https://togithub.com/abhinav) in [go-git/go-git#421 - config: add branch description support by [@​ninedraft](https://togithub.com/ninedraft) in [go-git/go-git#409 - revision: fix endless looping in revision parser by [@​michenriksen](https://togithub.com/michenriksen) in [go-git/go-git#475 - pumbling: optimise zlib reader and consolidate sync.Pools by [@​pjbgf](https://togithub.com/pjbgf) in [go-git/go-git#608 - pumbling: parse optimisations by [@​pjbgf](https://togithub.com/pjbgf) in [go-git/go-git#602 - plumbing: object, rename calculation uses too much memory by [@​jfontan](https://togithub.com/jfontan) in [go-git/go-git#503 - plumbing: protocol/pakp and server, include the contents of `GO_GIT_USER_AGENT_EXTRA`. Fixes [#​529](https://togithub.com/go-git/go-git/issues/529) by [@​stewing](https://togithub.com/stewing) in [go-git/go-git#531 - plumbing: protocol/pakp, avoid duplicate encoding when overriding a Capability value. by [@​tylerchr](https://togithub.com/tylerchr) in [go-git/go-git#521 - plumbing: protocol/pakp, update agent by [@​caarlos0](https://togithub.com/caarlos0) in [go-git/go-git#453 - plumbing: protocol/pakp: Actions should have type Action by [@​abhinav](https://togithub.com/abhinav) in [go-git/go-git#420 - plumbing: protocol/pakp: allow unsupported `multi_ack` capability by [@​pjbgf](https://togithub.com/pjbgf) in [go-git/go-git#613 - plumbing: transport/ssh, auto-populate HostKeyAlgorithms. Fixes [#​411](https://togithub.com/go-git/go-git/issues/411) by [@​evanelias](https://togithub.com/evanelias) in [go-git/go-git#548 - pumbling: format/packfile, resolve external reference delta by [@​ga-paul-t](https://togithub.com/ga-paul-t) in [go-git/go-git#392 - plumbing: format/packfile, prevent large objects from being read into memory completely by [@​zeripath](https://togithub.com/zeripath) in [go-git/go-git#330 - plumbing: format/index, support v3 index by [@​john-cai](https://togithub.com/john-cai) in [go-git/go-git#407 - plumbing: format/gitignore, Read .git/info/exclude file too. by [@​enisdenjo](https://togithub.com/enisdenjo) in [go-git/go-git#402 - plumbing: format/gitattributes, Avoid index out of range by [@​To1ne](https://togithub.com/To1ne) in [go-git/go-git#598 - plumbing: format/config, Branch name with hash can be cloned. Fixes [#​309](https://togithub.com/go-git/go-git/issues/309) by [@​dowy](https://togithub.com/dowy) in [go-git/go-git#354 - go.mod: update github.com/xanzy/ssh-agent to v0.3.1 by [@​tklauser](https://togithub.com/tklauser) in [go-git/go-git#403 - go.mod: update dependencies to remove supply chain CVEs by [@​pjbgf](https://togithub.com/pjbgf) in [go-git/go-git#620 - examples: added "tag find if head is tagged" by [@​snebel29](https://togithub.com/snebel29) in [go-git/go-git#374 - examples: remote fix typo by [@​nep-0](https://togithub.com/nep-0) in [go-git/go-git#408 **Full Changelog**: go-git/go-git@v5.4.2...v5.5.0 </details> <details> <summary>google/go-cmp</summary> ### [`v0.5.9`](https://togithub.com/google/go-cmp/releases/tag/v0.5.9) [Compare Source](https://togithub.com/google/go-cmp/compare/v0.5.8...v0.5.9) Reporter changes: - ([#​299](https://togithub.com/google/go-cmp/issues/299)) Adjust heuristic for line-based versus byte-based diffing - ([#​306](https://togithub.com/google/go-cmp/issues/306)) Use `value.TypeString` in `PathStep.String` Code cleanup changes: - ([#​297](https://togithub.com/google/go-cmp/issues/297)) Use `reflect.Value.IsZero` - ([#​304](https://togithub.com/google/go-cmp/issues/304)) Format with Go 1.19 formatter - ([#​300](https://togithub.com/google/go-cmp/issues/300) )Fix typo in Result documentation - ([#​302](https://togithub.com/google/go-cmp/issues/302)) Pre-declare global type variables - ([#​309](https://togithub.com/google/go-cmp/issues/309)) Run tests on Go 1.19 </details> <details> <summary>googleapis/google-api-go-client</summary> ### [`v0.103.0`](https://togithub.com/googleapis/google-api-go-client/releases/tag/v0.103.0) [Compare Source](https://togithub.com/googleapis/google-api-go-client/compare/v0.102.0...v0.103.0) ##### Features - **all:** Auto-regenerate discovery clients ([#​1737](https://togithub.com/googleapis/google-api-go-client/issues/1737)) ([de99200](https://togithub.com/googleapis/google-api-go-client/commit/de9920088db16562740c31183eca6651f669e582)) - **all:** Auto-regenerate discovery clients ([#​1739](https://togithub.com/googleapis/google-api-go-client/issues/1739)) ([bbd4259](https://togithub.com/googleapis/google-api-go-client/commit/bbd42597f4710f527f83fd900cb7f9e6706bc195)) - **all:** Auto-regenerate discovery clients ([#​1743](https://togithub.com/googleapis/google-api-go-client/issues/1743)) ([4248dc3](https://togithub.com/googleapis/google-api-go-client/commit/4248dc3db6b32d00720293980fb8e845b684fbd8)) - **googleapi:** Inject gax apierror.APIError into googleapi.Error ([#​1730](https://togithub.com/googleapis/google-api-go-client/issues/1730)) ([ee25e29](https://togithub.com/googleapis/google-api-go-client/commit/ee25e29fd586cde25a006504d0059194a90f19ac)) - Rm hard dep on x/sys ([#​1742](https://togithub.com/googleapis/google-api-go-client/issues/1742)) ([9695aa1](https://togithub.com/googleapis/google-api-go-client/commit/9695aa13a084c1ad9857db4a6c12d57e13fc00dc)) ### [`v0.102.0`](https://togithub.com/googleapis/google-api-go-client/releases/tag/v0.102.0) [Compare Source](https://togithub.com/googleapis/google-api-go-client/compare/v0.101.0...v0.102.0) ##### Features - **all:** Auto-regenerate discovery clients ([#​1725](https://togithub.com/googleapis/google-api-go-client/issues/1725)) ([06360d8](https://togithub.com/googleapis/google-api-go-client/commit/06360d8f37b88e064a8a60788077f376b597d942)) - **all:** Auto-regenerate discovery clients ([#​1727](https://togithub.com/googleapis/google-api-go-client/issues/1727)) ([1e1eab9](https://togithub.com/googleapis/google-api-go-client/commit/1e1eab98aac0e967a6c52b65fe9eb5a4d6d8a946)) - **all:** Auto-regenerate discovery clients ([#​1734](https://togithub.com/googleapis/google-api-go-client/issues/1734)) ([ce57a67](https://togithub.com/googleapis/google-api-go-client/commit/ce57a67eddb98f3ccd21c1c01dfcb18df0d77009)) - Rely on new compute metadata module directly ([#​1736](https://togithub.com/googleapis/google-api-go-client/issues/1736)) ([0528475](https://togithub.com/googleapis/google-api-go-client/commit/0528475d51393bb6e3244816d9c6ea8c16275677)) ### [`v0.101.0`](https://togithub.com/googleapis/google-api-go-client/releases/tag/v0.101.0) [Compare Source](https://togithub.com/googleapis/google-api-go-client/compare/v0.100.0...v0.101.0) ##### Features - **all:** Auto-regenerate discovery clients ([#​1718](https://togithub.com/googleapis/google-api-go-client/issues/1718)) ([453b81a](https://togithub.com/googleapis/google-api-go-client/commit/453b81ac138e6572e9d6a3373c033c5abbcefbcc)) - **all:** Auto-regenerate discovery clients ([#​1720](https://togithub.com/googleapis/google-api-go-client/issues/1720)) ([9140608](https://togithub.com/googleapis/google-api-go-client/commit/91406081538e06ab580f59d6fba001dc34f8574a)) - **all:** Auto-regenerate discovery clients ([#​1723](https://togithub.com/googleapis/google-api-go-client/issues/1723)) ([f4788b3](https://togithub.com/googleapis/google-api-go-client/commit/f4788b325bd76337216a54e02e49cec4e3ee6987)) ### [`v0.100.0`](https://togithub.com/googleapis/google-api-go-client/releases/tag/v0.100.0) [Compare Source](https://togithub.com/googleapis/google-api-go-client/compare/v0.99.0...v0.100.0) ##### Features - **all:** Auto-regenerate discovery clients ([#​1712](https://togithub.com/googleapis/google-api-go-client/issues/1712)) ([f9e15f2](https://togithub.com/googleapis/google-api-go-client/commit/f9e15f2159928974af1a2ec539e20f17f94aab4d)) - **all:** Auto-regenerate discovery clients ([#​1717](https://togithub.com/googleapis/google-api-go-client/issues/1717)) ([f990a2a](https://togithub.com/googleapis/google-api-go-client/commit/f990a2af6cd6210c8764bbe273a575886ea97038)) - **internal/gensupport:** Remove DetermineContentType, use gax-go copy ([#​1716](https://togithub.com/googleapis/google-api-go-client/issues/1716)) ([37f90e9](https://togithub.com/googleapis/google-api-go-client/commit/37f90e974e83f06962ac923c502cd1b405c7f0fb)) ##### Bug Fixes - **idtoken:** Allow missing age in http response header ([#​1715](https://togithub.com/googleapis/google-api-go-client/issues/1715)) ([b235b1f](https://togithub.com/googleapis/google-api-go-client/commit/b235b1f8c718be6b8f361074d371768617a3da3a)) ### [`v0.99.0`](https://togithub.com/googleapis/google-api-go-client/releases/tag/v0.99.0) [Compare Source](https://togithub.com/googleapis/google-api-go-client/compare/v0.98.0...v0.99.0) ##### Features - **all:** Auto-regenerate discovery clients ([#​1701](https://togithub.com/googleapis/google-api-go-client/issues/1701)) ([6b81c83](https://togithub.com/googleapis/google-api-go-client/commit/6b81c8355addd65f718bb9195e1c2356117e1a1b)) ### [`v0.98.0`](https://togithub.com/googleapis/google-api-go-client/releases/tag/v0.98.0) [Compare Source](https://togithub.com/googleapis/google-api-go-client/compare/v0.97.0...v0.98.0) ##### Features - **all:** Auto-regenerate discovery clients ([#​1696](https://togithub.com/googleapis/google-api-go-client/issues/1696)) ([aa775b4](https://togithub.com/googleapis/google-api-go-client/commit/aa775b41d2e419002d4e7e7a390745dd2d07110a)) - **all:** Auto-regenerate discovery clients ([#​1699](https://togithub.com/googleapis/google-api-go-client/issues/1699)) ([25b7450](https://togithub.com/googleapis/google-api-go-client/commit/25b7450d0d9efc46d4095d827f597ac85bb8b5b4)) ### [`v0.97.0`](https://togithub.com/googleapis/google-api-go-client/releases/tag/v0.97.0) [Compare Source](https://togithub.com/googleapis/google-api-go-client/compare/v0.96.0...v0.97.0) ##### Features - **all:** Auto-regenerate discovery clients ([#​1693](https://togithub.com/googleapis/google-api-go-client/issues/1693)) ([a87400b](https://togithub.com/googleapis/google-api-go-client/commit/a87400be9341608f73e9ae1b5dbbecc7adfbf609)) - **all:** Auto-regenerate discovery clients ([#​1695](https://togithub.com/googleapis/google-api-go-client/issues/1695)) ([b8f2556](https://togithub.com/googleapis/google-api-go-client/commit/b8f25561a76841c7549a358925eb7bfc2236465e)) - **internal/gensupport:** Wrap retry failures with context and prev error ([#​1684](https://togithub.com/googleapis/google-api-go-client/issues/1684)) ([f427ee3](https://togithub.com/googleapis/google-api-go-client/commit/f427ee3edede981524c2ffb57fd2d8981f8cf8b4)), refs [#​1685](https://togithub.com/googleapis/google-api-go-client/issues/1685) ##### Bug Fixes - Build script bash error ([#​1697](https://togithub.com/googleapis/google-api-go-client/issues/1697)) ([6b0515b](https://togithub.com/googleapis/google-api-go-client/commit/6b0515bf05d8c62007748827eed486c607af483b)) - **gensupport:** Allow initial request for resumable uploads to retry w/ non-nil getBody ([#​1690](https://togithub.com/googleapis/google-api-go-client/issues/1690)) ([2c3e863](https://togithub.com/googleapis/google-api-go-client/commit/2c3e8638afc6702dcba732a1aa07ccb33eb9304b)) ### [`v0.96.0`](https://togithub.com/googleapis/google-api-go-client/releases/tag/v0.96.0) [Compare Source](https://togithub.com/googleapis/google-api-go-client/compare/v0.95.0...v0.96.0) ##### Features - **all:** Auto-regenerate discovery clients ([#​1686](https://togithub.com/googleapis/google-api-go-client/issues/1686)) ([ce5ed41](https://togithub.com/googleapis/google-api-go-client/commit/ce5ed411756019b79c77e580670fccc8c08cccca)) - **all:** Auto-regenerate discovery clients ([#​1688](https://togithub.com/googleapis/google-api-go-client/issues/1688)) ([bc29a6b](https://togithub.com/googleapis/google-api-go-client/commit/bc29a6b8a0489e88796d5a00d4c06769793ace0d)) - **all:** Auto-regenerate discovery clients ([#​1689](https://togithub.com/googleapis/google-api-go-client/issues/1689)) ([e801e10](https://togithub.com/googleapis/google-api-go-client/commit/e801e1051020e6721f2217f5aa3a4064399115e1)) ##### Bug Fixes - Upgrade version of golang.org/x/net ([#​1692](https://togithub.com/googleapis/google-api-go-client/issues/1692)) ([0f7c1ed](https://togithub.com/googleapis/google-api-go-client/commit/0f7c1ed65ca2c6212f21e7fce20aa5ab9952bdbc)), refs [#​1691](https://togithub.com/googleapis/google-api-go-client/issues/1691) ### [`v0.95.0`](https://togithub.com/googleapis/google-api-go-client/releases/tag/v0.95.0) [Compare Source](https://togithub.com/googleapis/google-api-go-client/compare/v0.94.0...v0.95.0) ##### Features - **all:** Auto-regenerate discovery clients ([#​1677](https://togithub.com/googleapis/google-api-go-client/issues/1677)) ([8757dbf](https://togithub.com/googleapis/google-api-go-client/commit/8757dbf5811cc9f4092a8259d859c35ad3cc6442)) - **all:** Auto-regenerate discovery clients ([#​1680](https://togithub.com/googleapis/google-api-go-client/issues/1680)) ([8c72fb3](https://togithub.com/googleapis/google-api-go-client/commit/8c72fb345fb6e377fa984053ca9c00aa0c3a0985)) - **option:** Officially deprecate ImpersonateCredentials ([#​1683](https://togithub.com/googleapis/google-api-go-client/issues/1683)) ([9a84077](https://togithub.com/googleapis/google-api-go-client/commit/9a84077014f9a37335d29132e373b92adf49f904)) ### [`v0.94.0`](https://togithub.com/googleapis/google-api-go-client/releases/tag/v0.94.0) [Compare Source](https://togithub.com/googleapis/google-api-go-client/compare/v0.93.0...v0.94.0) ##### Features - **all:** auto-regenerate discovery clients, refs [#​1676](https://togithub.com/googleapis/google-api-go-client/issues/1676) [#​1673](https://togithub.com/googleapis/google-api-go-client/issues/1673) [#​1672](https://togithub.com/googleapis/google-api-go-client/issues/1672) [#​1671](https://togithub.com/googleapis/google-api-go-client/issues/1671) [#​1667](https://togithub.com/googleapis/google-api-go-client/issues/1667) ##### Bug Fixes - **storage:** \*int64 instead of int64 for Age cond ([#​1598](https://togithub.com/googleapis/google-api-go-client/issues/1598)) ([9ea025d](https://togithub.com/googleapis/google-api-go-client/commit/9ea025dcfe9b67a95e08f4ec94ed4fb6a9767b8c)) ##### Documentation - **option:** clarify behavior of WithScopes ([#​1670](https://togithub.com/googleapis/google-api-go-client/issues/1670)) ([07ceb9d](https://togithub.com/googleapis/google-api-go-client/commit/07ceb9d607c85ffaa5bea97be66cf9d426ec55bb)), refs [#​1644](https://togithub.com/googleapis/google-api-go-client/issues/1644) ### [`v0.93.0`](https://togithub.com/googleapis/google-api-go-client/releases/tag/v0.93.0) [Compare Source](https://togithub.com/googleapis/google-api-go-client/compare/v0.92.0...v0.93.0) ##### Features - **all:** auto-regenerate discovery clients, refs [#​1664](https://togithub.com/googleapis/google-api-go-client/issues/1664) [#​1662](https://togithub.com/googleapis/google-api-go-client/issues/1662) [#​1661](https://togithub.com/googleapis/google-api-go-client/issues/1661) [#​1652](https://togithub.com/googleapis/google-api-go-client/issues/1652) - **google-api-go-generator:** Change field PaymentState to pointer ([#​1663](https://togithub.com/googleapis/google-api-go-client/issues/1663)) ([d6ee425](https://togithub.com/googleapis/google-api-go-client/commit/d6ee425a65668ee28ff97c6fb70f3497865d6572)), refs [#​727](https://togithub.com/googleapis/google-api-go-client/issues/727) ### [`v0.92.0`](https://togithub.com/googleapis/google-api-go-client/releases/tag/v0.92.0) [Compare Source](https://togithub.com/googleapis/google-api-go-client/compare/v0.91.0...v0.92.0) ##### Features - **all:** auto-regenerate discovery clients, refs [#​1649](https://togithub.com/googleapis/google-api-go-client/issues/1649) [#​1646](https://togithub.com/googleapis/google-api-go-client/issues/1646) [#​1645](https://togithub.com/googleapis/google-api-go-client/issues/1645) [#​1643](https://togithub.com/googleapis/google-api-go-client/issues/1643) [#​1641](https://togithub.com/googleapis/google-api-go-client/issues/1641) </details> --- ### Configuration 📅 **Schedule**: Branch creation - "before 6am on monday" in timezone Australia/Sydney, Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://togithub.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/google/osv.dev). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNC45LjEiLCJ1cGRhdGVkSW5WZXIiOiIzNC40OC4xIn0=-->
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github.com/go-git/go-git/v5](https://github.com/go-git/go-git) | require | minor | `v5.4.2` -> `v5.8.1` | --- ### ⚠ Dependency Lookup Warnings ⚠ Warnings were logged while processing this repo. Please check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>go-git/go-git (github.com/go-git/go-git/v5)</summary> ### [`v5.8.1`](https://github.com/go-git/go-git/releases/tag/v5.8.1) [Compare Source](go-git/go-git@v5.8.0...v5.8.1) #### What's Changed - \*: Bump dependencies by [@​pjbgf](https://github.com/pjbgf) in go-git/go-git#815 **Full Changelog**: go-git/go-git@v5.8.0...v5.8.1 ### [`v5.8.0`](https://github.com/go-git/go-git/releases/tag/v5.8.0) [Compare Source](go-git/go-git@v5.7.0...v5.8.0) #### What's Changed - git: Fix fetching after shallow clone. Fixes [#​305](go-git/go-git#305) by [@​AriehSchneier](https://github.com/AriehSchneier) in go-git/go-git#778 - git: enable fetch with unqualified references by [@​AriehSchneier](https://github.com/AriehSchneier) in go-git/go-git#762 - git: don't add to want if exists, shallow and depth 1 by [@​AriehSchneier](https://github.com/AriehSchneier) in go-git/go-git#763 - git: Clone HEAD should not force master. Fixes [#​363](go-git/go-git#363) by [@​AriehSchneier](https://github.com/AriehSchneier) in go-git/go-git#758 - git: fix the issue with submodules having the SCP style URL fail due to the wrong URL parsing by [@​matejrisek](https://github.com/matejrisek) in go-git/go-git#756 - git: add a clone option to allow for shallow cloning of submodules by [@​matejrisek](https://github.com/matejrisek) in go-git/go-git#765 - worktree: minor speedup for `doAddDirectory` by [@​ThinkChaos](https://github.com/ThinkChaos) in go-git/go-git#702 - \_examples: Remove wrong comment by [@​pascal-hofmann](https://github.com/pascal-hofmann) in go-git/go-git#357 - \*: Handle paths starting with tilde by [@​ricci2511](https://github.com/ricci2511) in go-git/go-git#808 - \*: Handle paths starting with ~Username by [@​AriehSchneier](https://github.com/AriehSchneier) in go-git/go-git#809 - storage: filesystem/dotgit, add support for tmp_objdir prefix by [@​L11R](https://github.com/L11R) in go-git/go-git#812 - plumbing: gitignore, replace user dir in path by [@​Jleagle](https://github.com/Jleagle) in go-git/go-git#772 - plumbing: gitignore, fix incorrect parsing. Fixes [#​500](go-git/go-git#500) by [@​AriehSchneier](https://github.com/AriehSchneier) in go-git/go-git#781 - plumbing: http, Fix empty repos on Git v2.41+ by [@​pjbgf](https://github.com/pjbgf) in go-git/go-git#802 - plumbing: packp, A request is not empty if it contains shallows. Fixes [#​328](go-git/go-git#328) by [@​AriehSchneier](https://github.com/AriehSchneier) in go-git/go-git#792 - plumbing: blame, Complete rewrite. Fixes [#​603](go-git/go-git#603) by [@​AriehSchneier](https://github.com/AriehSchneier) in go-git/go-git#789 - plumbing: gitignore, Allow gitconfig to contain a gitignore relative to any user home. Fixes [#​578](go-git/go-git#578) by [@​AriehSchneier](https://github.com/AriehSchneier) in go-git/go-git#785 #### New Contributors - [@​Jleagle](https://github.com/Jleagle) made their first contribution in go-git/go-git#772 - [@​pascal-hofmann](https://github.com/pascal-hofmann) made their first contribution in go-git/go-git#357 - [@​ricci2511](https://github.com/ricci2511) made their first contribution in go-git/go-git#808 - [@​L11R](https://github.com/L11R) made their first contribution in go-git/go-git#812 **Full Changelog**: go-git/go-git@v5.7.0...v5.7.1 ### [`v5.7.0`](https://github.com/go-git/go-git/releases/tag/v5.7.0) [Compare Source](go-git/go-git@v5.6.1...v5.7.0) #### What's Changed - \*: Add support for initializing SHA256 repositories by [@​pjbgf](https://github.com/pjbgf) in go-git/go-git#707 - git: add mirror clone option by [@​aymanbagabas](https://github.com/aymanbagabas) in go-git/go-git#735 - git: Add support to ls-remote with peeled references. Fixes [#​749](go-git/go-git#749) by [@​pjbgf](https://github.com/pjbgf) in go-git/go-git#750 - git: fix cloning with branch name by [@​AriehSchneier](https://github.com/AriehSchneier) in go-git/go-git#755 - git: Worktree, add check to see if file already checked in. Fixes [#​718](go-git/go-git#718) by [@​cbbm142](https://github.com/cbbm142) in go-git/go-git#719 - git: Worktree, git grep bare repositories by [@​aymanbagabas](https://github.com/aymanbagabas) in go-git/go-git#728 - git: Add Depth to SubmoduleUpdateOptions by [@​matejrisek](https://github.com/matejrisek) in go-git/go-git#754 - git: Testing, Fix tests not cleaning temp folders by [@​AriehSchneier](https://github.com/AriehSchneier) in go-git/go-git#769 - git: remote, add support for a configurable timeout. by [@​andrewpollock](https://github.com/andrewpollock) in go-git/go-git#753 - git: Allow Initial Branch to be configurable by [@​techknowlogick](https://github.com/techknowlogick) in go-git/go-git#764 - storage: filesystem/dotgit, Improve load packed-refs by [@​fcharlie](https://github.com/fcharlie) in go-git/go-git#743 - storage: filesystem, Populate index before use. Fixes [#​148](go-git/go-git#148) by [@​AriehSchneier](https://github.com/AriehSchneier) in go-git/go-git#722 - plumbing: resolve non-external delta references by [@​ZauberNerd](https://github.com/ZauberNerd) in go-git/go-git#485 - plumbing/transport: fix regression in scp-like match by [@​jotadrilo](https://github.com/jotadrilo) in go-git/go-git#715 - plumbing/transport: Add support for custom proxy settings by [@​aryan9600](https://github.com/aryan9600) in go-git/go-git#744 - \*: small fixes across the codebase by [@​pjbgf](https://github.com/pjbgf) in go-git/go-git#770 - \*: bump github.com/cloudflare/circl from 1.1.0 to 1.3.3 by [@​dependabot](https://github.com/dependabot) in go-git/go-git#776 - \*: bump dependencies by [@​pjbgf](https://github.com/pjbgf) in go-git/go-git#748 - \*: bump Go version to 1.18 on go.mod by [@​pjbgf](https://github.com/pjbgf) in go-git/go-git#774 - \*: add Codeql workflow and bump dependencies by [@​pjbgf](https://github.com/pjbgf) in go-git/go-git#775 - ci: fix upstream git build for master branch by [@​pjbgf](https://github.com/pjbgf) in go-git/go-git#739 #### New Contributors - [@​ZauberNerd](https://github.com/ZauberNerd) made their first contribution in go-git/go-git#485 - [@​jotadrilo](https://github.com/jotadrilo) made their first contribution in go-git/go-git#715 - [@​fcharlie](https://github.com/fcharlie) made their first contribution in go-git/go-git#743 - [@​AriehSchneier](https://github.com/AriehSchneier) made their first contribution in go-git/go-git#755 - [@​cbbm142](https://github.com/cbbm142) made their first contribution in go-git/go-git#719 - [@​aryan9600](https://github.com/aryan9600) made their first contribution in go-git/go-git#744 - [@​matejrisek](https://github.com/matejrisek) made their first contribution in go-git/go-git#754 - [@​andrewpollock](https://github.com/andrewpollock) made their first contribution in go-git/go-git#753 - [@​techknowlogick](https://github.com/techknowlogick) made their first contribution in go-git/go-git#764 **Full Changelog**: go-git/go-git@v5.6.1...v5.7.0 ### [`v5.6.1`](https://github.com/go-git/go-git/releases/tag/v5.6.1) [Compare Source](go-git/go-git@v5.6.0...v5.6.1) #### What's Changed - plumbing/transport: don't use the `firstErrLine` when it is empty by [@​ThinkChaos](https://github.com/ThinkChaos) in go-git/go-git#682 - plumbing/transport: ssh, unable to pass a custom HostKeyCallback func by [@​aymanbagabas](https://github.com/aymanbagabas) in go-git/go-git#655 - storage/filesystem: dotgit: fix a filesystem race in Refs/walkReferencesTree by [@​MichaelMure](https://github.com/MichaelMure) in go-git/go-git#659 - \*: bump golang.org/x/net from 0.2.0 to 0.7.0 by [@​dependabot](https://github.com/dependabot) in go-git/go-git#684 - \*: bump dependencies by [@​pjbgf](https://github.com/pjbgf) in go-git/go-git#697 - \*: fix panic for empty revisions by [@​pjbgf](https://github.com/pjbgf) in go-git/go-git#696 - ci: bump GitHub actions, enable go test race detection and stop using developer's GPG keys during test execution by [@​pjbgf](https://github.com/pjbgf) in go-git/go-git#701 **Full Changelog**: go-git/go-git@v5.6.0...v5.6.1 ### [`v5.6.0`](https://github.com/go-git/go-git/releases/tag/v5.6.0) [Compare Source](go-git/go-git@v5.5.2...v5.6.0) #### What's Changed - Worktree, check for empty parent dirs during Reset (Fixes [#​670](go-git/go-git#670)) by [@​mbohy](https://github.com/mbohy) in go-git/go-git#671 - \*: remove need to build with CGO by [@​pjbgf](https://github.com/pjbgf) in go-git/go-git#688 - plumbing: support SSH/X509 signed tags by [@​hiddeco](https://github.com/hiddeco) in go-git/go-git#690 **Full Changelog**: go-git/go-git@v5.5.2...v5.6.0 ### [`v5.5.2`](https://github.com/go-git/go-git/releases/tag/v5.5.2) [Compare Source](go-git/go-git@v5.5.1...v5.5.2) #### What's Changed - \*: update go-billy v5.4.0, removes data races. Fixes [#​629](go-git/go-git#629) by [@​mcuadros](https://github.com/mcuadros) in go-git/go-git#653 - Worktree: Add, fix add removed files. Fixes [#​223](go-git/go-git#223) by [@​tfujiwar](https://github.com/tfujiwar) in go-git/go-git#652 **Full Changelog**: go-git/go-git@v5.5.1...v5.5.2 ### [`v5.5.1`](https://github.com/go-git/go-git/releases/tag/v5.5.1) [Compare Source](go-git/go-git@v5.5.0...v5.5.1) #### What's Changed - \*: fix error when building with `CGO_ENABLED=0` by [@​pjbgf](https://github.com/pjbgf) in go-git/go-git#625 - plumbing: transport/ssh: fix panic on Windows 10 with paegent as ssh-agent by [@​doxsch](https://github.com/doxsch) in go-git/go-git#617 - CommitOptions: AllowEmptyCommits, return an error instead of creating empty commits by [@​pjbgf](https://github.com/pjbgf) in go-git/go-git#623 **Full Changelog**: go-git/go-git@v5.5.0...v5.5.1 ### [`v5.5.0`](https://github.com/go-git/go-git/releases/tag/v5.5.0) [Compare Source](go-git/go-git@v5.4.2...v5.5.0) #### What's Changed - \*: add collision resistent SHA1 implementation by [@​pjbgf](https://github.com/pjbgf) in go-git/go-git#618 - \*: replace go-homedir with os.UserHomeDir by [@​mvdan](https://github.com/mvdan) in go-git/go-git#535 - Remote: add RemoteURL to {Fetch,Pull,Push}Options by [@​noerw](https://github.com/noerw) in go-git/go-git#375 - Remote: Push, add support to push commits per hashes by [@​tjamet](https://github.com/tjamet) in go-git/go-git#325 - Remote: Push, add ForceWithLease Push Option by [@​john-cai](https://github.com/john-cai) in go-git/go-git#404 - Remote: PushOptions add push-options by [@​S-Bohn](https://github.com/S-Bohn) in go-git/go-git#399 - Remote: Push, add atomic to push options by [@​john-cai](https://github.com/john-cai) in go-git/go-git#406 - Remote: add FollowTags option for pushes by [@​john-cai](https://github.com/john-cai) in go-git/go-git#385 - Worktree: use syscall.Timespec.Unix by [@​tklauser](https://github.com/tklauser) in go-git/go-git#437 - Worktree: Checkout, simplified sparse checkout by [@​john-cai](https://github.com/john-cai) in go-git/go-git#410 - Repository: don't crash accessing invalid pathinfo by [@​muesli](https://github.com/muesli) in go-git/go-git#443 - storage: filesystem, switch from os.SEEK_\* to io.Seek\* by [@​abhinav](https://github.com/abhinav) in go-git/go-git#421 - config: add branch description support by [@​ninedraft](https://github.com/ninedraft) in go-git/go-git#409 - revision: fix endless looping in revision parser by [@​michenriksen](https://github.com/michenriksen) in go-git/go-git#475 - pumbling: optimise zlib reader and consolidate sync.Pools by [@​pjbgf](https://github.com/pjbgf) in go-git/go-git#608 - pumbling: parse optimisations by [@​pjbgf](https://github.com/pjbgf) in go-git/go-git#602 - plumbing: object, rename calculation uses too much memory by [@​jfontan](https://github.com/jfontan) in go-git/go-git#503 - plumbing: protocol/pakp and server, include the contents of `GO_GIT_USER_AGENT_EXTRA`. Fixes [#​529](go-git/go-git#529) by [@​stewing](https://github.com/stewing) in go-git/go-git#531 - plumbing: protocol/pakp, avoid duplicate encoding when overriding a Capability value. by [@​tylerchr](https://github.com/tylerchr) in go-git/go-git#521 - plumbing: protocol/pakp, update agent by [@​caarlos0](https://github.com/caarlos0) in go-git/go-git#453 - plumbing: protocol/pakp: Actions should have type Action by [@​abhinav](https://github.com/abhinav) in go-git/go-git#420 - plumbing: protocol/pakp: allow unsupported `multi_ack` capability by [@​pjbgf](https://github.com/pjbgf) in go-git/go-git#613 - plumbing: transport/ssh, auto-populate HostKeyAlgorithms. Fixes [#​411](go-git/go-git#411) by [@​evanelias](https://github.com/evanelias) in go-git/go-git#548 - pumbling: format/packfile, resolve external reference delta by [@​ga-paul-t](https://github.com/ga-paul-t) in go-git/go-git#392 - plumbing: format/packfile, prevent large objects from being read into memory completely by [@​zeripath](https://github.com/zeripath) in go-git/go-git#330 - plumbing: format/index, support v3 index by [@​john-cai](https://github.com/john-cai) in go-git/go-git#407 - plumbing: format/gitignore, Read .git/info/exclude file too. by [@​enisdenjo](https://github.com/enisdenjo) in go-git/go-git#402 - plumbing: format/gitattributes, Avoid index out of range by [@​To1ne](https://github.com/To1ne) in go-git/go-git#598 - plumbing: format/config, Branch name with hash can be cloned. Fixes [#​309](go-git/go-git#309) by [@​dowy](https://github.com/dowy) in go-git/go-git#354 - go.mod: update github.com/xanzy/ssh-agent to v0.3.1 by [@​tklauser](https://github.com/tklauser) in go-git/go-git#403 - go.mod: update dependencies to remove supply chain CVEs by [@​pjbgf](https://github.com/pjbgf) in go-git/go-git#620 - examples: added "tag find if head is tagged" by [@​snebel29](https://github.com/snebel29) in go-git/go-git#374 - examples: remote fix typo by [@​nep-0](https://github.com/nep-0) in go-git/go-git#408 **Full Changelog**: go-git/go-git@v5.4.2...v5.5.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNi43OS4xIiwidXBkYXRlZEluVmVyIjoiMzYuNzkuMSIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==--> Reviewed-on: https://gitea.com/gitea/tea/pulls/578 Co-authored-by: Renovate Bot <renovate-bot@gitea.com> Co-committed-by: Renovate Bot <renovate-bot@gitea.com>
This PR adds code to prevent large objects from being read into memory
from packfiles or the filesystem through the use of a new
Option
, theLargeObjectThreshold
.Objects greater than the provided (optional)
LargeObjectThreshold
are now no longer directly stored in the cacheor read completely into memory.
This PR differs and improves the previous broken #303 by fixing several
bugs in the reader and transparently wrapping ReaderAt as a Reader.
Signed-off-by: Andrew Thornton art27@cantab.net
(UPDATED: Make LOT an Option on the ObjectStorage - defaulted to off.)