clone large project , fatal error:out of memory

[zhou27]

os : windows

code:

git file:

`.runtime: out of memory: cannot allocate 1073741824-byte block (1434288128 in use)
fatal error: out of memory

runtime stack:
runtime.throw(0x115def3, 0xd)
C:/soft/go/src/runtime/panic.go:1116 +0x64
runtime.largeAlloc(0x3ffffe00, 0x23d0101, 0x1)
C:/soft/go/src/runtime/malloc.go:1179 +0x15d
runtime.mallocgc.func1()
C:/soft/go/src/runtime/malloc.go:1071 +0x39
runtime.systemstack(0x0)
C:/soft/go/src/runtime/asm_386.s:391 +0x53
runtime.mstart()
C:/soft/go/src/runtime/proc.go:1116

goroutine 56 [running]:
runtime.systemstack_switch()
C:/soft/go/src/runtime/asm_386.s:352 fp=0x1284dc64 sp=0x1284dc60 pc=0xb0f3a0
runtime.mallocgc(0x3ffffe00, 0x10a6ca0, 0xf55a9b01, 0xc66cda)
C:/soft/go/src/runtime/malloc.go:1070 +0x698 fp=0x1284dcb4 sp=0x1284dc64 pc=0xabbad8
runtime.makeslice(0x10a6ca0, 0x3ffffe00, 0x3ffffe00, 0x7e00)
C:/soft/go/src/runtime/slice.go:98 +0x4f fp=0x1284dcc8 sp=0x1284dcb4 pc=0xaf6eff
bytes.makeSlice(0x3ffffe00, 0x0, 0x0, 0x0)
C:/soft/go/src/bytes/buffer.go:229 +0x64 fp=0x1284dce4 sp=0x1284dcc8 pc=0xb94e14
bytes.(*Buffer).grow(0x12b33f60, 0x200, 0x7e00)
C:/soft/go/src/bytes/buffer.go:142 +0x10c fp=0x1284dd0c sp=0x1284dce4 pc=0xb948bc
bytes.(*Buffer).ReadFrom(0x12b33f60, 0x33cd0008, 0x12890000, 0x2400020, 0x12b33f60, 0x244a601, 0x0)
C:/soft/go/src/bytes/buffer.go:202 +0x45 fp=0x1284dd48 sp=0x1284dd0c pc=0xb94c45
io.copyBuffer(0x126c390, 0x12b33f60, 0x33cd0008, 0x12890000, 0x12e94000, 0x100000, 0x100000, 0x10c2a00, 0x10ea9e0, 0x0, ...)
C:/soft/go/src/io/io.go:395 +0x298 fp=0x1284dd88 sp=0x1284dd48 pc=0xb4b4f8
io.CopyBuffer(0x126c390, 0x12b33f60, 0x33cd0008, 0x12890000, 0x12e94000, 0x100000, 0x100000, 0x0, 0x3fffff, 0x0, ...)
C:/soft/go/src/io/io.go:382 +0x5d fp=0x1284ddb8 sp=0x1284dd88 pc=0xb4b20d
github.com/go-git/go-git/v5/plumbing/format/packfile.(*Scanner).copyObject(0x12b80a40, 0x126c390, 0x12b33f60, 0x0, 0x0, 0x0, 0x0)
C:/Users/chase/go/pkg/mod/github.com/go-git/go-git/v5@v5.2.0/plumbing/format/packfile/scanner.go:335 +0x2a0 fp=0x1284de28 sp=0x1284ddb8 pc=0xc74c80
github.com/go-git/go-git/v5/plumbing/format/packfile.(*Scanner).NextObject(0x12b80a40, 0x126c390, 0x12b33f60, 0x0, 0x13faf644, 0x22e9730e, 0x295941d, 0xc14db416)
C:/Users/chase/go/pkg/mod/github.com/go-git/go-git/v5@v5.2.0/plumbing/format/packfile/scanner.go:314 +0x4c fp=0x1284de5c sp=0x1284de28 pc=0xc7493c
github.com/go-git/go-git/v5/plumbing/format/packfile.(*Parser).indexObjects(0x12bee140, 0x0, 0x0)
C:/Users/chase/go/pkg/mod/github.com/go-git/go-git/v5@v5.2.0/plumbing/format/packfile/parser.go:223 +0x1e4 fp=0x1284df24 sp=0x1284de5c pc=0xc712d4
github.com/go-git/go-git/v5/plumbing/format/packfile.(*Parser).Parse(0x12bee140, 0x0, 0x0, 0x0, 0x0, 0x0, 0x13017e00, 0x0)
C:/Users/chase/go/pkg/mod/github.com/go-git/go-git/v5@v5.2.0/plumbing/format/packfile/parser.go:139 +0x4c fp=0x1284df7c sp=0x1284df24 pc=0xc70dcc
github.com/go-git/go-git/v5/storage/filesystem/dotgit.(*PackWriter).buildIndex(0x12bced00)
C:/Users/chase/go/pkg/mod/github.com/go-git/go-git/v5@v5.2.0/storage/filesystem/dotgit/writers.go:67 +0x181 fp=0x1284dfe8 sp=0x1284df7c pc=0xc7f6a1
runtime.goexit()
C:/soft/go/src/runtime/asm_386.s:1333 +0x1 fp=0x1284dfec sp=0x1284dfe8 pc=0xb109c1
created by github.com/go-git/go-git/v5/storage/filesystem/dotgit.newPackWrite
C:/Users/chase/go/pkg/mod/github.com/go-git/go-git/v5@v5.2.0/storage/filesystem/dotgit/writers.go:53 +0x272

`

[tooptoop4]

did u find a fix? i am having same OOM issue

[pjbgf]

@tooptoop4 what version of go-git are you using? There were two recent merges with memory optimisation changes.

[tooptoop4]

this fork: https://github.com/argoproj/argo-workflows/blob/v3.4.3/go.mod#L226

[coding-cgs]

did u find a fix?

[coding-cgs]

@pjbgf this problem still occurs with v5.6.1 , clone the 1G project on a VM with 2 GB memory

[pjbgf]

@tooptoop4 if your issue is specific to that fork, please contact the owner of that fork.

@coding-cgs I believe that using in-memory fs uses almost half of memory when compared to osfs. If you are keen on proposing a PR for a fix, feel free to tag me for a review.

[coding-cgs]

@pjbgf Thank you for your reply
I use plainClone method, is it the same as zhou27 used?
code:
repo, err := git.PlainClone("./testGit", false, &git.CloneOptions{})

[pjbgf]

The PR #799 aims to decrease the memory consumption of large repositories.

Can you please give it a try and see whether that improves your use cases?

go get github.com/go-git/go-git/v5@memory-churn

[github-actions]

To help us keep things tidy and focus on the active tasks, we've introduced a stale bot to spot issues/PRs that haven't had any activity in a while.

This particular issue hasn't had any updates or activity in the past 90 days, so it's been labeled as 'stale'. If it remains inactive for the next 30 days, it'll be automatically closed.

We understand everyone's busy, but if this issue is still important to you, please feel free to add a comment or make an update to keep it active.

Thanks for your understanding and cooperation!