Prune fails: too many files open #3752
Comments
|
Hey, Can you provide the output of running Also, we fixed at least one issue with file descriptor leaks in 2.8.0, so trying that version may help as well. |
|
Running into same issue on Logs: fd limit of 256, while low, is the default on MacOS. Ubuntu default IIRC is 1024. Might be related to #2694 as our repo is quite large (136k commits, 27GB on disk) |
|
Okay, it's helpful to know that 256 is the default on macOS. I think we'll probably need to bump down the number of processes we use. I probably haven't noticed because my shell configuration unlimits everything it can. I'll flag this accordingly. |
|
I have a draft PR in #3842 that should fix this. Could you possibly test with that branch and see if it fixes the issue for you? If not, I can adjust the heuristic. |
|
@bk2204 Unfortunately no-go. |
|
What looks like is happening is the underlying |
|
Can you try again with the version I've just pushed up which makes that change? |
|
I stuck in some prints to validate how many threads it would limit it to. Despite it supposedly using only 3 threads, there are still dozens of git processes. Consisting of multiples of these commands: And one of these: |
|
Just to make the implied workaround explicit, running git: 2.20.1 |
|
I hit this as well and setting a high ulimit helped. 8 core 16" 2019 MBP. |
|
I am also still seeing this behavior, and ulimit seems to have worked around it. If any additional logs would help debug, just let me know and I'm happy to provide. |
|
Also received this bug on macOS High Sierra, 2015 MBP. ulimit works for me as well. |
|
I ran into this problem yesterday on a Mac running Monterey (12.3.1). No amount of fussing with I started poking around in the Removing the multiplier, and using was enough to allow the prune to succeed. Obviously, this is a pretty crude "fix" and probably isn't reliable for every case. I guess I need to find where all the descriptors went. Are we failing to close files after we look at them, or is this merely an artifact of all the subprocesses being spawned? |
|
Ran into this issue on ubuntu 20.04 (ulimit is 1024 - cannot be increased by normal user). Workaround was to use |
|
According to this stackoverflow post, Can the concurrency be exposed via a config option so that we can overwrite it at runtime based on the container size? This is true for other operations as well, but I'll open a separate issue for that. |
We'd certainly accept a patch to that effect. It's possible we'll get to it without that, but given the difficulty of reproducing this problem in order to fix it, I think someone who's actually experiencing the problem would be best suited to do so. |
|
I should also note that the number of CPUs is actually a heuristic since we don't know how many file descriptors will be opened, so it doesn't really matter what the number of available cores actually is. As a result, it's not necessarily super useful to adjust the number of cores, but rather we should implement a more robust solution, which nobody has suggested yet (and, because I cannot reproduce the problem repeatedly, I cannot implement). |
Running
git lfs pruneon a large repository fails. This is on Ubuntu 16.04.Here is the log.
Versions:
git-lfs/2.7.2 (GitHub; linux amd64; go 1.12.4)
git version 2.7.4
The text was updated successfully, but these errors were encountered: