Big .git folder

[McPatate]

Hello, I've noticed that when adding files to a git repository after running git lfs install and having the file types tracked in .gitattributes, the .git folder has a size equal to the sum of the files in the directory.
Looking more closely, the blobs are stored in the .git/lfs/objects/... folder and are a copy of the objects.

Is this expected behavior ? When having big repos (think over a TiB), this can be cumbersome and feels like a waste of space. It may be that there's just some sort of symbolic link, but usually when running ls you have that information, when here you do not.
I'm basing my question on running du -h -d 1 . inside the git folder.

[chrisd8088]

Hey, thanks for the question. Yes, this is expected behaviour. To keep the size of the .git/lfs/objects directory under some limit, you can use the git lfs prune command, which has a variety of options.

You might also be to use the git lfs dedup command if you have an operating system and filesystem which support it.

[McPatate]

Ok, thanks for your answer ! Is it lfs that creates the copies ? What are they needed for ?

I'll check git lfs dedup.

[chrisd8088]

Is it lfs that creates the copies ? What are they needed for ?

Yes, Git LFS stores the full object data in .git/lfs/objects whenever it fetches an object from the remote storage. However, not all objects will be used in the particular revision that is checked out in the working tree. Since fetching remote objects is expensive, Git LFS caches them so you can check out different revisions, branches, etc. without the expense of re-fetching objects each time. But the cost of doing that is that the cache may consume more local storage space than desired. So that's why git lfs prune exists, to let you control what is in the cache.

[McPatate]

I see, this makes sense ! Thank you for the explanation :)

[bbartoldson]

I am using git lfs pull to download 40 TB of data from a remote repo, but I don't have space for the git lfs objects and the data (80 TB total because all the data gets duplicated in.git/lfs/objects/*). Is there a way to keep the data AND get the assurance that I have grabbed all the data from the repo (which I think git lfs provides) without permanently storing the files in .git/lfs/objects/*?

For example, currently I have 20 TB of data downloaded, and I am using 40 TB of disk space because of git lfs objects. I want to download the next 20 TB of data, but I will run out of disk space (I only have 50 TB of space total) in the process. My plan is to remove the git lfs objects for the existing data to free up space for new data, but this introduces two concerns:

1. Once I have all the data and have deleted all their associated git lfs objects, will git be able to assure me that I have all the data downloaded? For example, I would hope that running git restore --source=HEAD :/ would suggest that everything is downloaded.
2. Will all of the data be usable without the associated git lfs objects? I know I may lose some git functionality by not have these objects, but I am only concerned about git's ability to tell me that I have downloaded everything.

[LiberiFatali]

@bbartoldson Did you find a solution?

[chrisd8088]

Hey, sorry I didn't notice this question earlier.

One thing to note here is that Git itself can't answer the question of whether your Git LFS objects have been downloaded or not, because it only sees the Git LFS pointer files in the actual Git repository contents. There are some Git LFS commands and options which might help you out, though.

I'm not entirely sure how your repository is structured, but one thing you might try would be something like:

$ GIT_LFS_SKIP_SMUDGE=1 git clone <url> repo
$ cd repo
$ git lfs ls-files
...
$ git lfs pull -I <path1>
$ git lfs pull -I <path2>

The GIT_LFS_SKIP_SMUDGE=1 option tells the Git LFS client not to download any Git LFS objects, so you'll just have the Git repository contents and you'll see Git LFS pointer files in place of your any Git LFS objects. For any such objects in HEAD, the git lfs ls-files command should list them, with a - instead of a * to show that only the pointer file resides in the working tree and Git LFS cache, and there's no copy of the object yet.

You can then selectively download specific paths using the -I option to git lfs pull. Once you've pulled down the objects you want, you can delete those and their copies in .git/lfs/objects (perhaps by using git lfs prune, or just by deleting those files directly), and proceed to download other Git LFS objects.

That all said, if you have Git LFS objects which are not in the HEAD commit's tree, but only exist at previous points in your Git commit history, or in other branches, you'll have to adjust your workflow to account for that (e.g., by using git lfs pull <ref> to find the Git LFS pointer files in specific commits or references). Or you could use GIT_LFS_SKIP_SMUDGE=1 git checkout to check out specific commits or refs without immediately pulling the Git LFS objects in them, and then pull any specific Git LFS objects you wanted after that.

[LiberiFatali]

Thank you @chrisd8088, that helps.