Remove old versions of a file

[memeplex]

Hi,

I've been trying to get rid of old versions of files in lfs storage. My strategy is the following:

1. Get rid of them locally
2. Remove the remote repository in GitHub
3. Push the cleaned up local repo to recreate the remote

Unfortunately, I think there is no way around 2, but that's not my current issue. Thing is I'm stuck at 1.

For example, to remove all but the last version of test.txt I'm doing this:

git filter-branch --tree-filter '[[ $GIT_COMMIT != $last_rev ]] && git rm -rf test.txt || true' HEAD

Where $last_rev is the revision to keep. That indeed removes the contents of the file but the pointers seem to remain:

❯ git lfs ls-files -a
7de1555df0 * test.txt
1121cfccd5 * test.txt
53c234e5e8 * test.txt
4355a46b19 * test.txt

Another thing I tried is to prune the repo with git lfs prune. But it doesn't seem to transfer lfs objects to a newly created repository (see detailed example in the next comment).

I would like to cleanly prune contents and pointers so that in the end I have a repo that is the same than the original except that ancient versions of lfs content have been removed.

Thank you in advance,
Carlos

[memeplex]

Here is a detailed example of the procedure I'm following to get rid of old versions by pruning, notice how it fails at the end.

1. Create a test repo in GitHub.
2. Create a test repo locally:

   cd /tmp
   mkdir r
   cd r
   git init
   git lfs install

3. Add some versions of a lfs-tracked file:

   git lfs track test.txt
   x=1; echo $x > test.txt && git add test.txt && git commit -m $x
   x=2; echo $x > test.txt && git add test.txt && git commit -m $x
   x=2; echo $x > test.txt && git add test.txt && git commit -m $x
   x=2; echo $x > test.txt && git add test.txt && git commit -m $x
   git lfs ls-files -a
      => 7de1555df0 * test.txt
         1121cfccd5 * test.txt
         53c234e5e8 * test.txt
         4355a46b19 * test.txt
   git lfs prune
     => prune: 4 local objects, 4 retained, done.

   No pruning because there is no remote copies yet, ok.
4. Add remote and push:

   git remote add origin git@github.com:memeplex/test.git
   git push -u origin master

5. Prune again:

   git lfs prune
      => prune: 4 local objects, 1 retained, done.
      => prune: Deleting objects: 100% (3/3), done.
    git lfs ls-files -a
      => 7de1555df0 * test.txt
       1121cfccd5 * test.txt
       53c234e5e8 * test.txt
       4355a46b19 * test.txt
    find .git/lfs/objects/ -type f
      => .git/lfs/objects//7d/e1/7de1555df0c2700329e815b93b32c571c3ea54dc967b89e81ab73b9972b72d1d

6. Remove remote repo, recreate it from scratch, push pruned local repo, clone it and fetch lfs files:

   git push -u origin master
   cd ..
   git clone git@github.com:memeplex/test.git r2
   cd r2
   git lfs ls-files -a
      => 7de1555df0 * test.txt
       1121cfccd5 * test.txt
       53c234e5e8 * test.txt
       4355a46b19 * test.txt
   git lfs fetch
      => [7de1555df0c2700329e815b93b32c571c3ea54dc967b89e81ab73b9972b72d1d] Object does not exist on the server: [404] Object does not exist on the server
         error: failed to fetch some objects from 'https://github.com/memeplex/test.git/info/lfs'
   

For some reason 7de1555df0 wasn't pushed to the new remote, even though it exists locally at r/.git/lfs/objects.

EDIT: I was able to get it pushed by running git lfs push --all origin, but I don't understand why this is needed. Perhaps this overrides git-lfs assumption that the contents are already at the remote?

EDIT: moreover, this strategy fails when I want to checkout a revision that has a dangling pointer:

❯ git checkout master~
Downloading test.txt (2.8 MB)
Error downloading object: test.txt (8d0dcd1): Smudge error: Error downloading test.txt (8d0dcd1357d4ed89bca5b98c6b3e8a6dabdae79e15d9f547314dcfa36af88c36): error transferring "8d0dcd1357d4ed89bca5b98c6b3e8a6dabdae79e15d9f547314dcfa36af88c36": [0] remote missing object 8d0dcd1357d4ed89bca5b98c6b3e8a6dabdae79e15d9f547314dcfa36af88c36

This makes sense but goes against my goal. Ideally it should work as if test.text was never there. It not, at least it should checkout the pointer file without failing.

[memeplex]

Here is another alternative that I've been thinking about, in order to avoid having to deal with pointers and pointees while filtering:

1. First export with git lfs migrate.
2. Then process the output with git filter-repo/branch
3. Finally import with git lfs migrate.

I've not tested this approach yet. But do you think it's sensible? Or maybe overkilling?

[chrisd8088]

Hey, thanks for the question and detailed descriptions of the approaches you've tried.

If I understand correctly, you're hoping to rewrite earlier commits in your Git history to remove Git LFS objects in those commits. I think the key point is that, assuming you want a "clean" history where you can checkout any particular commit without problems, you'll have to decide if those earlier commits should have (a) the original file, not converted to a Git LFS object, or (b) no file at all. If you leave Git LFS pointers in the history but remove all copies of the corresponding objects, I suspect you'll keep running into the problem you mentioned, i.e., you can't check out a commit with some Git LFS pointer files without the Git LFS client complaining:

Ideally it should work as if test.text was never there. It not, at least it should checkout the pointer file without failing.

Here's a working example of a simple repo, derived from yours above, where we just remove the files entirely from earlier commits.

$ git init test
$ cd test
$ git lfs track test.txt && echo foo >foo.txt && git add .gitattributes foo.txt && git commit -m init
$ x=1; echo $x > test.txt && git add test.txt && git commit -m $x
$ x=2; echo $x > test.txt && git add test.txt && git commit -m $x
$ x=3; echo $x > test.txt && git add test.txt && git commit -m $x
$ x=4; echo $x > test.txt && git add test.txt && git commit -m $x

$ FILTER_BRANCH_SQUELCH_WARNING=1 git filter-branch \
    --setup 'LAST_REV="$(git rev-list -n 1 HEAD)"' \
    --tree-filter 'if [ "$GIT_COMMIT" != "$LAST_REV" ]; then rm -f .gitattributes test.txt; fi' \
    HEAD

$ git log --name-status --oneline
e357873 (HEAD -> main) 4
A       .gitattributes
A       test.txt
db3c8f0 3
e651f9b 2
094a590 1
47d967a init
A       foo.txt

However, this is a very restricted example. Using git filter-branch has a lot of perils, and if you have more than one branch, you'd have to rewrite them individually (at least, with this approach).

Also, the example simply adds .gitattributes to the original commit history, and then removes it from the commits where the Git LFS object file is removed, so it appears in the rewritten history at the point where it is needed. But this won't be appropriate for a more complex history where there might be multiple files matching multiple .gitattributes rules (especially wildcard rules like *.txt), unless you plan to remove every Git LFS object from every commit except the most recent ones.

Personally, I think your proposal to use git lfs migrate export as a first step makes a lot of sense, as that should handle .gitattributes and you can specify the branches and filename patterns you want to deal with, and you'll get a "clean" repository without Git LFS objects but the corresponding plain Git file blobs in their place. Then, assuming you want to remove those Git file blobs entirely from your history, rewrite your history as appropriate, and then convert back to Git LFS with git lfs migrate import, as you suggested. Good luck!

[memeplex]

Thank you for the detailed and kind answer, it was worth the wait. I'll go with the export/import approach then.