object not found when trying to pull a repository cloned with Depth: 1

[stingalleman]

Hi,

When you clone a repository with Depth: 1, you cannot pull it. It'll error with object not found.

Cloning:

_, err := git.PlainClone(repoPath, false, &git.CloneOptions{
	URL: gitURL,
	Depth: 1,
	})

Pulling:

// Open repo
r, err := git.PlainOpen(repo.Path)
if err != nil {
	panic(fmt.Errorf("error while opening git repo (%s) %s", repo.Name, err))
}
// Get worktree
tree, err := r.Worktree()
if err != nil {
	panic(fmt.Errorf("error while opening git repo (%s) %s", repo.Name, err))
}

// Pull
err = tree.Pull(&git.PullOptions{})
// If repo is already up to date, do nothing
if err == git.NoErrAlreadyUpToDate {
	// do nothing
} else if err != nil {
	panic(fmt.Errorf("error while pulling git repo (%s) %s", repo.Name, err))
}

It's critical that I can pull with Depth: 1, because the repository I'm pulling might be very large (hundreds of thousands of commits). Can somebody take a look at this?

[stingalleman]

@jfontan @mcarmonaa any idea?

[wyarde]

TL;DR - Use file://path/to/repo and you should be good

I ran into similar problems recently. Initially I worked around the shallow fetch issue by doing a full fetch. The fetch was actually fast, but then the push itself was taking 10 seconds. Weird... So I ended up spending more time trying to understand what was going wrong.

Unfortunately I'm not familiar enough with the code base to fix the actual issue, but hopefully my brain dump helps for people to locate the issue, or at least provides useful clues how to work around it:

Without further ado, here are my findings:

For local urls, the code in revlist.go // ObjectsWithStorageForIgnores constructs a list of all objects to be pushed. Internally, it constructs a list of objects to be ignored first. This ignore list also includes the commits which as a result of the shallow fetch don't have a parent present:

	ignore, err := objects(ignoreStore, ignore, nil, true)
	if err != nil {
		return nil, err
	}

	return objects(s, objs, ignore, false)

The first issue I ran into when doing a shallow fetch and then a push is I think what you run into as well. The first objects() call is unable to locate the objects to be ignored, because the path to the objects it constructed did not include the .git folder. That's where I realized I provided the local repository path without ".git". Formally this I guess is a mistake on my side since documentation I could find on how to clone a local repository says you should point to the ".git" folder. However, the git CLI doesn't care it works in either case. And also go-git seems to largely work and just fail in this unexpected way. So I think an improvement here would be to either enforce the path in the expected format, or fix the logic to work either way.

However when including ".git" in the path url, I ran into the next issue: The first call to objects() in the code above, the one to obtain all objects to ignore, was taking 5 seconds to complete. It returned a list of 32K objects which I presume is the full repository. Well that's twice as fast as the non-shallow fetch/push, but still horribly slow to push just a single object.

So then I realized I could bypass the "local url" logic by providing the repository in the file://path/to/repo format (Including .git is no longer required). The push now takes less than .1 second.

[stingalleman]

@wyarde thank you so much for posting your findings! I’m going to try this when I’m home- thank you so much

[stingalleman]

I hope that the maintainers can fix this bug with this information.

[stingalleman]

@wyarde, I tried opening the repository with file://, but it can't find the actual repository now.

r, err := git.PlainOpen(repo.Path)

[wyarde]

What does your file path look like?
If this is on windows, you'll need to use forward slashes, for example file://c:/my/path/to/the/repo.

[wyarde]

(If not on windows, you would still need to use forward slashes, but I guess it's more intuitive then)

[marguerite]

same issue here, the code I use is exactly the plain clone code the issue author mentioned except for the gitURL which is “https://github.com/rime/rime-cantonese”

and here’re the things I found so far.

the commit treeobject got from getTreeFromCommitHash in worktree.Reset() is still good, you can print the names etc. but when getting the actual file with t.File in checkoutChangeRegularFile, the file blob can’t have a Reader(). eg in plumbing/object/object.go. the blob obj is actually a plumbing.EncodedObject.

And then I lost my direction because EncodedObject is an interface, there are so many ways in go-git to create one.

BTW, the file in trouble in my previous git url is “jyut6ping3.dict.yaml” and the commit in trouble is “ 1168d4ca475656748938475ca5d96e41096d8b5a“.

Hope this helps for who interested in further debugging. For now I have to add a “use-system-git” option for my app...sad

[marguerite]

aha...sorry for my example, you can’t even plain clone https://github.com/rime/rime-cantonese

[stingalleman]

@jfontan @mcarmonaa any idea?

[stingalleman]

What does your file path look like?
If this is on windows, you'll need to use forward slashes, for example file://c:/my/path/to/the/repo.

So sorry @wyarde I completely forgot and have been really, really busy. My file path is just file://Users/stingalleman/git-repo

[wyarde]

@stingalleman is this on windows? You need to include the drive letter as in my example

[stingalleman]

@wyarde No, I am on a MacOS system.

[wyarde]

@stingalleman sorry cannot help you there

[stingalleman]

Hi @jfontan @mcuadros, can you take a look at this? Still waiting :-)

[laszlocph]

I'm getting empty git-upload-pack given for the same scenario, with v5.4.2

[rotty3000]

A colleague and I have solved this issue. The iterator used in the fastforward check wasn't prepared for the case when a commit might have a missing parent (as in shallow clones). Handling that case solved the problem of pulling on shallow clones.

[rotty3000]

I think I've actually found a better solution to this problem. It seems that the concept of haves v.s wants is not balanced in that shallowness is only checked in the wants side of the checks but not in the haves side.

This iteration should check if c.Hash is a shallow reference, something like this:

@@ -691,6 +691,15 @@ func getHavesFromRef(
 	toVisit := maxHavesToVisitPerRef
 	return walker.ForEach(func(c *object.Commit) error {
 		haves[c.Hash] = true
+
+		if s, _ := s.Shallow(); len(s) > 0 {
+			for _, sh := range s {
+				if sh == c.Hash {
+					return storer.ErrStop
+				}
+			}
+		}
+
 		toVisit--
 		// If toVisit starts out at 0 (indicating there is no
 		// max), then it will be negative here and we won't stop

[williambanfield]

Hey! I was wondering when we may expect this to be released? I have a few changes that are ready to make use of this change and would be interested to know the timeline.
Thanks!

[pjbgf]

@williambanfield I am not sure when the next release will happen, but it will probably be by the end of this month. If you need the changes sooner, use the pseudo-version instead:

go get github.com/go-git/go-git/v5@master

[jmriebold]

Is this fix actually working for anyone? Because even with v5.8.1 I'm getting the same ErrObjectNotFound any time a commit is pushed to a branch in a shallow-cloned repo.

[AriehSchneier]

@jmriebold this bug was about pulling/fetching, if you are having issues with pushing then please see if there is another issue with that bug or open a new one.

[jmriebold]

Sorry, what I mean is that I still get object not found when I try to pull a new commit on a shallow-cloned repo.

[jmriebold]

@AriehSchneier do you have any suggestions? If not I think this issue should be reopened.

[AriehSchneier]

@jmriebold are you able to write a test, or tell us the steps to be able to reproduce it?

[stvnksslr]

@AriehSchneier ill put together something that can just be checked out and run but i can reproduce with the following.

1. checkout with git clone --depth 1 <anyrepo> anyrepo1
2. checkout another copy somewhere else git clone anyrepo2
3. use the example fetch code from go-git add depth to the PullOptions
4. make a change to the second such as adding a line of text to a text file.
5. run the git-go code example to pull in

I have been using this simple repo to reproduce https://github.com/stvnksslr/sandbox-git

here is a gist with modifications to the example pull https://gist.github.com/stvnksslr/589c029055b40b3e2736482e90fc784a

edit: made reproduction steps a little more clear.

[AriehSchneier]

Ohh, you are trying to change the depth (unshallow), that is probably being tracked in #328

[stvnksslr]

@AriehSchneier Thank you for responding! I really appreciate you helping us get to the root of this issue.

To be clear I dont think this is related to unshallowing, its a cloned repo with a depth of 1 and a pull with a depth of 1 returning "object not found" when a change is made in the repo and the pull is initiated.

created a repo that should reproduce the issue end to end fairly easily

https://github.com/stvnksslr/sandbox-go-git

build the project
./sandbox-go-git https://github.com/stvnksslr/sandbox-git.git

substitute the repository for one you are able to make changes too

1. run the app to checkout the repo ./sandbox-go-git
2. run the app to pull (should work)
3. make a change in the other repo
4. run the app to pull again "object not found" will be returned as an error

[stvnksslr]

@pjbgf @AriehSchneier Apologize for the tag, the steps for reproducing this issue seem straight forward unsure as to why it wouldn't show up in the tests. let me know if there is anything else I can do to help with any of the leg work for this issue.

Additionally I can confirm the issue is not present with the same code example if the depth flags are removed from both the clone and the pull.

[Tang8330]

Got really excited about #932, updated dependency to use this...but still encountering the git pull problem.

Any workarounds and also ideas on where this is happening?

[dhoizner]

Got really excited about #932, updated dependency to use this...but still encountering the git pull problem.

Any workarounds and also ideas on where this is happening?

+1 from me- the behavior that i'm running into is:

• pulls on a shallow clone when there have been no updates to the remote branch succeed
• pulls on a shallow clone when there are new changes in the remote branch faIl with the "object not found" error