Why git lfs binary files are converted to pointers during git checkout?

[withtimesgo1115]

Why git lfs binary files are converted to pointers during git checkout?

Describe your question in as much detail as possible:

In our ci job, the GIT_LFS_SKIP_SMUDGE is set to 0 because we need to use the newest git lfs binary files. We want to speed up the CI job and the idea is to use the pvc mounted to each k8s job pod for consistent storage, to be more specific, to reuse the git repo and the bazel cache. However, the git lfs pull process is too slow(around 5min) 😦 . It is unexpected because we suppose the ci job can use the git repo instead of pull everything each time.
To debug this problem, I did a lot of work:
I added some git configs to the runner to trace the git operation and performance like the following:
GIT_TRACE: 1
GIT_TRACE_PERFORMANCE: 1
GIT_TRACE_SETUP: 1
Meanwhile, I also added "ls -la /path" logs before and after the critical operation (git fetch/git checkout/git clean) in the Gitlab Runner source and build the binaries for gitlab runner.

You can see the lfs binaries in the pvc are converted to pointers during check out process! This is the reason why the following git lfs pull process is slow.

The question is that why git lfs binary files are converted to pointers during git checkout? Anyone can help me out??? Thank you!!!

Steps to reproduce

• create pvc/pv resource
• use pvc working for consistent storage for our project git repo and bazel cache.
• deploy the k8s runner by helm chart
• trigger a pipeline and the job is executed by the pvc runner

Versions

• git lfs 3.2.0
• Gitlab 16.8.1

[chrisd8088]

Hey, I'm sorry you're having trouble. Would you be able to provide the exact series of commands you are running, along with the relevant environment variables? For instance, something like:

$ GIT_LFS_SKIP_SMUDGE=0 git clone https://...
$ cd ...
$ GIT_LFS_SKIP_SMUDGE=0 git checkout ...

or whatever the specific, complete set of steps is for your case? We don't need private information, of course.

[chrisd8088]

Hmm, interesting. One thing I can say is that there's a difference between GIT_LFS_SKIP_SMUDGE=1 and GIT_LFS_SKIP_SMUDGE=0. The latter (=0) means "convert pointers to objects" and the former (=1) means don't convert pointers to objects.

So if you're mixing those two settings (you mention "GIT_LFS_SKIP_SMUDGE is set to 0" in the original question, and export GIT_LFS_SKIP_SMUDGE=1 above), you might get variable results.

[withtimesgo1115]

Actually, I set the GIT_LFS_SKIP_SMUDGE = 0 for my Gitlab CI Job but I found this value would be forced converted to 1 in the Gitlab Runner source code.

func (b *AbstractShell) writeCloneFetchCmds(w ShellWriter, info common.ShellScriptInfo) error {
	build := info.Build

	// If LFS smudging was disabled by the user (by setting the GIT_LFS_SKIP_SMUDGE variable
	// when defining the job) we're skipping this step.
	//
	// In other case we're disabling smudging here to prevent us from memory
	// allocation failures.
	//
	// Please read https://gitlab.com/gitlab-org/gitlab-runner/issues/3366 and
	// https://github.com/git-lfs/git-lfs/issues/3524 for context.
	if !build.IsLFSSmudgeDisabled() {
		w.Variable(common.JobVariable{Key: "GIT_LFS_SKIP_SMUDGE", Value: "1"})
	}

	err := b.handleGetSourcesStrategy(w, build)
	if err != nil {
		return err
	}

	if build.GetGitCheckout() {
		b.writeCheckoutCmd(w, build)

		// If LFS smudging was disabled by the user (by setting the GIT_LFS_SKIP_SMUDGE variable
		// when defining the job) we're skipping this step.
		//
		// In other case, because we've disabled LFS smudging above, we need now manually call
		// `git lfs pull` to fetch and checkout all LFS objects that may be present in
		// the repository.
		//
		// Repositories without LFS objects (and without any LFS metadata) will be not
		// affected by this command.
		//
		// Please read https://gitlab.com/gitlab-org/gitlab-runner/issues/3366 and
		// https://github.com/git-lfs/git-lfs/issues/3524 for context.
		if !build.IsLFSSmudgeDisabled() {
			w.IfCmd("git", "lfs", "version")
			w.Command("git", "lfs", "pull")
			w.EmptyLine()
			w.EndIf()
		}
	} else {
		w.Noticef("Skipping Git checkout")
	}

	return nil
}

[withtimesgo1115]

@bk2204 @chrisd8088 any suggestions? I also used strace to see what happened where I found that there are many unlink and write system calls during checkout process.

[chrisd8088]

I'm sorry, we're not too familiar with GitLab, so you might need to contact their Support team to find out what their code is doing. But if it's setting GIT_LFS_SKIP_SMUDGE=1 then it seems like it could be the source of the problem.

If you're able to reproduce this in a local environment, please do share the reproduction steps! Until then I fear it will be a bit difficult to analyze.

[bk2204]

I agree with Chris that the CI is the source of the problem here. It probably should not be forcing GIT_LFS_SKIP_SMUDGE to 1, and if it is, that's why you're seeing that. You'll probably want to open a bug with them asking them to stop that behaviour or allow it to be overridden.

In the mean time, you can get the large files by running git lfs pull.