Git LFS should print Kerberos service prinicpal for debugging

[AJH16]

Describe the bug
Attempting to authenticate to Azure Devops Server's (on prem) git repos via Kerberos fails on some machines. Kerberos authentication works correctly for Azure Devops Server website itself. This occurs only on select machines. Most machines in our environment work without issue and can authenticate GitLFS via Kerberos, however select environments fail to authenticate and fall back to basic auth (which fails due to a seperate issue with Azure Devops Server's treatment of git vs git-lfs.)

To Reproduce
Steps to reproduce the behavior:

1. On a computer experiencing the problem, attempt any operation that requires connecting to the GitLFS file server.

Expected behavior
GitLFS operates normally.

System environment
Latest version of Git for Windows, though it also has been a problem with previous. Running the latest version of GitLFS (3.4.1) as the 3.4.0 build that was bundled with Git had issues with accessing out of bounds values when trying to connect via basic auth after not using or failing to use Negotiate/Kerberos. Windows 10 machine.

Output of git lfs env

git-lfs/3.4.1 (GitHub; windows amd64; go 1.21.6)
Error getting Git version: exec: "git": cannot run executable found relative to current directory

Error: failed to call `git rev-parse --git-dir --show-toplevel`: ""
Error reading `git config`: exec: "git": cannot run executable found relative to current directory
LocalWorkingDir=
LocalGitDir=
LocalGitStorageDir=
LocalMediaDir=lfs\objects
LocalReferenceDirs=
TempDir=lfs\tmp
ConcurrentTransfers=8
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic,lfs-standalone-file,ssh
UploadTransfers=basic,lfs-standalone-file,ssh
GIT_EXEC_PATH=C:/Program Files/Git/mingw64/libexec/git-core
GIT_LFS_PATH=C:\Program Files\Git LFS
git config filter.lfs.process = ""
git config filter.lfs.smudge = ""
git config filter.lfs.clean = ""

Additional context
I have a locally built version of git lfs prepared and am attempting to debug, however I've been encountering problems getting it to run on the repo as git is not detected despite my cwd being set to a git repository root.

[AJH16]

I'm mostly looking for some assistance with figuring out the git lfs debugging issue while trying to troubleshoot this problem, but figured it wise to fill out for the primary issue I'm trying to troubleshoot. I'm running git lfs in VS Code and when the debugger is attached, I can not get it to run within the context of my Git repository, which prevents reaching the code path I need to debug. (Looking in to what is causing the negotiate option to be false and having it fall in to basic auth.)

Here is my current launch.json:

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [

        {
            "name": "Launch Package",
            "type": "go",
            "request": "launch",
            "mode": "auto",
            "program": "C:\\Git\\git-lfs\\",
            "cwd": "c:\\Git\\testreponame",
            "args": ["pull"],
            "console": "externalTerminal",
            "output": "debug"
        }

    ]
}

[bk2204]

Hey,

Let me provide a little bit of assistance. First, the message “Error reading git config: exec: "git": cannot run executable found relative to current directory” means that you have a git (probably git.exe) binary in the same directory as you're running in. That's simply not going to work at all on Windows and you'll have to move it out.

I'll just say that I don't use VS Code (I'm a Neovim user) and I also only incidentally use Windows, but I wonder if the line "program": "C:\\Git\\git-lfs\\", should be different. It looks like you've specified a directory rather than an actual program, which may explain why it's not running. Maybe you want to append bin\\git-lfs.exe or something like that?

If, while you're in the course of debugging this, you decide that you'd like to improve the error message logging in the trace output here, we'd of course love to see a patch for that. I think everyone would agree that it would be nicer to see this in the output of GIT_TRACE=1 git lfs pull (to be typed in Git Bash) rather than having to run a debugger. I vaguely recall having updated some of that in the past, so if you haven't tried that, it might be worth a shot first.

[AJH16]

That part seems to be right for the go debugger. I do end up with the debugger attached to git-lfs successfully, however it does a test to see if it is in a folder with a git repository and that check fails. (In particular, it's the "rev-parse" call to gitNoLFS on line 749 of git.go in the release-3.4 branch.) This is preventing me from ever getting to the pull I'm trying to run since the execution context for git lfs is off somehow.

I suppose I could try dumping the full error and environment from the gitNoLFS call and see if that gives me any more useful details to work on if there isn't anyone else familiar with running gitlfs straight out of vs code's go debugger.

[bk2204]

I don't think anyone else on the core team is familiar with VS Code or its debugger. I believe we're all Vim or Neovim users, and since we run on Unix, we'd use gdb directly (as gdb --args git-lfs pull).

[AJH16]

Oh wait, I think I might see it. When I switched my working folder over to the actual git repo and ran the commands from there I got a clean git lfs env without errors. I had been trying to run the git lfs env initially from where the binary actually is built by git lfs and it seems that doesn't work. Might just be that the cwd isn't sufficient by itself, so I may need to figure out how to get it to build in to the final path that git lfs is running from on my system so that git is also properly accessible. I'm a bit surprised my path isn't fixing that, but it seems like maybe it could be something along those lines.

Or maybe not. Looks like my original git lfs env output was just a bad directory I chose to run it in. After switching to either a real repo or the folder that VS Code is building git-lfs.exe to, I get the following as the git lfs env:

git-lfs/3.4.1 (GitHub; windows amd64; go 1.21.6)
git version 2.43.0.windows.1

Endpoint=https://github.com/git-lfs/git-lfs.git/info/lfs (auth=none)
LocalWorkingDir=C:\Git\git-lfs
LocalGitDir=C:\Git\git-lfs\.git
LocalGitStorageDir=C:\Git\git-lfs\.git
LocalMediaDir=C:\Git\git-lfs\.git\lfs\objects
LocalReferenceDirs=
TempDir=C:\Git\git-lfs\.git\lfs\tmp
ConcurrentTransfers=8
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=C:\Git\git-lfs\.git\lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic,lfs-standalone-file,ssh
UploadTransfers=basic,lfs-standalone-file,ssh
GIT_EXEC_PATH=C:/Program Files/Git/mingw64/libexec/git-core
GIT_LFS_PATH=C:\Program Files\Git LFS
git config filter.lfs.process = "git-lfs filter-process"
git config filter.lfs.smudge = "git-lfs smudge -- %f"
git config filter.lfs.clean = "git-lfs clean -- %f"

[AJH16]

For anyone trying to solve similar debugging issues on Windows in VS Code, what finally worked was to build outside of VS Code with go build -gcflags=all="-N -l" I further added a wait for console input to the beginning of main so that I had time to attach a debugger instead of launching from VS Code directly.

[AJH16]

Ok, so made a major breakthrough on the root of this problem. I still have to track back to why it's getting selected, but it appears that an incorrect SPN is assumed from the URL in gitlfs,, probably from the spnego library, but I haven't confirmed yet. On a machine where gitlfs works, the SPN used is a valid and expected one, however on machines where it does not work, a different endpoint on the machine is selected which is unrelated to the Azure Devops endpoint and does not have a configured SPN record.

As a short term fix I am going to try updating my SPN records to include SPNs for the selected SPN for the host being used. I'm also suggesting that we include the selected SPN in trace output to aid debugging for anyone who isn't capable of building and debugging their own version of gitlfs. I'll also continue looking in to why the incorrect SPN name was selected to determine if there is some way to prevent the issue in the first place.

[AJH16]

Ok, so it looks like the URL canonicalize functionality is entirely down in spnego so it does not appear possible to correct from gitlfs. So I think the best we can do here is try to get the used SPN out in the trace and then I'll reach out to the spnego team about the canonicalization issue since it should really be attempting a connection with the provided host prior to canonicalizing it through a DNS lookup/reverse lookup.

[AJH16]

Another note for anyone else having this problem. Check your hosts files. In our case, things were fixed when we removed a host file record for this ip from the machine. Alternately, if you need the hosts file record, just ensure that you also have a record for the proper SPN hostname earlier in the file for the same IP.

[bk2204]

Ok, so it looks like the URL canonicalize functionality is entirely down in spnego so it does not appear possible to correct from gitlfs. So I think the best we can do here is try to get the used SPN out in the trace and then I'll reach out to the spnego team about the canonicalization issue since it should really be attempting a connection with the provided host prior to canonicalizing it through a DNS lookup/reverse lookup.

That sounds like a good idea. Are you willing to put the SPN in the trace output here, since you're already in this particular piece of code? I do have a Kerberos setup at home, but it's not set up for Git LFS by default, so it would take me some time to add it.

[AJH16]

Unfortunately I'm not able to anymore without a lot of hoops to jump through. Ownership changed of the company I work for and getting approval for contributions is... difficult. Trying to document here as much as I can to give maximum help to anyone else that comes along the same issue though. :/

[bk2204]

Okay, I'm going to mark this as an enhancement, then, since it will take some time for me to set up and I'm not necessarily going to be able to get to it in a reasonable amount of time.