Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

git lfs does not honor git -c url.<url>.insteadOf when inside a submodule #5665

Open
svalo opened this issue Feb 29, 2024 · 2 comments
Open

Comments

@svalo
Copy link

svalo commented Feb 29, 2024

Describe the bug
When trying to fetch objects tracked by lfs inside a submodule, url.<url>.insteadOf is apparently not used and fetch fails.

If the submodule is fetched directly then git -c url.<url>.insteadOf lfs fetch -a works

I noticed this issue during CI execution where git -c url.<url>.insteadOf is used to pass authentication token to git server. Here the original issue I submitted

Note: No issue arise during submodule clone because the submodule at the current version does not contain LFS objects

Issues arise when performing both git lfs fetch -a and git submodule checkout <version-with-lfs-files>

To Reproduce
Steps to reproduce the behavior:

  1. Clone a repository with a submodule containing LFS-tracked files
    1. git -c url.http://valo:password@gitlab.lan.insteadOf=http://gitlab.lan clone --recurse-submodules http://gitlab.lan/group/myproject.git
  2. Enter the submodule path and perform an lfs fetch
    1. git -c url.http://valo:password@gitlab.lan.insteadOf=http://gitlab.lan lfs fetch -a
  3. User prompted for username

If the same steps are performed directly cloning the repository used as submodule everything works

Expected behavior

LFS-tracked files are correctly retrieved

System environment
The version of your operating system, plus any relevant information about platform or configuration (e.g., container or CI usage, Cygwin, WSL, or non-Basic authentication). If relevant, include the output of git config -l as a code block.

  • Debian 12
  • git version 2.39.2
  • git-lfs/3.3.0 (GitHub; linux amd64; go 1.19.8)

I tried with both lfs.transfer.enablehrefrewrite=true and lfs.transfer.enablehrefrewrite=false but I'm getting the same result

Output of git lfs env

git lfs env without git -c
$ git lfs env  
git-lfs/3.3.0 (GitHub; linux amd64; go 1.19.8)
git version 2.39.2

Endpoint=http://gitlab.lan/group/myproject-gui.git/info/lfs (auth=basic)
LocalWorkingDir=/home/valo/tmp/debuglfs/myproject/myproject-gui
LocalGitDir=/home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui
LocalGitStorageDir=/home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui
LocalMediaDir=/home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/lfs/objects
LocalReferenceDirs=
TempDir=/home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/lfs/tmp
ConcurrentTransfers=8
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=/home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/lfs
AccessDownload=basic
AccessUpload=basic
DownloadTransfers=basic,lfs-standalone-file,ssh
UploadTransfers=basic,lfs-standalone-file,ssh
GIT_EXEC_PATH=/usr/lib/git-core
git config filter.lfs.process = "git-lfs filter-process"
git config filter.lfs.smudge = "git-lfs smudge -- %f"
git config filter.lfs.clean = "git-lfs clean -- %f"
git lfs env using -c
$ git -c url.http://valo:password@gitlab.lan.insteadOf=http://gitlab.lan lfs env
git-lfs/3.3.0 (GitHub; linux amd64; go 1.19.8)
git version 2.39.2

Endpoint=http://gitlab.lan/group/myproject-gui.git/info/lfs (auth=basic)
LocalWorkingDir=/home/valo/tmp/debuglfs/myproject/myproject-gui
LocalGitDir=/home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui
LocalGitStorageDir=/home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui
LocalMediaDir=/home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/lfs/objects
LocalReferenceDirs=
TempDir=/home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/lfs/tmp
ConcurrentTransfers=8
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=/home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/lfs
AccessDownload=basic
AccessUpload=basic
DownloadTransfers=basic,lfs-standalone-file,ssh
UploadTransfers=basic,lfs-standalone-file,ssh
GIT_CONFIG_PARAMETERS='url.http://valo:password@gitlab.lan.insteadOf'='http://gitlab.lan'
GIT_EXEC_PATH=/usr/lib/git-core
git config filter.lfs.process = "git-lfs filter-process"
git config filter.lfs.smudge = "git-lfs smudge -- %f"
git config filter.lfs.clean = "git-lfs clean -- %f"

Additional context
Any other relevant context about the problem here.

Successfull run when cloning the project directly
valo@laptop ~/tmp/debuglfs % git -c url.http://valo:password@gitlab.lan.insteadOf=http://gitlab.lan clone http://gitlab.lan/group/myproject-gui.git
Cloning into 'myproject-gui'...
remote: Enumerating objects: 1043, done.
remote: Counting objects: 100% (157/157), done.
remote: Compressing objects: 100% (155/155), done.
remote: Total 1043 (delta 62), reused 0 (delta 0), pack-reused 886
Receiving objects: 100% (1043/1043), 76.38 MiB | 17.28 MiB/s, done.
Resolving deltas: 100% (292/292), done.
valo@laptop ~/tmp/debuglfs % cd myproject-gui 
valo@laptop ~/tmp/debuglfs/myproject-gui (git)-[dev] % git -c url.http://valo:password@gitlab.lan.insteadOf=http://gitlab.lan lfs fetch -a                                           
fetch: 4 objects found, done.                                                                                                                                                                                                                 
fetch: Fetching all references...
Unsuccessfull run when using submodule
git  -c url.http://valo:password@gitlab.lan.insteadOf=http://gitlab.lan clone --recurse-submodules http://gitlab.lan/group/myproject.git
Cloning into 'myproject'...
remote: Enumerating objects: 2542, done.
remote: Counting objects: 100% (548/548), done.
remote: Compressing objects: 100% (540/540), done.
remote: Total 2542 (delta 401), reused 0 (delta 0), pack-reused 1994
Receiving objects: 100% (2542/2542), 5.61 MiB | 74.59 MiB/s, done.
Resolving deltas: 100% (1614/1614), done.
Submodule 'myproject-gui' (http://gitlab.lan/group/myproject-gui.git/) registered for path 'myproject-gui'
Cloning into '/home/valo/tmp/debuglfs/myproject/myproject-gui'...
remote: Enumerating objects: 1043, done.        
remote: Counting objects: 100% (157/157), done.        
remote: Compressing objects: 100% (155/155), done.        
remote: Total 1043 (delta 62), reused 0 (delta 0), pack-reused 886        
Receiving objects: 100% (1043/1043), 76.38 MiB | 17.20 MiB/s, done.
Resolving deltas: 100% (292/292), done.
Submodule path 'myproject-gui': checked out '0226751ed31f1b78ace52564df40986c6e22fba7'
valo@sds15 ~/tmp/debuglfs % cd myproject/myproject-gui 
valo@sds15 ..p/debuglfs/myproject/myproject-gui (git)-[remotes/origin/group-edits] % git -c url.http://valo:password@gitlab.lan.insteadOf=http://gitlab.lan lfs fetch -a
fetch: 4 objects found, done.                                                                                                                                                                                                                 
fetch: Fetching all references...
Username for 'http://gitlab.lan': 0 B | 0 B/s                                                                                                                                                                                              

Very verbose output obtained using GIT_TRACE_PACK_ACCESS=1 GIT_TRANSFER_TRACE=1 GIT_TRACE_SETUP=1 GIT_TRACE=1
when not using the submodule

valo@sds15 ~/tmp/debuglfs % GIT_TRACE_PACK_ACCESS=1   GIT_TRANSFER_TRACE=1 GIT_TRACE_SETUP=1 GIT_TRACE=1git -c url.http://valo:password@gitlab.lan.insteadOf=http://gitlab.lan clone http://gitlab.lan/group/myproject-gui.git
Cloning into 'myproject-gui'...
remote: Enumerating objects: 1043, done.
remote: Counting objects: 100% (157/157), done.
remote: Compressing objects: 100% (155/155), done.
remote: Total 1043 (delta 62), reused 0 (delta 0), pack-reused 886
Receiving objects: 100% (1043/1043), 76.38 MiB | 16.55 MiB/s, done.
Resolving deltas: 100% (292/292), done.
valo@sds15 ~/tmp/debuglfs % git config  --global lfs.transfer.enablehrefrewrite                                                                                                
false
valo@sds15 ~/tmp/debuglfs % cd myproject-gui 
valo@sds15 ~/tmp/debuglfs/myproject-gui (git)-[dev] % git config  --global lfs.transfer.enablehrefrewrite 
false
valo@sds15 ~/tmp/debuglfs/myproject-gui (git)-[dev] % git config  lfs.transfer.enablehrefrewrite          
false
valo@sds15 ~/tmp/debuglfs/myproject-gui (git)-[dev] % GIT_TRACE_PACK_ACCESS=1   GIT_TRANSFER_TRACE=1 GIT_TRACE_SETUP=1 GIT_TRACE=1 git -c url.http://valo:password@gitlab.lan.insteadOf=http://gitlab.lan checkout 4fd16e7db4d7200f8f0b8b8c836ec745d5f7d825
16:27:20.144276 trace.c:311             setup: git_dir: .git
16:27:20.144292 trace.c:312             setup: git_common_dir: .git
16:27:20.144295 trace.c:313             setup: worktree: /home/valo/tmp/debuglfs/myproject-gui
16:27:20.144297 trace.c:314             setup: cwd: /home/valo/tmp/debuglfs/myproject-gui
16:27:20.144298 trace.c:315             setup: prefix: (null)
16:27:20.144302 chdir-notify.c:65       setup: chdir from '/home/valo/tmp/debuglfs/myproject-gui' to '/home/valo/tmp/debuglfs/myproject-gui'
16:27:20.144305 git.c:460               trace: built-in: git checkout 4fd16e7db4d7200f8f0b8b8c836ec745d5f7d825
16:27:20.144524 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 22403012
16:27:20.144585 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 22405040
16:27:20.214306 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80086394
16:27:20.214357 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80085017
16:27:20.214365 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80080786
16:27:20.214367 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80085068
16:27:20.214378 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80085113
16:27:20.214384 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80085160
16:27:20.214403 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80083293
16:27:20.214410 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 384675
16:27:20.214416 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 22412233
16:27:20.214418 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 22412439
16:27:20.214433 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 384884
16:27:20.214502 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 585201
16:27:20.214509 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 22414544
16:27:20.214515 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 22414854
16:27:20.214523 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 585513
16:27:20.214547 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80083027
16:27:20.214556 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80083069
16:27:20.214562 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80083413
16:27:20.214581 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80083463
16:27:20.214584 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80080834
16:27:20.214590 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 4410913
16:27:20.214595 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 22424405
16:27:20.214607 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 9906310
16:27:20.214612 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80080879
16:27:20.214614 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 9923575
16:27:20.214617 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80081011
16:27:20.214619 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 9923661
16:27:20.214621 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80081096
16:27:20.214649 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80083867
16:27:20.214652 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80079038
16:27:20.214765 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80080663
16:27:20.214785 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80080663
16:27:20.214827 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 22405819
16:27:20.214860 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 22408988
16:27:20.214905 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 385043
16:27:20.214925 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 285520
16:27:20.214934 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 22412907
16:27:20.214948 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 22412994
16:27:20.214969 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 22414045
16:27:20.214991 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 22414264
16:27:20.215018 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 587069
16:27:20.215045 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 285520
16:27:20.215060 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 22415018
16:27:20.215089 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 22415387
16:27:20.215116 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 22416988
16:27:20.215144 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 22418837
16:27:20.215172 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 22424559
16:27:20.215961 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80081226
16:27:20.215978 run-command.c:655       trace: run_command: 'git-lfs filter-process'
16:27:20.218673 trace git-lfs: exec: git 'version'
16:27:20.219616 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
16:27:20.220434 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
16:27:20.221166 trace git-lfs: exec: git 'config' '--includes' '--local' 'lfs.repositoryformatversion'
16:27:20.221870 trace git-lfs: exec: git 'config' '--includes' '--replace-all' 'lfs.repositoryformatversion' '0'
16:27:20.222502 trace git-lfs: exec: git 'config' '--includes' '-l'
16:27:20.222966 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
16:27:20.223695 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' ':.lfsconfig'
16:27:20.224651 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' 'HEAD:.lfsconfig'
16:27:20.225677 trace git-lfs: Install hook: pre-push, force=false, path=/home/valo/tmp/debuglfs/myproject-gui/.git/hooks/pre-push
16:27:20.225712 trace git-lfs: Install hook: post-checkout, force=false, path=/home/valo/tmp/debuglfs/myproject-gui/.git/hooks/post-checkout
16:27:20.225733 trace git-lfs: Install hook: post-commit, force=false, path=/home/valo/tmp/debuglfs/myproject-gui/.git/hooks/post-commit
16:27:20.225752 trace git-lfs: Install hook: post-merge, force=false, path=/home/valo/tmp/debuglfs/myproject-gui/.git/hooks/post-merge
16:27:20.225767 trace git-lfs: Initialize filter-process
16:27:20.225847 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
16:27:20.226607 trace git-lfs: tq: running as batched queue, batch size of 100
16:27:20.226726 trace git-lfs: filepathfilter: accepting "src/assets/js/js9/dist/3c273.fits"
16:27:20.226761 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80081348
16:27:20.226824 trace git-lfs: filepathfilter: accepting "src/assets/js/js9/dist/J00_20220920221935_Q2B_2snr_182_133_bdfm.fit"
16:27:20.226933 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80081469
16:27:20.226993 trace git-lfs: filepathfilter: accepting "src/assets/js/js9/dist/coma.fits"
16:27:20.227026 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80080278
16:27:20.227077 packfile.c:1640         .git/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80079347
16:27:20.227119 trace git-lfs: tq: sending batch of size 3
16:27:20.227238 trace git-lfs: api: batch 3 files
16:27:20.227399 trace git-lfs: HTTP: POST http://valo:password@gitlab.lan/group/myproject-gui.git/info/lfs/objects/batch
16:27:20.327685 trace git-lfs: HTTP: 200
16:27:20.327723 trace git-lfs: HTTP: {"objects":[{"oid":"7f2d685f1aa8b769906507990fa90e2071a3eae55181dfd97a3900f2e34d56c0","size":67383360,"actions":{"download":{"href":"http://gitlab.lan/group/myproject-gui.git/gitlab-lfs/objects/7f2d685f1aa8b769906507990fa90e2071a3eae55181dfd97a3900f2e34d56c0","header":{"Authorization":"Basic [...]
16:27:20.327733 trace git-lfs: HTTP: [...]"}}}},{"oid":"9805bbf6ee4c0457936e8df983a4c4a924be7e871c15e744f2b476a91071dda0","size":13014720,"actions":{"download":{"href":"http://gitlab.lan/group/myproject-gui.git/gitlab-lfs/objects/9805bbf6ee4c0457936e8df983a4c4a924be7e871c15e744f2b476a91071dda0","header":{"Authorization":"Basic [...]="}}}},{"oid":"111ec9e2e9036e3eb1862ec8aa38b0b8b2e8569dfd1525ec4614f6ac41e99ae1","size":1004756,"actions":{"download":{"href":"http://gitlab.lan/group/myproject-gui.git/gitlab-lfs/objects/111ec9e2e9036e3eb1862ec8a
16:27:20.327762 trace git-lfs: HTTP: a38b0b8b2e8569dfd1525ec4614f6ac41e99ae1","header":{"Authorization":"Basic [...]="}}}}]}
16:27:20.327837 trace git-lfs: tq: starting transfer adapter "basic"
16:27:20.327843 trace git-lfs: xfer: adapter "basic" Begin() with 8 workers
16:27:20.327855 trace git-lfs: xfer: adapter "basic" started
16:27:20.327869 trace git-lfs: xfer: adapter "basic" worker 2 starting
16:27:20.327886 trace git-lfs: xfer: adapter "basic" worker 2 waiting for Auth
16:27:20.327884 trace git-lfs: xfer: adapter "basic" worker 0 starting
16:27:20.327858 trace git-lfs: xfer: adapter "basic" worker 1 starting
16:27:20.327893 trace git-lfs: xfer: adapter "basic" worker 0 processing job for "7f2d685f1aa8b769906507990fa90e2071a3eae55181dfd97a3900f2e34d56c0"
16:27:20.327898 trace git-lfs: xfer: adapter "basic" worker 1 waiting for Auth
16:27:20.327902 trace git-lfs: xfer: adapter "basic" worker 6 starting
16:27:20.327909 trace git-lfs: xfer: adapter "basic" worker 6 waiting for Auth
16:27:20.327895 trace git-lfs: xfer: adapter "basic" worker 5 starting
16:27:20.327915 trace git-lfs: xfer: adapter "basic" worker 5 waiting for Auth
16:27:20.327914 trace git-lfs: xfer: adapter "basic" worker 4 starting
16:27:20.327918 trace git-lfs: xfer: adapter "basic" worker 7 starting
16:27:20.327931 trace git-lfs: xfer: adapter "basic" worker 7 waiting for Auth
16:27:20.327908 trace git-lfs: xfer: adapter "basic" worker 3 starting
16:27:20.327922 trace git-lfs: xfer: adapter "basic" worker 4 waiting for Auth
16:27:20.327937 trace git-lfs: xfer: adapter "basic" worker 3 waiting for Auth
16:27:20.328140 trace git-lfs: HTTP: GET http://gitlab.lan/group/myproject-gui.git/gitlab-lfs/objects/7f2d685f1aa8b769906507990fa90e2071a3eae55181dfd97a3900f2e34d56c0
16:27:20.364960 trace git-lfs: HTTP: 200
16:27:20.364985 trace git-lfs: xfer: adapter "basic" worker 6 auth signal received
16:27:20.364995 trace git-lfs: xfer: adapter "basic" worker 2 auth signal received
16:27:20.364999 trace git-lfs: xfer: adapter "basic" worker 1 auth signal received
16:27:20.365001 trace git-lfs: xfer: adapter "basic" worker 2 processing job for "111ec9e2e9036e3eb1862ec8aa38b0b8b2e8569dfd1525ec4614f6ac41e99ae1"
16:27:20.365005 trace git-lfs: xfer: adapter "basic" worker 5 auth signal received
16:27:20.364995 trace git-lfs: xfer: adapter "basic" worker 6 processing job for "9805bbf6ee4c0457936e8df983a4c4a924be7e871c15e744f2b476a91071dda0"
16:27:20.365009 trace git-lfs: xfer: adapter "basic" worker 4 auth signal received
16:27:20.365004 trace git-lfs: xfer: adapter "basic" worker 7 auth signal received
16:27:20.365060 trace git-lfs: xfer: adapter "basic" worker 3 auth signal received
16:27:20.365580 trace git-lfs: HTTP: GET http://gitlab.lan/group/myproject-gui.git/gitlab-lfs/objects/9805bbf6ee4c0457936e8df983a4c4a924be7e871c15e744f2b476a91071dda0
16:27:20.365608 trace git-lfs: HTTP: GET http://gitlab.lan/group/myproject-gui.git/gitlab-lfs/objects/111ec9e2e9036e3eb1862ec8aa38b0b8b2e8569dfd1525ec4614f6ac41e99ae1
16:27:20.417050 trace git-lfs: HTTP: 200
16:27:20.441895 trace git-lfs: HTTP: 200
16:27:20.480685 trace git-lfs: xfer: adapter "basic" worker 2 finished job for "111ec9e2e9036e3eb1862ec8aa38b0b8b2e8569dfd1525ec4614f6ac41e99ae1"
16:27:20.480869 trace git-lfs: filepathfilter: accepting "src/assets/js/js9/dist/coma.fits"
16:27:20.767264 trace git-lfs: xfer: adapter "basic" worker 6 finished job for "9805bbf6ee4c0457936e8df983a4c4a924be7e871c15e744f2b476a91071dda0"
16:27:20.767474 trace git-lfs: filepathfilter: accepting "src/assets/js/js9/dist/3c273.fits"
16:27:21.057565 trace git-lfs: xfer: adapter "basic" worker 0 finished job for "7f2d685f1aa8b769906507990fa90e2071a3eae55181dfd97a3900f2e34d56c0"
16:27:21.057593 trace git-lfs: xfer: adapter "basic" End()
16:27:21.057607 trace git-lfs: xfer: adapter "basic" worker 1 stopping
16:27:21.057611 trace git-lfs: xfer: adapter "basic" worker 7 stopping
16:27:21.057614 trace git-lfs: xfer: adapter "basic" worker 4 stopping
16:27:21.057616 trace git-lfs: xfer: adapter "basic" worker 5 stopping
16:27:21.057618 trace git-lfs: xfer: adapter "basic" worker 6 stopping
16:27:21.057613 trace git-lfs: xfer: adapter "basic" worker 3 stopping
16:27:21.057620 trace git-lfs: xfer: adapter "basic" worker 2 stopping
16:27:21.057622 trace git-lfs: xfer: adapter "basic" worker 0 stopping
16:27:21.057633 trace git-lfs: xfer: adapter "basic" stopped
16:27:21.057696 trace git-lfs: filepathfilter: accepting "src/assets/js/js9/dist/J00_20220920221935_Q2B_2snr_182_133_bdfm.fit"
Note: switching to '4fd16e7db4d7200f8f0b8b8c836ec745d5f7d825'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

 git switch -c <new-branch-name>

Or undo this operation with:

 git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 4fd16e7 Removes shapes drawn over image
16:27:21.137480 run-command.c:1523      run_processes_parallel: preparing to run up to 1 tasks
16:27:21.137485 run-command.c:655       trace: run_command: .git/hooks/post-checkout b74510e59ce476c270d63afdc0f3eb31d195b704 4fd16e7db4d7200f8f0b8b8c836ec745d5f7d825 1
16:27:21.139060 git.c:750               trace: exec: git-lfs post-checkout b74510e59ce476c270d63afdc0f3eb31d195b704 4fd16e7db4d7200f8f0b8b8c836ec745d5f7d825 1
16:27:21.139084 run-command.c:655       trace: run_command: git-lfs post-checkout b74510e59ce476c270d63afdc0f3eb31d195b704 4fd16e7db4d7200f8f0b8b8c836ec745d5f7d825 1
16:27:21.141268 trace git-lfs: exec: git 'version'
16:27:21.142227 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
16:27:21.143007 trace git-lfs: exec: git 'config' '--includes' '-l'
16:27:21.143730 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
16:27:21.144399 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' ':.lfsconfig'
16:27:21.145228 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' 'HEAD:.lfsconfig'
16:27:21.146153 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
16:27:21.146974 trace git-lfs: NewLsFiles: running in /home/valo/tmp/debuglfs/myproject-gui git ls-files -z --cached --sparse --exclude-standard --others
16:27:21.146996 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'ls-files' '-z' '--cached' '--sparse' '--exclude-standard' '--others'
16:27:21.148410 trace git-lfs: findAttributeFiles: located .gitattributes
16:27:21.148893 run-command.c:1551      run_processes_parallel: done
16:27:21.148912 run-command.c:46        trace: run_command: running exit handler for pid 203225
16:27:21.148969 trace git-lfs: filepathfilter: creating pattern ".git" of type gitignore
16:27:21.148980 trace git-lfs: filepathfilter: creating pattern "**/.git" of type gitignore
16:27:21.149011 trace git-lfs: filepathfilter: accepting "tmp"

Very verbose output obtained using GIT_TRACE_PACK_ACCESS=1 GIT_TRANSFER_TRACE=1 GIT_TRACE_SETUP=1 GIT_TRACE=1
when using the submodule and failing

130 valo@laptop ~/tmp/debuglfs % git  -c url.http://valo:password@gitlab.lan.insteadOf=http://gitlab.lan clone --recurse-submodules http://gitlab.lan/group/myproject.git
Cloning into 'myproject'...
remote: Enumerating objects: 2542, done.
remote: Counting objects: 100% (548/548), done.
remote: Compressing objects: 100% (540/540), done.
remote: Total 2542 (delta 401), reused 0 (delta 0), pack-reused 1994
Receiving objects: 100% (2542/2542), 5.61 MiB | 69.19 MiB/s, done.
Resolving deltas: 100% (1614/1614), done.
Submodule 'myproject-gui' (http://gitlab.lan/group/myproject-gui.git/) registered for path 'myproject-gui'
Cloning into '/home/valo/tmp/debuglfs/myproject/myproject-gui'...
remote: Enumerating objects: 1043, done.        
remote: Counting objects: 100% (157/157), done.        
remote: Compressing objects: 100% (155/155), done.        
remote: Total 1043 (delta 62), reused 0 (delta 0), pack-reused 886        
Receiving objects: 100% (1043/1043), 76.38 MiB | 16.74 MiB/s, done.
Resolving deltas: 100% (292/292), done.
Submodule path 'myproject-gui': checked out '0226751ed31f1b78ace52564df40986c6e22fba7'
valo@laptop ~/tmp/debuglfs % cd myproject/myproject-gui 
valo@laptop ..p/debuglfs/myproject/myproject-gui (git)-[remotes/origin/group-edits] % GIT_TRACE_PACK_ACCESS=1   GIT_TRANSFER_TRACE=1 GIT_TRACE_SETUP=1 GIT_TRACE=1 git -c url.http://valo:password@gitlab.lan.insteadOf=http://gitlab.lan checkout 4fd16e7db4d7200f8f0b8b8c836ec745d5f7d825
17:19:46.085102 trace.c:311             setup: git_dir: /home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui
17:19:46.085120 trace.c:312             setup: git_common_dir: /home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui
17:19:46.085123 trace.c:313             setup: worktree: /home/valo/tmp/debuglfs/myproject/myproject-gui
17:19:46.085125 trace.c:314             setup: cwd: /home/valo/tmp/debuglfs/myproject/myproject-gui
17:19:46.085126 trace.c:315             setup: prefix: (null)
17:19:46.085131 chdir-notify.c:65       setup: chdir from '/home/valo/tmp/debuglfs/myproject/myproject-gui' to '/home/valo/tmp/debuglfs/myproject/myproject-gui'
17:19:46.085134 git.c:460               trace: built-in: git checkout 4fd16e7db4d7200f8f0b8b8c836ec745d5f7d825
17:19:46.085366 packfile.c:1640         /home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 22403012
17:19:46.085428 packfile.c:1640         /home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 22402404
17:19:46.254305 packfile.c:1640         /home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 22405771
17:19:46.254355 packfile.c:1640         /home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 22406392
17:19:46.254360 packfile.c:1640         /home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80080786
17:19:46.254407 packfile.c:1640         /home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 22424252
17:19:46.254413 packfile.c:1640         /home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80080834
17:19:46.254430 packfile.c:1640         /home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 22426727
17:19:46.254432 packfile.c:1640         /home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80080879
17:19:46.254435 packfile.c:1640         /home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 22426860
17:19:46.254437 packfile.c:1640         /home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80081011
17:19:46.254439 packfile.c:1640         /home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 22426945
17:19:46.254442 packfile.c:1640         /home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80081096
17:19:46.260457 packfile.c:1640         /home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80080663
17:19:46.260486 packfile.c:1640         /home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80080663
17:19:46.261619 packfile.c:1640         /home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80081226
17:19:46.261634 run-command.c:655       trace: run_command: 'git-lfs filter-process'
17:19:46.264592 trace git-lfs: exec: git 'version'
17:19:46.265666 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
17:19:46.266513 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
17:19:46.267218 trace git-lfs: exec: git 'config' '--includes' '--local' 'lfs.repositoryformatversion'
17:19:46.267883 trace git-lfs: exec: git 'config' '--includes' '--replace-all' 'lfs.repositoryformatversion' '0'
17:19:46.268787 trace git-lfs: exec: git 'config' '--includes' '-l'
17:19:46.269508 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
17:19:46.270205 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' ':.lfsconfig'
17:19:46.271074 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' 'HEAD:.lfsconfig'
17:19:46.272011 trace git-lfs: Install hook: pre-push, force=false, path=/home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/hooks/pre-push
17:19:46.272045 trace git-lfs: Install hook: post-checkout, force=false, path=/home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/hooks/post-checkout
17:19:46.272066 trace git-lfs: Install hook: post-commit, force=false, path=/home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/hooks/post-commit
17:19:46.272085 trace git-lfs: Install hook: post-merge, force=false, path=/home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/hooks/post-merge
17:19:46.272102 trace git-lfs: Initialize filter-process
17:19:46.272171 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
17:19:46.272977 trace git-lfs: tq: running as batched queue, batch size of 100
17:19:46.273090 trace git-lfs: filepathfilter: accepting "src/assets/js/js9/dist/3c273.fits"
17:19:46.273137 packfile.c:1640         /home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80081348
17:19:46.273379 trace git-lfs: filepathfilter: accepting "src/assets/js/js9/dist/J00_20220920221935_Q2B_2snr_182_133_bdfm.fit"
17:19:46.273538 packfile.c:1640         /home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80081469
17:19:46.273611 trace git-lfs: filepathfilter: accepting "src/assets/js/js9/dist/coma.fits"
17:19:46.273662 packfile.c:1640         /home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui/objects/pack/pack-34b3e46eeca1f637a02e6d026728ef6521ef0659.pack 80080278
17:19:46.273764 trace git-lfs: tq: sending batch of size 3
17:19:46.273888 trace git-lfs: api: batch 3 files
17:19:46.274051 trace git-lfs: HTTP: POST http://gitlab.lan/group/myproject-gui.git/info/lfs/objects/batch
17:19:46.311830 trace git-lfs: HTTP: 401
17:19:46.311873 trace git-lfs: setting repository access to basic
17:19:46.311888 trace git-lfs: exec: git 'config' '--includes' '--replace-all' 'lfs.http://gitlab.lan/group/myproject-gui.git/info/lfs.access' 'basic'
17:19:46.312834 trace git-lfs: api: http response indicates "basic" authentication. Resubmitting...
17:19:46.312899 trace git-lfs: creds: git credential fill ("http", "gitlab.lan", "")
17:19:46.312914 trace git-lfs: exec: git 'credential' 'fill'
17:19:46.313582 trace.c:311             setup: git_dir: /home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui
17:19:46.313596 trace.c:312             setup: git_common_dir: /home/valo/tmp/debuglfs/myproject/.git/modules/myproject-gui
17:19:46.313599 trace.c:313             setup: worktree: /home/valo/tmp/debuglfs/myproject/myproject-gui
17:19:46.313601 trace.c:314             setup: cwd: /home/valo/tmp/debuglfs/myproject/myproject-gui
17:19:46.313603 trace.c:315             setup: prefix: (null)
Username for 'http://gitlab.lan': 
@svalo
Copy link
Author

svalo commented Mar 2, 2024

On my laptop I was unable to reproduce the issue today, on Monday I'll try to test it at work again. On the gitlab ticket the analysis is ongoing

@chrisd8088
Copy link
Contributor

Hey, thanks for the report, and I'm sorry you're having trouble. I don't have GitLab to test with, and so far I haven't been able to reproduce something similar to this issue (while fetching from GitHub), so I'll be interested in what you find from your ongoing investigation.

Here's what I've tried so far, which seems to work OK, meaning that url.<url>.insteadOf is honoured when fetching Git LFS objects inside a submodule. First, I create some test repositories, one containing some Git LFS objects, and the other containing that one as a submodule, but with a "broken" .gitmodules which doesn't actually point to the correct URL.

$ git clone https://github.com/chrisd8088/testsub
$ cd testsub
$ git lfs track '*.bin'
$ echo foo >foo.bin
$ echo bar >bar.bin
$ git add *.bin .gitattributes
$ git commit -m init
$ git push origin main

$ cd ..
$ git clone https://github.com/chrisd8088/test
$ cd test
$ echo foo >foo.txt
$ git add foo.txt
$ git commit -m init
$ git submodule add https://github.com/chrisd8088/testsub
$ sed -e 's/\/testsub/\/testfake/' .gitmodules >tmp
$ mv tmp .gitmodules
$ git add .gitmodules testsub
$ git commit -m sub
$ git push origin main

Then I clone the repos, including the submodule repo, but without fetching any Git LFS objects, and with a Git config rule to rewrite the broken URL to the correct one:

$ cd ..
$ rm -rf test
$ GIT_LFS_SKIP_SMUDGE=1 \
    git -c url.https://github.com/chrisd8088/testsub.insteadOf=https://github.com/chrisd8088/testfake \
    clone --recurse-submodules https://github.com/chrisd8088/test

$ cd test/testsub
$ git lfs env
git-lfs/3.4.1 (GitHub; darwin arm64; go 1.21.5)
git version 2.43.0

Endpoint=https://github.com/chrisd8088/testfake.git/info/lfs (auth=none)
...

$ cat bar.bin
version https://git-lfs.github.com/spec/v1
oid sha256:7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730
size 4

Finally, I fetch the Git LFS objects within the submodule repo, and the URL rewriting seems to work as expected:

$ GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 \
    git -c url.https://github.com/chrisd8088/testsub.insteadOf=https://github.com/chrisd8088/testfake \
    lfs fetch -a
...
> POST /chrisd8088/testsub.git/info/lfs/objects/batch HTTP/1.1
> Host: github.com
> Accept: application/vnd.git-lfs+json
> Authorization: Basic * * * * *
> Content-Length: 297

< HTTP/2.0 200 OK
...

$ cat bar.bin
version https://git-lfs.github.com/spec/v1
oid sha256:7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730
size 4
$ git lfs ls-files
7d865e959b - bar.bin
b5bb9d8014 - foo.bin

$ git lfs checkout
$ cat bar.bin
bar
$ git lfs ls-files
7d865e959b * bar.bin
b5bb9d8014 * foo.bin

Whereas, if I leave out the URL rewriting rule, the request to the Git LFS Batch API endpoint receives a 403, since the URL is incorrect.

One difference between my test setup and yours is that you are trying to inject authorization credentials into the rewritten URL. It would be interesting if you are able to reproduce the problem when trying to do that, but not when trying to rewrite the URL in some other way, as in my example above.

At any rate, keep us posted with anything you find out!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants