Fix bundle install when older revisions of git source

[krororo]

What was the end-user or developer problem that led to this PR?

Fixes #6929.

bundle install may fail if an old revision is specified for a gem of git source.
At that time, the following error occurs.

Git error: command `git reset --hard 41982d4703e4a89eaeaa1c90ae4982ff574e48eb` in directory
/tmp/path/to/app/vendor/bundle/ruby/3.1.0/bundler/gems/foo-41982d4703e4 has failed.
fatal: Could not parse object '41982d4703e4a89eaeaa1c90ae4982ff574e48eb'.

What is your fix for the problem, implemented in this PR?

This implementation is to execute fetch after copying to the installation directory. Because the target commit may exist in the repository in cache, but not in the destination repository.

Make sure the following tasks are checked

• Describe the problem / feature
• Write tests for features and bug fixes
• Write code to solve the problem
• Make sure you follow the current code style and write meaningful commit messages without tags

[welcome]

Thanks for opening a pull request and helping make RubyGems and Bundler better! Someone from the RubyGems team will take a look at your pull request shortly and leave any feedback. Please make sure that your pull request has tests for any changes or added functionality.

We use GitHub Actions to test and make sure your change works functionally and uses acceptable conventions, you can review the current progress of GitHub Actions in the PR status window below.

If you have any questions or concerns that you wish to ask, feel free to leave a comment in this PR or join our #rubygems or #bundler channel on Slack.

For more information about contributing to the RubyGems project feel free to review our CONTRIBUTING guide

[deivid-rodriguez]

Looks great, just a small comment!

[igorpeshansky]

This seems to break with really old versions of git, e.g.:

$ git --version
git version 2.13.7
$ grep omnibus-software Gemfile
gem 'omnibus-software', :github => 'chef/omnibus-software'
$ bundle install
Git error: command `git fetch --force --quiet /tmp/path/to/app/vendor/bundle/ruby/3.1.0/cache/bundler/git/omnibus-software-338444b7bc50110d52bb35857ad8f946482db0ef --depth 1 4fef36786ead96e7f2f23e6f01f09efb71894992` in directory /tmp/path/to/app/vendor/bundle/ruby/3.1.0/bundler/gems/omnibus-software-4fef36786ead has failed.
error: Server does not allow request for unadvertised object 4fef36786ead96e7f2f23e6f01f09efb71894992

One fix is to set git config uploadpack.allowAnySHA1InWant true in bundler's git cache repositories, but I'm not sure how to make bundler do that… My attempts to add

          git "config", "uploadpack.allowAnySHA1InWant", "true", :dir => destination if ref

here didn't seem to actually change the cache repository config. If someone can point me at the right place in the code, I can send a PR.

Environment

Bundler             2.4.21
  Platforms         ruby, x86_64-linux
Ruby                3.1.2p20 (2022-04-12 revision 4491bb740a9506d76391ac44bb2fe6e483fec952) [x86_64-linux]
  Full Path         /usr/local/rvm/rubies/ruby-3.1.2/bin/ruby
  Config Dir        /usr/local/rvm/rubies/ruby-3.1.2/etc
RubyGems            3.3.7
  Gem Home          /usr/local/rvm/gems/ruby-3.1.2
  Gem Path          /usr/local/rvm/gems/ruby-3.1.2:/usr/local/rvm/rubies/ruby-3.1.2/lib/ruby/gems/3.1.0
  User Home         …
  User Path         …/.local/share/gem/ruby/3.1.0
  Bin Dir           /usr/local/rvm/gems/ruby-3.1.2/bin
Tools               
  Git               2.13.7
  RVM               1.29.12 (latest)
  rbenv             not installed
  chruby            not installed
  rubygems-bundler  (1.4.5)

Bundler Build Metadata

Built At          2023-10-17
Git SHA           d10b46bd15
Released Version  true

[deivid-rodriguez]

Hei! Interesting!

So this used to work for you and only broke after the latest release including this PR?

My understanding is that uploadpack.allowAnySHA1InWant is a server setting and most git servers have it set. Github should definitely allow it. So this would seem unrelated to git version to me. Did you verify it works with newer git?

I googled the error and found this thread which points to submodule issues? Is a git submodule involved here?

[bkabrda]

👋 I just wanted to report we're seeing the similar happen when using https://github.com/DataDog/omnibus-software as a git source in our Gemfile (no git submoduled involved). We see this with git 2.7 and 2.10, but not with 2.40.

[deivid-rodriguez]

Thank you, since that's an open source repo, it should be easily reproducible, right? What do I need to do to see the issue for myself?

[deivid-rodriguez]

Since this sounds fixed in git a fix for us would be a matter of reverting this change for older git versions. But it'd be nice to bisect which particular git version started behaving just fine.

[igorpeshansky]

I'm in the same boat as @bkabrda — no submodules involved, also erroring out on an omnibus-software clone. FWIW, I was looking at https://stackoverflow.com/a/61492323, referenced from the thread you posted. The setting needs to be on the local clone of the repo, which is in the cache (/tmp/path/to/app/vendor/bundle/ruby/3.1.0/cache/bundler/git/<gem>-<hash>). I've verified this by manually setting uploadpack.allowAnySHA1InWant in the cache directory, which allowed bundle install to proceed. I just couldn't figure out how to do this automatically. I'll respond soon with repro instructions.

Unfortunately, upgrading git is not an option for us — we build on a variety of older platforms in Docker containers, and we're stuck with the system git.

[igorpeshansky]

Repro instructions

Save the following files in a new directory:

6980.Dockerfile:

FROM opensuse/leap:42.3 as base
RUN zypper -n install git curl which tar
RUN mv /var/lib/ca-certificates/pem/DST_Root_CA_X3.pem /etc/pki/trust/blacklist/ \
 && update-ca-certificates
RUN gpg2 --keyserver hkp://keyserver.ubuntu.com --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
RUN curl -sSL -o get-rvm-io.sh https://get.rvm.io \
 && /bin/bash get-rvm-io.sh stable
RUN ["/bin/bash", "-lc", "rvm requirements && rvm install 3.1.2"]

ADD "https://www.random.org/cgi-bin/randbyte?nbytes=10&format=h" skipcache

ARG BUNDLER_VERSION=2.4.21
RUN ["/bin/bash", "-lc", "gem install bundler --no-document --version ${BUNDLER_VERSION}"]
COPY 6980.rb /6980.rb
RUN ["/bin/bash", "-lc", "rvm use 3.1.2 && bundle --version && ruby /6980.rb"]

6980.rb:

require 'tmpdir'

Dir.mktmpdir do |dir|
  app_path = "#{dir}/app"
  Dir.mkdir(app_path)
  File.write("#{app_path}/Gemfile", <<~GEMFILE)
    source 'https://rubygems.org'
    gem 'omnibus-software', :github => 'chef/omnibus-software'
  GEMFILE
  Dir.chdir(app_path) do
    system 'bundle config --local path vendor/bundle'
    system 'bundle config --local clean true'
  end

  Dir.chdir(app_path) do
    system 'bundle install' # => Git error
  end
end

Invoke as follows:

docker build -f 6980.Dockerfile --build-arg BUNDLER_VERSION=2.4.21 .

(or just docker build -f 6980.Dockerfile .).
Observe the git error in the output:

Git error: command `git fetch --force --quiet /tmp/d20231019-204-elytvg/app/vendor/bundle/ruby/3.1.0/cache/bundler/git/omnibus-software-338444b7bc50110d52bb35857ad8f946482db0ef --depth 1 4fef36786ead96e7f2f23e6f01f09efb71894992` in directory /tmp/d20231019-204-elytvg/app/vendor/bundle/ruby/3.1.0/bundler/gems/omnibus-software-4fef36786ead has failed.
error: Server does not allow request for unadvertised object 4fef36786ead96e7f2f23e6f01f09efb71894992

Invoke to see it succeed with bundler 2.4.20:

docker build -f 6980.Dockerfile --build-arg BUNDLER_VERSION=2.4.20 .

Note

To see the effect of uploadpack.allowAnySHA1InWant, use this instead:

6980.rb:

require 'tmpdir'

Dir.mktmpdir do |dir|
  app_path = "#{dir}/app"
  Dir.mkdir(app_path)
  File.write("#{app_path}/Gemfile", <<~GEMFILE)
    source 'https://rubygems.org'
    gem 'omnibus-software', :github => 'chef/omnibus-software'
  GEMFILE
  Dir.chdir(app_path) do
    system 'bundle config --local path vendor/bundle'
    system 'bundle config --local clean true'
  end

  Dir.chdir(app_path) do
    system 'bundle install' # => Git error
  end

  Dir.chdir("#{app_path}/vendor/bundle/ruby/3.1.0/cache/bundler/git/omnibus-software-338444b7bc50110d52bb35857ad8f946482db0ef") do
    system 'git config uploadpack.allowAnySHA1InWant true'
  end

  Dir.chdir(app_path) do
    system 'bundle install' # => Success
  end
end

@deivid-rodriguez

[deivid-rodriguez]

Nice investigation and repro @igorpeshansky 😍

I think you are basically on the right track but you were running the git command in the wrong folder, it should be:

git "config", "uploadpack.allowAnySHA1InWant", "true", :dir => path.to_s

Besides that, this setting is not the default in git, so I'd feel more confortable keeping it like that on git versions without this bug. So I'd go with something like:

diff --git a/bundler/lib/bundler/source/git/git_proxy.rb b/bundler/lib/bundler/source/git/git_proxy.rb
index 93625145c..0b507abc7 100644
--- a/bundler/lib/bundler/source/git/git_proxy.rb
+++ b/bundler/lib/bundler/source/git/git_proxy.rb
@@ -131,7 +131,11 @@ def copy_to(destination, submodules = false)
           end
 
           ref = @commit_ref || (locked_to_full_sha? && @revision)
-          git "fetch", "--force", "--quiet", *extra_fetch_args(ref), :dir => destination if ref
+          if ref
+            git "config", "uploadpack.allowAnySHA1InWant", "true", :dir => path.to_s if @commit_ref.nil? && needs_allow_any_sha1_in_want?
+
+            git "fetch", "--force", "--quiet", *extra_fetch_args(ref), :dir => destination
+          end
 
           git "reset", "--hard", @revision, :dir => destination
 
@@ -434,6 +438,10 @@ def supports_minus_c?
           @supports_minus_c ||= Gem::Version.new(version) >= Gem::Version.new("1.8.5")
         end
 
+        def needs_allow_any_sha1_in_want?
+          @needs_allow_any_sha1_in_want ||= Gem::Version.new(version) <= Gem::Version.new("2.13.7")
+        end
+
         def supports_fetching_unreachable_refs?
           @supports_fetching_unreachable_refs ||= Gem::Version.new(version) >= Gem::Version.new("2.5.0")
         end

I'm not sure which git version fixed this bug, but I guess we workaround it for everyone who has noticed it here and move on, that's why I set 2.13.7. We could also do a proper bisection by manually building from source different git versions, or try to rescue the error message and retry configuring that setting right before, but feels overkill to me.