Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

git stdout: GitLab: The project you were looking for could not be found #2037

Open
jedrekdomanski opened this issue Sep 14, 2019 · 13 comments
Open

Comments

@jedrekdomanski
Copy link

Important: GitHub issues are for feature requests or bug reports. The Capistrano team recommends you use Stack Overflow for general questions. For more details, please see our contribution policy.


Steps to reproduce

Deploy an application to a remote host, change app repo url in Git, update repo url in Capistrano config, try to deploy the app. It won't deploy. You'll get an error (see below). To resovle the problem you need to ssh onto the remote host and remove repo folder, then re-deploy. It looks like Capistrano relies on repo url from repo folder. I am posting this thread as a suggestion for improvement, because I think that it would be nice if we didn't have to remove repo folder manually, secondly, I struggled to figured it out for an hour or two so we could save other people's lives trying to solve similar problem.

Expected behavior

App should deploy

Actual behavior

git exit status: 128 (SSHKit::Runner::ExecuteError)
git stdout: GitLab: The project you were looking for could not be found.
fatal: Could not read from remote repository.

System configuration

Please link to the output of cap <stage> doctor in a GitHub Gist.
https://gist.github.com/jedrekdomanski/64e7ee0f4bb57c9631cbdbb60e3ca792

Thanks for helping improve Capistrano!

@mattbrictson
Copy link
Member

Thanks for the detailed report!

Capistrano should automatically update the remote git URL if it changes. This feature was added in #1826 . Do you have any idea why git remote set-url origin <url> is not working in your case? Is there evidence in the log output that the set-url is getting called?

@jedrekdomanski
Copy link
Author

jedrekdomanski commented Sep 14, 2019

Let me give you a bit more details about my case. The code was pushed to GitLab but capistrano config was still referencing the old repo (this a takeover project). The code was deployed. Then, I changed the repo url because I realized that my predecessors forgot to update capistrano repo url which should be pointing to the GitLab repo, not their own gitlab instance. I updated the repo url in capistrano and tried to deploy but it errored out. I looked at repo folder on the remote host and the repo url was from before my update. I do not have the logs anymore, sorry. This interesting, because, as you mentioned, the change has been already implemented but something is still not working. The case described here #1826 is the same as mine so ...very interesting...

@mattbrictson
Copy link
Member

I just tested this and it seems that capistrano is doing the right thing. I did a cap production deploy, then updated deploy.rb and changed the :git_url, then did another cap production deploy. It succeeded and after the fact I checked the repo folder on the remote host and repo/config contained the new URL.

I did see this warning in the git output after changing the URL:

warning: no common commits

But otherwise it seemed to work fine.

Perhaps in your case it had something to do with cached credentials? Was the previous URL an https one?

I'll leave this ticket open in case other people run into a similar problem and have additional information to report.

@jedrekdomanski
Copy link
Author

No, definitely nothing related to cached credentials or https. Thanks for checking!

@greenclaw
Copy link

greenclaw commented Sep 17, 2019

Hey,
I also encountered this problem in the past.
The original issue I've posted on stackoverflow: issue link.
As noted in issue, the problem solved by removing content of repo folder and scripts in /tmp folder.

@mattbrictson
Copy link
Member

Full disclosure here: I am officially stumped on this one. Sorry I can't be of help!

I am particularly confused as to why deleting scripts in /tmp would have any effect. You can see that capistrano replaces the /tmp script every time it is run, so any existing script should not make a difference:

execute :mkdir, "-p", File.dirname(fetch(:git_wrapper_path)).shellescape
upload! StringIO.new("#!/bin/sh -e\nexec /usr/bin/ssh -o PasswordAuthentication=no -o StrictHostKeyChecking=no \"$@\"\n"), fetch(:git_wrapper_path)
execute :chmod, "700", fetch(:git_wrapper_path).shellescape

Furthermore, the git:check task does not use the repo folder at all. As you can verify in the output, it is a git ls-remote command. It does not cd into the repo folder, nor does it reference the repo folder in any way.

git :'ls-remote', git_repo_url, "HEAD"

I do not understand how deleting repo or /tmp would have any effect on this task.

Perhaps someone with more expertise in git or GitLab specifically can help me here?

@bfcoder
Copy link

bfcoder commented Jan 13, 2020

For me, it doesn't seem to have anything to do with the remote. On my Mac machine, I can deploy without problems. But on my Linux (Ubuntu) machine, I am all of a sudden getting this error.

I did recently install ClamAV, so I thought maybe that was interfering. But turning it completely off, I still get this error. ¯_(ツ)_/¯

@bfcoder
Copy link

bfcoder commented Jan 13, 2020

Hello I.T., Did you try turning it off and on again?

Just did, and now I am able to deploy from my Linux machine. There must have been a bad git ssh cache somehow on my machine.

@mattbrictson
Copy link
Member

For me, it doesn't seem to have anything to do with the remote. On my Mac machine, I can deploy without problems. But on my Linux (Ubuntu) machine, I am all of a sudden getting this error.

Hmm, in that case this could be related to the private key you are using on your local machine for SSH. If that key is not associated with the correct GitLab account, I can see how that might give you a "The project you were looking for could not be found". Maybe?

@leehambley
Copy link
Member

Definitely plausible @mattbrictson I was thinking the same, since if you have multiple keys loaded into your agent (whether you know it or not, check with $ ssh-add -l) then the likes of GitHub and GitLab will let in the first key that authenticates, but it might not be a key with access to the project you want.

FWIW they'll also only try a few keys, I think up to 3..5 before giving up. Working as a contractor I have to be super careful which keys are in my agent else I can run into this kind of thing (not necessarily just with Capsitrano)

@bfcoder
Copy link

bfcoder commented Jan 13, 2020

My ssh key was loaded. I was able to run the git ls-remote --heads git@gitlab.com:project/project.git manually just fine. And I only had 1 key loaded at the time, so it wasn't a problem with running through the first 5 keys keys loaded and then giving up.

@bfcoder
Copy link

bfcoder commented Jan 27, 2020

I just replicated this.

Capistrano 3.6.1
Ubuntu 18.04
git 2.17.1

I had 2 ssh keys loaded, neither were the correct key for the server I was attempting to deploy to. Received:

#<Thread:0x0000562c62bdca10@/home/<username>/.asdf/installs/ruby/2.5.7/lib/ruby/gems/2.5.0/gems/sshkit-1.17.0/lib/sshkit/runners/parallel.rb:10 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
  1: from /home/<username>/.asdf/installs/ruby/2.5.7/lib/ruby/gems/2.5.0/gems/sshkit-1.17.0/lib/sshkit/runners/parallel.rb:11:in `block (2 levels) in execute'
/home/<username>/.asdf/installs/ruby/2.5.7/lib/ruby/gems/2.5.0/gems/sshkit-1.17.0/lib/sshkit/runners/parallel.rb:15:in `rescue in block (2 levels) in execute': Exception while executing as ubuntu@<dns>.compute.amazonaws.com: git exit status: 128 (SSHKit::Runner::ExecuteError)
git stdout: Nothing written
git stderr: remote: 
remote: ========================================================================
remote: 
remote: The project you were looking for could not be found.
remote: 
remote: ========================================================================
remote: 
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
(Backtrace restricted to imported tasks)
cap aborted!

After ssh-add the correct key to login to the machine, I still get this error. Which is odd since it isn't the key that is needed to authenticate with gitlab. That key was loaded(albeit it was loaded manually after cancelling the passphrase prompt, and then manually running ssh-add, earlier in the day when trying to git pull). I also got this error message if I did load the server key, but not the gitlab key, cancelled out of the passphrase prompt, manually ssh-add the gitlab key, then try to deploy. It holds onto that original denial.

If I reboot my machine (I'm sure there is a service I could restart too), ssh-add both my gitlab key, and my server key, I can then cap deploy just fine.

And maybe this is actually a problem with git and it is just manifesting here while using cap. I don't really know.

@leehambley
Copy link
Member

leehambley commented Jan 28, 2020

Thanks for commenting back @bfcoder

This is indeed a problem with Git. You can modify the git+ssh wrapper script that Capistrano exports to expose the following env vars GIT_CURL_VERBOSE=1 GIT_SSH_COMMAND="ssh -v" GIT_TRACE=1 (or export them in your shell) and you can see exactly what Git is doing under the hood.

Git is written as a series of tiny tools that get piped together, so on the simplest level there's going to be a call to git ls-remote <the remote> which is expected to show what commit the remote is at, followed by some calls to something like git receive-pack <the remote> ....0xffff -- git receive pack is usually "wired" with a pipe to something that reads from SSH.

I'm recounting this a lot from memory, so your results may vary, a bit. What may be happening is that with the "wrong" keys loaded into the agent is that Git is using up some of it's retries using the wrong keys, and proceeding on with no "retries" left on the counter, so a single failure (even if you should be allowed to try n keys z times) is enough to make it bail out.

You might also experiment with GIT_SSH_COMMAND="ssh -vvv" rather than just one -v.

Regarding clearing/rebooting, an ssh-add -D ( Deletes all identities from the agent.) should be sufficient to get to a "clean" state.

When using Capistrano on the server-side, we write out a git wrapper script (which is used in place of which ssh, which allows us to pass options). It's not uncommon to do this in environments where you want to set SSH options to specify which identity file should be used, to skip, or pre-approve SSH host keys, to change logging levels, etc, etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants