-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
deploy gets stuck and doesn't print anything #2066
Comments
I would suggest you reopen this on I'll leave this issue open for now, but I don't think we can assist really. |
Yes @anyelopetit please follow up with us when you track down what was causing the freeze. There may be something capistrano can do better in terms of error messaging. My assumption is that set :connection_timeout, 5 We probably can't help you fix the reason for the timeout (that is likely due to a firewall between you and the remote host, or an incorrect port number). But we may be able to improve capistrano to provide better error messaging, or maybe a more reasonable default timeout so capistrano doesn't appear to hang indefinitely. |
Hey guys! @leehambley @mattbrictson Thank you for answering so quickly! set :connection_timeout, 5 but it doesn't change anything. Do you have any idea about what is causing that error? How can I find out the reason why it gets stuck? |
Sorry I don't know what would cause this. My first test would be to try using Perhaps the hang is because ssh is waiting for something on stdin? Like asking for a password? Make sure you have the necessary public/private key and ssh-agent setup so that you aren't challenged for a passphrase when authenticating. There is information about that here: https://capistranorb.com/documentation/getting-started/authentication-and-authorisation/ |
Hi @mattbrictson! I just verified that I can access to the server specified in |
You can try to look at https://capistranorb.com/documentation/faq/why-does-something-work-in-my-ssh-session-but-not-in-capistrano/ and try to simulate more and more of what the |
To be clear, are you using https://github.com/rvm/rvm1-capistrano3 or capistrano/rvm#83 or something else? |
@leehambley I'm using group :development do
gem 'capistrano'
gem 'capistrano-bundler'
gem 'capistrano-passenger'
gem 'capistrano-rails'
gem 'capistrano-rbenv', github: 'capistrano/rbenv'
gem 'capistrano-sidekiq'
end |
Hi guys! @mattbrictson @leehambley I was trying what is in https://capistranorb.com/documentation/faq/why-does-something-work-in-my-ssh-session-but-not-in-capistrano/ and I could notice this: In staging, my shell is Interactive and Not Login, but in production my shell is Interactive and Login. So, I tried running this commands in staging: me@localhost $ ssh user@staging_ip "[[ $- == *i* ]] && echo 'Interactive' || echo 'Not interactive'"
Interactive
me@localhost $ ssh user@staging_ip "shopt -q login_shell && echo 'Login shell' || echo 'Not login shell'"
Not login shell It works! But I tried in production: me@localhost $ ssh user@production_ip "[[ $- == *i* ]] && echo 'Interactive' || echo 'Not interactive'"
# NO ANSWER, it keeps waiting for something
me@localhost $ ssh user@production_ip "shopt -q login_shell && echo 'Login shell' || echo 'Not login shell'"
# SAME HERE, NO ANSWER How can I change my shell to a Not Login shell? |
Great bit of research, I'm not sure how that can change the behaviour of rbenv much, but at least we have something to work on. I'd suggest to check with the people who configured the servers, there's a bunch of settings on the SSH daemon, and the default shell that would change this behaviour. What do you know about the configurations of both environment's servers? |
A "log-in" shell, by the way isn't necessarily the problem, it just means that the server has been configured to treat connections like a log-in shell, meaning loading your "dot files" if any.. it's not impossible that one of them is acting weird when you connect without a tty (without a terminal) |
I have the same issue. But I can catch it only using one internet provider. In another, all works perfectly. I'm not sure if this info can help you |
I have the same problem. Deploy get stuck on macOS. In my case, these are 2 different fixes:
|
I suspect something is blocking waiting for input that you are not able to provide. If some part of the deploy is .. say, requesting a password, maybe on Linux it works "by accident" for you because of some combination of shared environment vars, something in your ruby env. Can you clarify please @mamantoha what you mean "another provider", like internet provider?! |
@leehambley yes, internet provider. |
I've also noticed this at the end of all my tasks and it's waiting for the connection pool to close. When I have Activity Monitor open while running tasks, I notice a ruby task starts followed by three SSH tasks (for my servers) and then another 3 open and then they never close. Tried restarting my machine and same things happens. Strange |
We are facing a similar issue. Deploy runs fine on Linux on all servers. However, on MacOS with docker, the deploy runs fine on our production server but not on the staging server. The deployment is stuck at:
I can see on the server that the pre-compilation is running but Capistrano is stuck. However, when it is run on a Linux machine or a CI server, it works fine.
In all scenarios, Capistrano is running from inside a Docker container. The problem is only on MacOS and for only one server. Exact same Capistrano config and setup works for production deployments on MacOS. I'm not sure on how to continue debugging this further and I would be very happy if someone could help and point me into the right direction. Additionally, if I run Capistrano only for |
There is a chance this could be an SSH or network timeout issue. See the keep-alive solutions discussed in capistrano/sshkit#503. Edit: fixed issue link |
Don't know if this is still an issue for people but it is for me. Findings:
|
@stoicAlchemist sorry to hear you are running into this. Have you tried tweaking the SSHKit connection pool? I would try disabling connection pooling entirely and seeing if that makes a difference. SSHKit::Backend::Netssh.pool.idle_timeout = 0 # disabled https://github.com/capistrano/sshkit?tab=readme-ov-file#connection-pooling Also, you could check your |
@mattbrictson thanks for the fast response. Using the pool config you mentioned got the problem worst 😅 , executing the first task it got just "frozen" and didn't go through. I'm trying to deploy to our QA environment and it just hangs:
This is the output using the Without the config, the log file ends on:
** I had to anonymize user and project name, other than that, all is current output It seems that it is hanging after the |
@stoicAlchemist that seems to suggest that closing the connection is indeed the problem. With the pool enabled, a single connection is reused for the entire deploy, then closed at the end. With the pool disabled, it closes the connection after each step. And sure enough, the hang happens right after the first step. I am a bit stumped, but if I were to make a wild guess, I would say that there is something in the shell init script (e.g. |
That sounds perfect, I'll start debugging that part on the server, might take some time as I may need a hand from my OPS team. Will update here with results. |
@stoicAlchemist Did you ever find a root cause for this? I'm experiencing it now and can't find the cause. |
For me, we were using a bastion host so this was part of our
Removing the
Best I could tell, it was preventing the connection pool from closing and thus completing the command. |
This makes a lot of sense. I'm using EC2 connect to avoid a bastion host, but there's still an underlying hop. I'll look into this. Thank you so much, @JonTheStone. |
Good luck! |
The proxy command for an EC2 connection is |
Quickfix to address this for EC2 Instance Connect EndpointFor those who are using (or considering) the newer EC2 Instance Connect Endpoints instead of bastion hosts, I wanted to provide details on an issue relating to this as well as a workaround. BackgroundEC2 Instance Connect allows you to connect to a private EC2 instance using a local AWS CLI command
Where [INSTANCE] is the instance-id of our EC2 server. Our SetupOur keys and users were already managed for Capistrano, so we simplified this to adding the following
Allowing us to have a
The ProblemThis worked, but I immediately saw the behavior in this ticket. The deployment would hang following the As noted in this comment, setting After some extensive debugging, I figured out that the What's notable about this is that Less a solution than a hackI found that if I manually
Note, this is a pretty rough hack, and that task can be fleshed out quite a bit to make it more robust, but this gives |
Steps to reproduce
cap production deploy
and that's it, just gets stuck forever until I abort it
Expected behavior
Actual behavior
When I run:
cap production deploy
it shows nothing, just keeps waitingBut when I run:
cap production deploy --trace
it shows this:System configuration
https://gist.github.com/anyelopetit/7320fc8141554ea4b75ec88bb02f8132
The text was updated successfully, but these errors were encountered: