Improve performance when linking shared files and directories

[eloyesp]

I've noticed that linking directories and files does take a lot of time because it is made one by one and with multiple checks. I would like to work on that by running a single command on the destination (joined with &&).

Mi idea would be to add an execute_multiple to sshkit and use it to run a simplified version that requires no test:

execute_multiple :ln, "-sfb", source, target

That should make that step run much faster and should be functionally equivalent.

I wanted to know your opinion about this before I start coding.

Regards.

[leehambley]

We prefer to keep the API small and tight, there's nothing to prevent you modifying the link step dynamically to replace this with a line you constructed yourself.

I'd love to see a hotpatch that you've been running yourself, and once you have some confidence that it works well, we can definitely look at whether it has a place in Capistrano, or we can document this as a "recommended" solution to the problem if you experience slow linking.

I'd guess from your question, you're not a typical Rails app linking 2/3 directories after each deploy?

[eloyesp]

@leehambley thanks for your feedback, I've just tested it with a hotpatch on my environment and it was great.

I'd guess from your question, you're not a typical Rails app linking 2/3 directories after each deploy?

No, it is a pretty small app, and deployment just takes a bit more than a minute (that may be the reason I notice the symlinking time), the linked files and dirs are:

append :linked_dirs, 'log', 'tmp', '.bundle', 'public/system', 'node_modules'
append :linked_files, 'config/database.yml', 'Passengerfile.json'

As mentioned, I've been testing, and the results are promising:

Results

Without any changes, the symlinking went from second 25 to second 43.

Then I used multiple execute command, but ln -sf to make the test unnecessary, and with that change the time were from 20 to 27 (I could not understand why it started sooner).

Then I used a single execute line for all the links, joining everything with && and then times were 19 to 22.

The code I've used is ugly, but it does the trick:

namespace :deploy do
  namespace :symlink do

    Rake::Task['deploy:symlink:linked_dirs'].clear
    desc "Symlink linked directories"
    task linked_dirs: :fast do
    end

    Rake::Task['deploy:symlink:linked_files'].clear
    desc "Symlink linked files"
    task linked_files: :fast do
    end

    desc "Make symlinks faster"
    task :fast do
      on release_roles :all do
        execute :mkdir, "-p", linked_dir_parents(release_path) + linked_file_dirs(release_path)
        files_to_link = fetch(:linked_files) + fetch(:linked_dirs)
        execute :rm, "-rf", fetch(:linked_dirs).map{ |dir| release_path.join(dir) }
        link_commands = files_to_link.map do |file|
          target = release_path.join(file)
          source = shared_path.join(file)
          [:ln, "-sTfb", source, target]
        end 
        execute(*link_commands.product(["&&"]).flatten[0..-2])
      end
    end
  end
end

The full deployment took 1:24 before the change and 1:08 after the change.

I'll keep using it for a while, then we can see if that needs to be integrated on capistrano or not.

Regards.

[leehambley]

Thanks for coming back to us. Is it possible that you're not reusing SSH connections or something? And each call to execute is actually re-opening a connection? Performance should never be so slow like this.

[eloyesp]

I don't know, I'm on GNU/Linux, I've always used Debian derivatives, my ssh_config has the following config:

ServerAliveInterval 15
ControlMaster auto
ControlPath ~/.ssh/master-%r@%h:%p
ControlPersist 30
AddKeysToAgent yes

I will ask other people about their timings, may be there is a config missing, how are your results, can you test those changes on your setup?

[leehambley]

That actually looks sane, indeed.

Do you have rvm or something on the server that might make starting a shell slow? Do you tweak Capistrano's config to start a login shell, or an interactive shell in a way that may cause it to load dotfiles?

[eloyesp]

I've did the following test:

$ time ssh server -- echo ok
ok

real    0m2,644s
user    0m0,021s
sys     0m0,013s
$ time server -- echo ok
ok

real    0m0,432s
user    0m0,003s
sys     0m0,007s

So it seems that it can persist the connection, and the remote loads quite fast (under 1 sec). May be something is preventing ssh to persist the connection when it is called by capistrano... is there any way to debug that?

[robot-den]

Hi! I was trying to figure out why rails app deployment is so slow, and I found this topic. I can confirm that it takes ~1.5 minute to deploy empty rails 7 app, and up to ~10s to do deploy:check. Every command that Capistrano does takes at least ~1s (so deploy:symlink:linked_dirs is a first place you notice that takes too much time).

I agree that the problem is somewhere in the SSH connection. I tried to adjust ssh config with Control options but it didn't affect the timings.

No solution from me, just wanted to 'touch' the issue.

rails (7.0.4.3)
capistrano (3.17.2)
capistrano-bundler (2.1.0)
capistrano-rails (1.6.2)
capistrano-rbenv (2.2.0)