Failed deploy with a worker in the Procfile

[bradbajuz]

Hello. I'm trying to deploy a rails app that's using the GoodJob gem for queuing. I keep on getting a failed deploy when I add worker: bundle exec good_job start to my Procfile. GoodJob is running in external mode.

Error output:

remote:  !     Failure in name='port listening check': container state is not running
9ce809c75076aedd2ba479c448af21dbc28d9cdfcdbd461d020dd01ac5e12eec
remote:  !     Could not start due to 1 failed checks (web.1)
=====> Start of ordersapp container output (web.1)
       bundler: failed to load command: puma (/app/vendor/bundle/ruby/3.3.0/bin/puma)
remote:  (Errno::ENOENT)
        from /app/vendor/bundle/ruby/3.3.0/gems/puma-6.4.2/lib/puma/dsl.rb:133:in `_load_from'
        from /app/vendor/bundle/ruby/3.3.0/gems/puma-6.4.2/lib/puma/configuration.rb:239:in `block in load'
        from /app/vendor/bundle/ruby/3.3.0/gems/puma-6.4.2/lib/puma/configuration.rb:239:in `each'
        from /app/vendor/bundle/ruby/3.3.0/gems/puma-6.4.2/lib/puma/configuration.rb:239:in `load'
        from /app/vendor/bundle/ruby/3.3.0/gems/puma-6.4.2/lib/puma/launcher.rb:54:in `initialize'
        from /app/vendor/bundle/ruby/3.3.0/gems/puma-6.4.2/lib/puma/cli.rb:66:in `new'
        from /app/vendor/bundle/ruby/3.3.0/gems/puma-6.4.2/lib/puma/cli.rb:66:in `initialize'
        from /app/vendor/bundle/ruby/3.3.0/gems/puma-6.4.2/bin/puma:8:in `new'
        from /app/vendor/bundle/ruby/3.3.0/gems/puma-6.4.2/bin/puma:8:in `<top (required)>'
        from /app/vendor/bundle/ruby/3.3.0/bin/puma:25:in `load'
        from /app/vendor/bundle/ruby/3.3.0/bin/puma:25:in `<top (required)>'
        from /app/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/cli/exec.rb:58:in `load'
        from /app/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/cli/exec.rb:58:in `kernel_load'
        from /app/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/cli/exec.rb:23:in `run'
        from /app/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/cli.rb:451:in `exec'
        from /app/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
        from /app/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
        from /app/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
        from /app/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/cli.rb:34:in `dispatch'
        from /app/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
        from /app/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/cli.rb:28:in `start'
        from /app/vendor/ruby-3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.3/exe/bundle:28:in `block in <top (required)>'
        from /app/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/friendly_errors.rb:117:in `with_friendly_errors'
        from /app/vendor/ruby-3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.3/exe/bundle:20:in `<top (required)>'
        from /app/bin/bundle:3:in `load'
        from /app/bin/bundle:3:in `<main>'

Procfile:

web: bundle exec puma -C config/puma.rb
worker: bundle exec good_job start

[josegonzalez]

The process that is failing is the web command, not the worker one. Are you sure that one starts on it's own correctly? If you remove the worker entry in your Procfile, does it work?

[josegonzalez]

Just to confirm, the only actual change between starting and not starting the web process is the line in the Procfile, correct? You're not reverting code or removing any gems when you remove the line, right?

[bradbajuz]

Correct. The app deploys fine once I remove the worker line. The only way I can get GoodJob working is to change it from external mode to async mode. Async mode I don't add any worker line to the Procfile.

[josegonzalez]

Mind jumping on our discord or slack to chat about this? I feel like I'm missing something here, as unless your Procfile change is somehow impacting what gets loaded for the web process, this should work fine.

[bradbajuz]

Issue had to do with line separator of the Procfile being CRLF. Changed it to LF and deploy worked.

[josegonzalez]

This will also be fixed in Dokku 0.33.2.

[ajcollins]

@josegonzalez I have experienced a similar issue after upgrading dokku (0.33.5) and finding an existing django app deployed with git wouldn't start.

I tried to redeploy (with git push). With any additional entry in the Procfile except for the web entry, the app failed to deploy. Procfile used LF rather than CRLF (checked with xxd).

Eventually, I tried destroying and recreating the app afresh. This did not help.

With dokku trace:on I noticed that there were errors with a sigil template variable e.g. (DOKKU_APP_WORKER) was not defined (variable changed between the process types). There were spaces in the sigil command in these variables.

The earlier UPP_PROC_TYPE trace line included these additional spaces.

Editing /var/lib/dokku/core-plugins/available/nginx-vhosts/functions after line 385 as follows enabled me to deploy the app.

      while read -r line || [[ -n "$line" ]]; do
        PROC_TYPE=${line%%=*}
        # Added following line to trim trailing spaces to enable deploy
        PROC_TYPE="${PROC_TYPE%"${PROC_TYPE##*[![:space:]]}"}"
        LISTENERS="$(plugn trigger network-get-listeners "$APP" "$PROC_TYPE" | xargs)"

However, I could not scale these workers.

$ ps:scale $APP beat=1 worker=1
-----> Scaling $APP processes: beat=1 worker=1
 !     beat is not a valid process name to scale up

In /var/lib/dokku/config/ps/$APP/scale there were additional spaces visible, and an additional web entry appeared with spaces and zero instances set, for example:

web=4
worker=1
beat                                                                            =0
web                                                                             =0

I manually editing this file to remove spaces and set the instances. Running dokku ps:restart $APP allowed me to get these running.

I've not tried to look where these spaces are coming from yet, but sharing now in case it's helpful to anyone else as I'm unlikely to get a chance to look or debug further for several days.

Thank you for all the work that goes into Dokku.

[josegonzalez]

Okay I think the bug is actually from procfile-util: https://github.com/dokku/procfile-util

I'll post more here: #6577