Dokku retire service fails to acquire ps:retire lock #5432
Comments
|
When you see that, are you running any other dokku commands? I wonder if you have a ps:retire command still running in the background (there are no timeouts implemented at this time), which to me points to a funky docker issue (since docker is performing the majority of the work here, with a small amount of shell scripting on top). |
|
I'm not running any other dokku commands manually, it probably is the previous scheduled retire command. |
|
What is the output of |
|
|
Can you |
|
Removed the last three containers. Output of |
|
Alright well I guess its working now? It should only error about locks if there is a concurrent retire occurring (which could happen if a previous one takes longer than expected). Can you see if you're still getting errors? |
|
The systemd job running every five minutes is still failing intermittently. |
|
I'll add some extra logging to note if the lock already exists (which would indicate that a separate ps:retire call somehow exited without cleaning up) but otherwise, what would you like to happen in this case? |
|
I would do:
|
|
There isn't a a handle on the lock age (see gofrs/flock#62 for maybe future developments) so for now, I can only implement the first part there (which is ~good enough). |
…ved when calling ps:retire Closes #5432
|
Closing as there is a pull request open. |
Description of problem
We're seeing frequent failures of dokku-retire.service in systemd.
Server is an EC2 running Ubuntu 20.04.5 LTS
Steps to reproduce
dokku-retire.timer fires.
dokku report $APP_NAME
-----> uname: Linux prod 5.15.0-1022-aws #26~20.04.1-Ubuntu SMP Sat Oct 15 03:22:07 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
-----> memory:
total used free shared buff/cache available
Mem: 3921 527 2159 1 1234 3158
Swap: 0 0 0
-----> docker version:
Client: Docker Engine - Community
Version: 20.10.21
API version: 1.41
Go version: go1.18.7
Git commit: baeda1f
Built: Tue Oct 25 18:02:21 2022
OS/Arch: linux/amd64
Context: default
Experimental: true
-----> docker daemon info:
Client:
Context: default
Debug Mode: true
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Docker Buildx (Docker Inc., v0.9.1-docker)
scan: Docker Scan (Docker Inc., v0.21.0)
-----> git version: git version 2.25.1
-----> sigil version: 0.9.0build+bc921b7
-----> herokuish version:
herokuish: 0.5.37
buildpacks:
heroku-buildpack-multi v1.2.0
heroku-buildpack-ruby v244
heroku-buildpack-nodejs v198
heroku-buildpack-clojure v90
heroku-buildpack-python v214
heroku-buildpack-java v72
heroku-buildpack-gradle v38
heroku-buildpack-scala v94
heroku-buildpack-play v26
heroku-buildpack-php v223
heroku-buildpack-go v166
heroku-buildpack-nginx v16
buildpack-null v3
-----> dokku version: dokku version 0.28.4
-----> plugn version: plugn: 0.12.0build+3a27594
-----> dokku plugins:
00_dokku-standard 0.28.4 enabled dokku core standard plugin
20_events 0.28.4 enabled dokku core events logging plugin
app-json 0.28.4 enabled dokku core app-json plugin
apps 0.28.4 enabled dokku core apps plugin
builder 0.28.4 enabled dokku core builder plugin
builder-dockerfile 0.28.4 enabled dokku core builder-dockerfile plugin
builder-herokuish 0.28.4 enabled dokku core builder-herokuish plugin
builder-lambda 0.28.4 enabled dokku core builder-lambda plugin
builder-null 0.28.4 enabled dokku core builder-null plugin
builder-pack 0.28.4 enabled dokku core builder-pack plugin
buildpacks 0.28.4 enabled dokku core buildpacks plugin
caddy-vhosts 0.28.4 enabled dokku core caddy-vhosts plugin
certs 0.28.4 enabled dokku core certificate management plugin
checks 0.28.4 enabled dokku core checks plugin
common 0.28.4 enabled dokku core common plugin
config 0.28.4 enabled dokku core config plugin
cron 0.28.4 enabled dokku core cron plugin
docker-options 0.28.4 enabled dokku core docker-options plugin
domains 0.28.4 enabled dokku core domains plugin
enter 0.28.4 enabled dokku core enter plugin
git 0.28.4 enabled dokku core git plugin
letsencrypt 0.13.0 enabled Automated installation of let's encrypt TLS certificates
logs 0.28.4 enabled dokku core logs plugin
network 0.28.4 enabled dokku core network plugin
nginx-vhosts 0.28.4 enabled dokku core nginx-vhosts plugin
plugin 0.28.4 enabled dokku core plugin plugin
postgres 1.19.3 enabled dokku postgres service plugin
proxy 0.28.4 enabled dokku core proxy plugin
ps 0.28.4 enabled dokku core ps plugin
registry 0.28.4 enabled dokku core registry plugin
repo 0.28.4 enabled dokku core repo plugin
resource 0.28.4 enabled dokku core resource plugin
run 0.28.4 enabled dokku core run plugin
scheduler 0.28.4 enabled dokku core scheduler plugin
scheduler-docker-local 0.28.4 enabled dokku core scheduler-docker-local plugin
scheduler-null 0.28.4 enabled dokku core scheduler-null plugin
shell 0.28.4 enabled dokku core shell plugin
ssh-keys 0.28.4 enabled dokku core ssh-keys plugin
storage 0.28.4 enabled dokku core storage plugin
trace 0.28.4 enabled dokku core trace plugin
traefik-vhosts 0.28.4 enabled dokku core traefik-vhosts plugin
=====> portal app-json information
App json computed selected: app.json
App json global selected: app.json
App json selected:
=====> portal app information
App created at: 1666907959
App deploy source: portal
App deploy source metadata: portal
App dir: /home/dokku/portal
App locked: false
=====> portal builder information
Builder build dir:
Builder computed build dir:
Builder computed selected:
Builder global build dir:
Builder global selected:
Builder selected:
=====> portal builder-dockerfile information
Builder dockerfile computed dockerfile path: Dockerfile
Builder dockerfile global dockerfile path: Dockerfile
Builder dockerfile dockerfile path:
=====> portal builder-lambda information
Builder lambda computed lambdayml path: lambda.yml
Builder lambda global lambdayml path: lambda.yml
Builder lambda lambdayml path:
=====> portal builder-pack information
Builder pack computed projecttoml path: project.toml
Builder pack global projecttoml path: project.toml
Builder pack projecttoml path:
=====> portal buildpacks information
Buildpacks computed stack: gliderlabs/herokuish:latest-20
Buildpacks global stack:
Buildpacks list:
Buildpacks stack:
=====> portal ssl information
Ssl dir: /home/dokku/portal/tls
Ssl enabled: true
Ssl hostnames: *.XXXXXX XXXXXX
Ssl expires at: Jan 11 22:20:08 2023 GMT
Ssl issuer: C = US, O = Let's Encrypt, CN = R3
Ssl starts at: Oct 13 22:20:09 2022 GMT
Ssl subject: subject=CN = *.XXXXXX
Ssl verified: self signed
=====> portal checks information
Checks disabled list: none
Checks skipped list: none
=====> portal cron information
Cron task count: 0
=====> portal docker options information
Docker options build:
Docker options deploy: --restart=on-failure:10
Docker options run:
=====> portal domains information
Domains app enabled: true
Domains app vhosts: portal.XXXXXX
Domains global enabled: true
Domains global vhosts: XXXXXX
=====> portal git information
Git deploy branch: master
Git global deploy branch: master
Git keep git dir: false
Git rev env var: GIT_REV
Git sha: 1acea54
Git last updated at: 1665693701
=====> portal logs information
Logs computed max size: 10m
Logs global max size: 10m
Logs global vector sink:
Logs max size:
Logs vector sink:
=====> portal network information
Network attach post create:
Network attach post deploy:
Network bind all interfaces: false
Network computed attach post create:
Network computed attach post deploy:
Network computed bind all interfaces: false
Network computed initial network:
Network computed tld:
Network global attach post create:
Network global attach post deploy:
Network global bind all interfaces: false
Network global initial network:
Network global tld:
Network initial network:
Network static web listener:
Network tld:
Network web listeners: 172.17.0.3:5000
=====> portal nginx information
Nginx access log format:
Nginx access log path: /var/log/nginx/portal-access.log
Nginx bind address ipv4:
Nginx bind address ipv6: ::
Nginx client max body size:
Nginx disable custom config: false
Nginx error log path: /var/log/nginx/portal-error.log
Nginx global hsts: true
Nginx computed hsts: true
Nginx hsts:
Nginx hsts include subdomains: true
Nginx hsts max age: 15724800
Nginx hsts preload: false
Nginx proxy buffer size: 4096
Nginx proxy buffering: on
Nginx proxy buffers: 8 4096
Nginx proxy busy buffers size: 8192
Nginx proxy read timeout: 60s
Nginx last visited at: 1666977368
Nginx x forwarded for value: $remote_addr
Nginx x forwarded port value: $server_port
Nginx x forwarded proto value: $scheme
Nginx x forwarded ssl:
=====> portal proxy information
Proxy enabled: true
Proxy port map: http:80:5000 https:443:5000
Proxy type: nginx
=====> portal ps information
Deployed: true
Processes: 1
Ps can scale: true
Ps computed procfile path: Procfile
Ps global procfile path: Procfile
Ps procfile path:
Ps restart policy: on-failure:10
Restore: true
Running: true
Status web 1: running (CID: a0219d12b15)
=====> portal registry information
Registry computed image repo: dokku/portal
Registry computed push on release: false
Registry computed server:
Registry global push on release:
Registry global server:
Registry image repo:
Registry push on release:
Registry server:
Registry tag version:
=====> portal resource information
=====> portal scheduler information
Scheduler computed selected: docker-local
Scheduler global selected: docker-local
Scheduler selected:
=====> portal scheduler-docker-local information
Scheduler docker local disable chown:
Scheduler docker local init process: true
Scheduler docker local parallel schedule count:
=====> portal storage information
Storage build mounts:
Storage deploy mounts:
Storage run mounts:
Additional information
No response
Output of failing commands after running: dokku trace:off
Output of failing commands after running: dokku trace:on
The text was updated successfully, but these errors were encountered: