Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PIDFILE contains only a single PID after starting multiple workers #1721

Open
z0marlin opened this issue May 14, 2020 · 4 comments · May be fixed by #1733
Open

PIDFILE contains only a single PID after starting multiple workers #1721

z0marlin opened this issue May 14, 2020 · 4 comments · May be fixed by #1733

Comments

@z0marlin
Copy link

When starting multiple workers using rake task resque:workers and the PIDFILE env variable, only a single pid is available in the specified file, although multiple workers were started.
@tjozwik
Copy link

tjozwik commented Sep 10, 2020

@z0marlin it is caused because in fact resque:workers calls resque:worker task COUNTER times. Besides, I think that there is a race condition for the PIDFILE access.

@z0marlin
Copy link
Author

True. What could be a solution to this? I think appending to the file instead of overwriting and having a file lock should solve the problem. The file can be truncated initially to remove PIDs of the previous execution.

@tjozwik
Copy link

tjozwik commented Sep 26, 2020

Yeah, that sounds great for me. Additionally, there will be a resque:worker task's refactor required as we would need to truncate the file for single-worker requests too. So, in general I think we can fix it in such a way:

  1. Rename resque:worker task to e.g. resque:run,
  2. Create resque:worker task that will be truncating PIDFILE (if env variable is defined, maybe using additional task for truncating) and calling resque:run once,
  3. Change resque:workers task to truncate PIDFILE and use resque:run instead resque:worker,
  4. Change prepare method in lib/resque/worker.rb so as to lock PIDFILE (looks like File#flock) and append current worker PID instead of overwriting it.

Unfortunately, I am afraid I don't have a time for creating PR for this issue right now/soon.

@z0marlin
Copy link
Author

No problem. Thanks for the describing the changes required. I'll start working on the patch.

z0marlin added a commit to z0marlin/resque that referenced this issue Nov 6, 2020
@z0marlin
Ensure PIDFILE has pids of all running workers
8829443 
The workers now append to the PIDFILE instead of overwriting it.
Adding file locking to handle race conditions when multiple
workers are started using resque:workers.

To clear any older data in the specified PIDFILE, it is now truncated
once when the rake tasks resque:work and resque:workers are called.
Created a proxy task resque:run to achive this.

Fixes resque#1721
z0marlin added a commit to z0marlin/resque that referenced this issue Nov 6, 2020
@z0marlin
Ensure PIDFILE has pids of all running workers
a3f624f 
The workers now append to the PIDFILE instead of overwriting it.
Adding file locking to handle race conditions when multiple
workers are started using resque:workers.

To clear any older data in the specified PIDFILE, it is now truncated
once when the rake tasks resque:work and resque:workers are called.
Created a proxy task resque:run to achive this.

Fixes resque#1721
@z0marlin z0marlin linked a pull request Nov 7, 2020 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Ensure PIDFILE has pids of all running workers z0marlin/resque
2 participants
@tjozwik @z0marlin