[💡 FEATURE REQUEST]: Option to limit the concurrency of the JOBS drivers

[rustatian]

Plugin

JOBS

I have an idea!

A new option that can limit the parallelism per pipeline. For example, if you use the SQS FIFO queue, at the moment, you might receive three messages at the same time. The workers might serve these three messages randomly (e.g. 2->1->3). For the cases when that is required to follow the FIFO order (for example), we need a new option to consume a new message from any queue in a particular order.
For example:
parallelism: 1: would mean we can have only 1 message (JOB) in the RR's internal queue before the ACK. So, the SQS FIFO order will be preserved.

parallelism: N: would mean we can have N messages simultaneously in the internal priorities queue.

EDIT:

1. AMQP -> supports this option, no changes are needed.
2. SQS -> doesn't support, add the prefetch configuration option.
3. Beanstalk -> doesn't support, add the prefetch configuration option.
4. NATS -> doesn't support, add the prefetch configuration option.
5. Kafka -> not sure, need to investigate.
6. in-memory -> update the prefetch configuration option.
7. localdb -> update the prefetch configuration option.

[rauanmayemir]

I’m doing this currently with AMQP by setting a fetch size. Will this option just be a generalization on top of that?

[wolfy-j]

Pretty much - yes.

[rustatian]

Yeah, the AMQP driver already has this option via prefetch. I'm unsure about Kafka (max.poll.records) and SQS (Java SDK has smt like batches, but I don't see this option for the Golang driver).

[wolfy-j]

Except it will be done on RR level, not driver level. Before it reaches internal task queue.

[rustatian]

I think that the more natural and, let's say, 'expected' name for this option is - prefetch.

But on the other hand, we have drivers (like amqp) that support this option in that 'expected' way, and our new option will conflict with the natural option, which in my opinion, will confuse users. It's like prefetch <-> rr_prefetch 😕.

I think adding the prefetch option to the drivers that don't support that option via their configuration is essential (by configuration here, I mean natural driver's configuration).