Should rate-limited tasks increment the prefetch limit?

[RJPercival]

When Celery is configured with a prefetch limit of 1, and receives a task with an ETA but that ETA is in the future, Celery will increment the prefetch limit and schedule that task to be run when its ETA elapses. The incremented prefetch limit allows the worker to fetch another task while holding the future-dated task. This seems sensible.

The behaviour is different for rate-limited tasks. If a worker receives a task but there are not sufficient tokens (due to rate-limiting) to execute that task, it will similarly schedule that task to be run when it is expected that a token will be available, but the prefetch limit is not incremented. This means the worker is unable to fetch any more tasks and sits idle. It does so even though the next task it could fetch might not be rate-limited. This seems like a bug - should the prefetch limit not also be incremented in this scenario?

Here is the relevant code:

celery/celery/worker/strategy.py

Lines 190 to 202 in febe695

	if rate_limits_enabled:
	bucket = get_bucket(task.name)
	if eta and bucket:
	consumer.qos.increment_eventually()
	return call_at(eta, limit_post_eta, (req, bucket, 1),
	priority=6)
	if eta:
	consumer.qos.increment_eventually()
	call_at(eta, apply_eta_task, (req,), priority=6)
	return task_message_handler
	if bucket:
	return limit_task(req, bucket, 1)

[auvipy]

can you share a proof of concept of what you are thinking? with a failing test?

[auvipy]

@thedrow are you/ your team up for a gig?

[RJPercival]

@auvipy / @thedrow: Any chance of getting some support on this in the near future? I'm happy for that to take the form of some pairing where someone talks me through what the fix and tests should look like.

[RJPercival]

@auvipy, I guess not?

[auvipy]

can you shoot me a line on auvipy@gmail.com please?

[RJPercival]

Sorry @auvipy, I missed this response. I'll email you now.

[auvipy]

@RJPercival can you please try incrementing this and see what happens?

[RJPercival]

There don't appear to be any tests for this, so there's no straightforward way to "see what happens". That was one of things I was looking for support with - getting some tests written that cover rate-limiting comprehensively.

[RJPercival]

Related issue: #8119

[dbartenstein]

Has there been any progress on this?

[RJPercival]

Not that I'm aware of. I haven't been able to get any support from the Celery maintainers unfortunately.