rabbitmq support for celery

[amirreza8002]

Description

adding the option to choose rabbitmq as message broker for celery projects

Rationale

well not all projects need a storage backend for celery so using redis instead of rabbitmq doesn't make much sese for them
also the combination of redis and rabbitmq is a popular choice

[shywn-mrk]

I can work on this and add this to the project if needed. AFAIK we should add the image for local and production environments and the related config. We do need a list of plugins that we may need for the project to enable and include in RabbitMQ config. The question to ask is if the user wants RabbitMQ as a broker or not. Since it may need to also use Redis for cache purposes we should set the value of the celery broker URL based on this option.

[browniebroke]

I'm personally against this: as it was just stated above we need/use redis for caching, so we won't be able to eliminate the redis service.

IMO it's about simplifying the stack by avoiding a separate service just for the Celery broker, adding RabbitMQ means:

• it's one more technology to explain to beginners: "why do I need/want RabbitMQ?"
• one more decision point while setting up a project
• it's more resources needed to run on the server (if using Docker) or one extra service provider to find (e.g. Heroku)
• one more combination to test against

So unless there is a massive demand for it, and it's a simple implementation, then maybe.

[amirreza8002]

i do understand your point of view
i personally use rabbitmq and have been in situations where i didn't really need redis
so i thought it is a good option to have here
but i understand that it might get you some trouble to maintain
but i still would love to see it happen if you found the time 🙃

[Andrew-Chen-Wang]

For many organizations I've worked with, not having Redis eventually made us feel lots of pain. I've used Redis for billions of tasks in a project before. There weren't any reasons to use RabbitMQ except for extremely volatile workloads where compute resources could be maximized at any time or having a bad continuous deployment setup . RabbitMQ's also beginning to look like Kafka's API too; once there's that sort of complexity, that's when you can think about using RabbitMQ for Celery. But for project gen, it's worth just sticking to Redis.

[browniebroke]

I've left this open for a while, to see if was interest for it. I think there hasn't been enough interest to justify the extra maintenance complexity I mentioned earlier.

[github-actions]

As discussed, we won't be implementing this. Automatically closing.