You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We noticed that every worker creates two connections to the redis server. Does this have technical reasons? If we can reduce the total number of connections, this would improve scalability of rq with the same number of redis servers. We are happy to contribute code/time, since we run thousands of workers.
I placed this at the head of redis-py's connection.AbstractConnection.__init__:
import traceback
traceback.print_stack()
which showed the when the connections were opened when running the command line worker from rq 1.15.1:
rq worker -c config QUEUENAME
File "PYENV/bin/rq", line 8, in <module>
sys.exit(main())
File "PYENV/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "PYENV/lib/python3.11/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "PYENV/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "PYENV/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "PYENV/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "PYENV/lib/python3.11/site-packages/rq/cli/helpers.py", line 422, in wrapper
return ctx.invoke(func, cli_config, *args[1:], **kwargs)
File "PYENV/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "PYENV/lib/python3.11/site-packages/rq/cli/cli.py", line 259, in worker
cleanup_ghosts(cli_config.connection)
File "PYENV/lib/python3.11/site-packages/rq/contrib/legacy.py", line 19, in cleanup_ghosts
for worker in Worker.all(connection=conn):
File "PYENV/lib/python3.11/site-packages/rq/worker.py", line 245, in all
worker_keys = worker_registration.get_keys(queue=queue, connection=connection)
File "PYENV/lib/python3.11/site-packages/rq/worker_registration.py", line 78, in get_keys
return {as_text(key) for key in redis.smembers(redis_key)}
File "PYENV/lib/python3.11/site-packages/redis/commands/core.py", line 3399, in smembers
return self.execute_command("SMEMBERS", name)
File "PYENV/lib/python3.11/site-packages/redis/client.py", line 533, in execute_command
conn = self.connection or pool.get_connection(command_name, **options)
File "PYENV/lib/python3.11/site-packages/redis/connection.py", line 1084, in get_connection
connection = self.make_connection()
File "PYENV/lib/python3.11/site-packages/redis/connection.py", line 1124, in make_connection
return self.connection_class(**self.connection_kwargs)
File "PYENV/lib/python3.11/site-packages/redis/connection.py", line 584, in __init__
super().__init__(**kwargs)
File "PYENV/lib/python3.11/site-packages/redis/connection.py", line 168, in __init__
traceback.print_stack()
10:08:04 Worker rq:worker:c1c2c26d2b434ec6b6ece70c0bf8160f started with PID 12784, version 1.15.1
10:08:04 Subscribing to channel rq:pubsub:c1c2c26d2b434ec6b6ece70c0bf8160f
File "PYENV/bin/rq", line 8, in <module>
sys.exit(main())
File "PYENV/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "PYENV/lib/python3.11/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "PYENV/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "PYENV/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "PYENV/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "PYENV/lib/python3.11/site-packages/rq/cli/helpers.py", line 422, in wrapper
return ctx.invoke(func, cli_config, *args[1:], **kwargs)
File "PYENV/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "PYENV/lib/python3.11/site-packages/rq/cli/cli.py", line 301, in worker
worker.work(
File "PYENV/lib/python3.11/site-packages/rq/worker.py", line 403, in work
self.bootstrap(logging_level, date_format, log_format)
File "PYENV/lib/python3.11/site-packages/rq/worker.py", line 571, in bootstrap
self.set_state(WorkerStatus.STARTED)
File "PYENV/lib/python3.11/site-packages/rq/worker.py", line 861, in set_state
connection.hset(self.key, 'state', state)
File "PYENV/lib/python3.11/site-packages/redis/commands/core.py", line 5019, in hset
return self.execute_command("HSET", name, *items)
File "PYENV/lib/python3.11/site-packages/redis/client.py", line 533, in execute_command
conn = self.connection or pool.get_connection(command_name, **options)
File "PYENV/lib/python3.11/site-packages/redis/connection.py", line 1084, in get_connection
connection = self.make_connection()
File "PYENV/lib/python3.11/site-packages/redis/connection.py", line 1124, in make_connection
return self.connection_class(**self.connection_kwargs)
File "PYENV/lib/python3.11/site-packages/redis/connection.py", line 584, in __init__
super().__init__(**kwargs)
File "PYENV/lib/python3.11/site-packages/redis/connection.py", line 168, in __init__
traceback.print_stack()
10:08:04 Worker rq:worker:c1c2c26d2b434ec6b6ece70c0bf8160f started with PID 12784, version 1.15.1
10:08:04 Subscribing to channel rq:pubsub:c1c2c26d2b434ec6b6ece70c0bf8160f```
Note that no job is running, so it is really just the bare worker waiting for work.
The text was updated successfully, but these errors were encountered:
Thanks for raising this up, this is a very valuable insight.
I have never profiled the number of connections each worker makes but if I could guess the two connections come from the worker itself and a worker thread that opens a pubsub connection to Redis, listening to remote commands.
This is just a guess though, if you're willing to investigate further and want to have an option to reduce the number of open connections, I'd be happy to accept contributions in these areas. An option I can think of right away would be to disable the pubsub thread if it's not used.
We noticed that every worker creates two connections to the redis server. Does this have technical reasons? If we can reduce the total number of connections, this would improve scalability of rq with the same number of redis servers. We are happy to contribute code/time, since we run thousands of workers.
Steps to reproduce:
I placed this at the head of
redis-py
'sconnection.AbstractConnection.__init__
:which showed the when the connections were opened when running the command line worker from rq 1.15.1:
The text was updated successfully, but these errors were encountered: