Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(refactor) disable listen for subscriptions for Bitfinex #6841

Draft
wants to merge 3 commits into
base: development
Choose a base branch
from

Conversation

0xferit
Copy link
Contributor

@0xferit 0xferit commented Feb 12, 2024

Before submitting this PR, please make sure:

  • Your code builds clean without any errors or warnings
  • You are using approved title ("feat/", "fix/", "docs/", "refactor/")

A description of the changes proposed in the pull request:

The following method (listen_for_subscriptions) is inherited but it's inner dependencies are not implemented in Bitfinex connector, thus it's permanently erroring ("Unexpected error occurred when listening to order book streams. Retrying in 5 seconds...").

async def listen_for_subscriptions(self):
"""
Connects to the trade events and order diffs websocket endpoints and listens to the messages sent by the
exchange. Each message is stored in its own queue.
"""
ws: Optional[WSAssistant] = None
while True:
try:
ws: WSAssistant = await self._connected_websocket_assistant()
await self._subscribe_channels(ws)
await self._process_websocket_messages(websocket_assistant=ws)
except asyncio.CancelledError:
raise
except ConnectionError as connection_exception:
self.logger().warning(f"The websocket connection was closed ({connection_exception})")
except Exception:
self.logger().exception(
"Unexpected error occurred when listening to order book streams. Retrying in 5 seconds...",
)
await self._sleep(1.0)
finally:
await self._on_order_stream_interruption(websocket_assistant=ws)

Moreover, it's not only error messages that is annoying, but somehow, it causes CPU utilization to max out and break the bot after several hours. I suspect the root cause is the execution logic repeatedly entering listen_for_subscriptions and repeatedly throwing an exception. Seems like a busy-wait issue.

By disabling the execution of listen_for_subscription for Bitfinex, I observed the CPU max-out problem disappear (given that there are no other error message spamming). Moreover, disabling this function seems to cause zero negative effect since the Bitfinex logic does not require listen_for_subscription.

Previously, I tried to implement this function instead of disabling it, but I haven't received adequate support from the team, and without pull request reviews, I was not able to complete it. @yancong001 tipped the idea of disabling the function instead, so I tried it.

Tests performed by the developer:

After disabling the function, the CPU max-out problem disappeared, given that there was no other error spamming. To test, I run PMM on Bitfinex in a Digital Ocean droplet with 4 vCPUs and 8GB of RAM for 3 days. As opposed to v1.23.0, which almost always crashes within the first day, this fork run smoothly for 3 days and counting.

Screenshot 2024-02-12 at 04 06 32

Tips for QA testing:

Run PMM on Bitfinex, compare the latest release versus this fork, and observe CPU utilization.

Fixes #5919.

@0xferit 0xferit marked this pull request as draft February 12, 2024 04:10
@0xferit 0xferit changed the base branch from master to development February 12, 2024 04:10
@0xferit 0xferit marked this pull request as ready for review February 12, 2024 04:11
@0xferit
Copy link
Contributor Author

0xferit commented Feb 12, 2024

UPDATE: Around the 3.5-day mark, CPU shot up to 100% again. Fuck that really, I thought it was solved.

@0xferit 0xferit marked this pull request as draft February 12, 2024 14:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants