Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ssl.SSLEOFError: EOF occurred in violation of protocol #942

Open
gborrageiro-ld opened this issue Aug 12, 2023 · 11 comments
Open

ssl.SSLEOFError: EOF occurred in violation of protocol #942

gborrageiro-ld opened this issue Aug 12, 2023 · 11 comments

Comments

@gborrageiro-ld
Copy link

Hello,

I am getting an exception when subscribing to a high throughput websocket endpoint, wss://fstream.binance.com/stream?streams=btcusdt@bookTicker/btcusdt@depth20@100ms/btcusdt@aggTrade/btcusdt@markPrice@1s/btcusdt@forceOrder.
To be precise, it is the btcusdt@bookTicker that is very busy, and even if I switch off any computation completely and just run websocket-client and rel, eventually I run into the below exception.

File "...", line 179, in __init__
    rel.dispatch()
  File ".../python3.10/site-packages/rel/rel.py", line 228, in dispatch
    registrar.dispatch()
  File ".../python3.10/site-packages/rel/registrar.py", line 125, in dispatch
    if not self.loop():
  File ".../python3.10/site-packages/rel/registrar.py", line 134, in loop
    e = self.check_events()
  File ".../python3.10/site-packages/rel/registrar.py", line 285, in check_events
    self.callback('read', fd)
  File ".../python3.10/site-packages/rel/registrar.py", line 178, in callback
    self.events[etype][fd].callback()
  File ".../python3.10/site-packages/rel/listener.py", line 127, in callback
    if not self.cb(*self.args) and not self.persist and self.active:
  File ".../python3.10/site-packages/websocket/_app.py", line 437, in read
    op_code, frame = self.sock.recv_data_frame(True)
  File ".../python3.10/site-packages/websocket/_core.py", line 427, in recv_data_frame
    self.pong(frame.data)
  File ".../python3.10/site-packages/websocket/_core.py", line 351, in pong
    self.send(payload, ABNF.OPCODE_PONG)
  File ".../python3.10/site-packages/websocket/_core.py", line 283, in send
    return self.send_frame(frame)
  File ".../python3.10/site-packages/websocket/_core.py", line 311, in send_frame
    l = self._send(data)
  File ".../python3.10/site-packages/websocket/_core.py", line 525, in _send
    return send(self.sock, data)
  File ".../python3.10/site-packages/websocket/_socket.py", line 170, in send
    return _send()
  File ".../python3.10/site-packages/websocket/_socket.py", line 147, in _send
    return sock.send(data)
  File ".../python3.10/ssl.py", line 1206, in send
    return self._sslobj.write(data)
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2426)

Do we expect with websocket-client and rel to cope with very busy websocket streams?

Can I stick rel.dispatch in some kind of a while loop, catch the exception and run rel.dispatch again?

My code looks like this:

for ws in self.ws_apps:
    ws.close()
self.ws_apps = []
for socket in [self.user_data_socket, self.mkt_data_socket]:
    ws = websocket.WebSocketApp(
        socket, on_open=self.on_open, on_message=self.on_message,
        on_error=self.on_error, on_close=self.on_close
    )
    self.ws_apps.append(ws)
    # noinspection PyTypeChecker
    ws.run_forever(
        dispatcher=rel, skip_utf8_validation=True, reconnect=3,
        ping_interval=180, ping_timeout=60
    )
rel.dispatch()

thanks
Gabriel
@bubbleboy14
Copy link
Collaborator

bubbleboy14 commented Aug 12, 2023 •
edited

@gborrageiro-ld thanks for the detailed report! Do you get the same SSLEOFError with the default (unspecified) dispatcher - that is, connected to the same endpoint, but without rel?

Also, since it looks like it's throwing this error in the course of trying to send a pong, I'm wondering whether the same problem is still encountered if you don't specify ping_interval and ping_timeout.

@gborrageiro-ld
Copy link
Author

hi @bubbleboy14,

I have two streams, one authenticated and the other is unauthenticated. I depend on a dispatcher like rel to handle multiple streams; what alternatives would you suggest?
If I remove the ping/pong, then the exception does eventually occur, yes.

Thanks
Gabriel

@gborrageiro-ld
Copy link
Author

I removed rel and let the busy, unauthenticated stream run, and I do not see this segmentation fault.

@bubbleboy14
Copy link
Collaborator

Hey @gborrageiro-ld I created a standalone test from the example code you provided:

import rel, websocket

BASEURL = "wss://fstream.binance.com/stream?streams=btcusdt@bookTicker/btcusdt@depth20@100ms/btcusdt@aggTrade/btcusdt@markPrice@1s/btcusdt@"
variants = ["bookTicker", "forceOrder"]

def onmsg(ws, msg):
    print("received:", msg)

for v in variants:
    ws = websocket.WebSocketApp(BASEURL + v, on_message=onmsg)
    ws.run_forever(dispatcher=rel, reconnect=3, ping_interval=180, ping_timeout=60)

rel.dispatch()

I've been running it for a while, and I haven't gotten that SSLEOFError. Does it take a long time? Are you able to reproduce the issue with this test case? If not, could you modify it to produce the error?

Anyway, thanks a million for the excellent report, we should be able to track this one down. LMK how it goes!

@andrea-mucci
Copy link

Hi,

I have the same error but with a different approach

i have a client that only send data to a server

.....
self.socket = websocket.WebSocket()
self.socket.connect(self.url)

#this is the method that send a command
def command(content):
     self.socket.send(....)

nothing special, the problem is that if i start to send messages from a loop, like:

for item in items:
     __method_that_send_ws_message()

in a certin point a get a failure:

  File "/home/andrea/PycharmProjects/ArchAI/archai/venv/lib/python3.10/site-packages/websocket/_core.py", line 283, in send
    return self.send_frame(frame)
  File "/home/andrea/PycharmProjects/ArchAI/archai/venv/lib/python3.10/site-packages/websocket/_core.py", line 311, in send_frame
    l = self._send(data)
  File "/home/andrea/PycharmProjects/ArchAI/archai/venv/lib/python3.10/site-packages/websocket/_core.py", line 525, in _send
    return send(self.sock, data)
  File "/home/andrea/PycharmProjects/ArchAI/archai/venv/lib/python3.10/site-packages/websocket/_socket.py", line 170, in send
    return _send()
  File "/home/andrea/PycharmProjects/ArchAI/archai/venv/lib/python3.10/site-packages/websocket/_socket.py", line 147, in _send
    return sock.send(data)
  File "/usr/lib/python3.10/ssl.py", line 1206, in send
    return self._sslobj.write(data)
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2426)

the strange thing is that before the error i have already sent a lot of messages, the problem come if a send the messages from a loop in a method

@bubbleboy14
Copy link
Collaborator

@andrea-mucci hm, interesting. the python ssl docs say:

A subclass of SSLError raised when the SSL connection has been terminated abruptly. Generally, you shouldn’t try to reuse the underlying transport when this error is encountered.

i don't know enough about the inner workings (of python's ssl module) to say how wide an array of circumstances (beyond a simple connection drop) can lead to this.

however, since you and @gborrageiro-ld both describe high throughput situations, i wonder whether, perhaps, some buffer is filling up... i mean, it really shouldn't be something like that. do you have any thoughts on this, @engn33r ?

i notice you're both running py3.10 - this may be related.

@andrea-mucci are you using the rel dispatcher?

can anyone provide a concise repro?

@engn33r
Copy link
Collaborator

engn33r commented Sep 10, 2023 •
edited

Checking any search engine for "python EOF occurred in violation of protocol" shows this a common error message and often related to the SSL library. I suggest sharing the OS that you are using (Windows, Mac, *nix) and the output of these commands when such an SSL error is encountered:

openssl version
python3 -c "import ssl; print(ssl.OPENSSL_VERSION); import websocket; print(websocket.__version__)"
python3 -V

@gborrageiro-ld
Copy link
Author

gborrageiro-ld commented Sep 13, 2023 via email

@bubbleboy14
Copy link
Collaborator

@gborrageiro-ld @andrea-mucci @engn33r

I put together a websocket throughput test in dez, which is an async network stack built on rel. The new test is in the latest version, 0.10.10.1, which can be installed thusly:

pip install dez

To run the test, first run this:

dez_test wsecho

That starts up a pretty basic websocket echo server (code) with a couple extra features.

Then, in a different terminal, run:

dez_test wsc

That (code) will prompt you to select a test configuration (high throughput in upstream/downstream/both/neither directions). It will then start up a WebSocketApp (with rel dispatcher) and proceed to direct heavy traffic in the specified direction. Anyway, it runs fine for me in every configuration.

Is anyone able to reproduce the issue using this test? Feel free to modify the code. Or is there a different websocket server that has this issue? For whatever reason, it doesn't seem to be a problem with dez.

Anyway, this shouldn't be hard to fix, if anyone can come up with a reliable repro - thoughts and test cases welcome!

@bubbleboy14
Copy link
Collaborator

@engn33r @gborrageiro-ld @andrea-mucci

I just encountered this issue myself! This patch fixed it for me:

#961

Does it fix things for you guys?

It's just two small changes in read() (in run_forever()): 1) check for SSLEOFError; 2) pass bool(reconnect) to handleDisconnect().

@engn33r, these seem like logical tweaks, right? Fixes the issue for me, anyway.

LMK what you think!

@ankitmaloo ankitmaloo mentioned this issue Jan 19, 2024
Closed
@bubbleboy14
Copy link
Collaborator

Anyone still experiencing this type of thing, check out this PR:

#983

I encountered some errors along these lines, and addressed them in that branch. LMK if it works for you!

@bubbleboy14 bubbleboy14 mentioned this issue May 9, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@bubbleboy14 @engn33r @andrea-mucci @gborrageiro-ld