Handling Intermittent ValueError in _handle_eio_message with Multithreading

[TheoBoyer]

Hello python-socketio community,

I've integrated python-socketio into a PyTorch DataLoader, which functions as a socketio client. It's designed to request and receive data in a multi-threaded setup using a ThreadPoolExecutor. However, I'm intermittently encountering a ValueError that I'm struggling to debug or handle gracefully.

Here's the pattern I'm using:

def worker(self):
    ...
    with emit_lock:
            self.sio.emit("request", {'relative_path': relative_path})
    ...
def __iter__(self):
    while True:
        ...
        with ThreadPoolExecutor(max_workers=self.batch_size) as executor:
            # Submit the function to the executor and collect Future objects
            futures = [executor.submit(self.worker) for _ in range(self.batch_size)]
            batch = [f.result() for f in futures]
        for b in batch:
            yield b
        ....

The server responds to "request" with dict data packaged in a JSON format (note that one of the values of the dict is bytes)

Occasionally, the client throws the following exception:

Exception in thread Thread-742668 (_handle_eio_message):
--
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/opt/conda/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/lib/python3.10/site-packages/socketio/client.py", line 483, in _handle_eio_message
pkt = self.packet_class(encoded_packet=data)
File "/opt/conda/lib/python3.10/site-packages/socketio/packet.py", line 43, in __init__
self.attachment_count = self.decode(encoded_packet) or 0
File "/opt/conda/lib/python3.10/site-packages/socketio/packet.py", line 77, in decode
self.packet_type = int(ep[0:1])
ValueError: invalid literal for int() with base 10: b'\x93'

This error is problematic because:

• It halts the entire data loading process.
• It's unclear how to catch and handle this exception since it occurs in a background thread managed by the library.

I suspect the issue might be related to one of the following:

• Multi-threading complexities, missing thread lock somewhere
• Encoding issues when sending bytes in JSON.
• Handling of packets that might be getting corrupted.

I am looking for advice on two fronts:

• How can I handle this exception to allow the DataLoader to continue operation, even if it means discarding a problematic packet?
• Insights into why this error might occur and potential strategies to prevent it.

Additional context: The error seems more frequent when I enable multi-processing, likely due to the increased packet exchange volume. It occurs with and without multi-processing enabled, though.

Any guidance or suggestions would be greatly appreciated.
Thank you for your time,

Théo

[miguelgrinberg]

This happens because you are receiving invalid data. The Socket.IO client is thread-safe, but you can't use the same client object concurrently on different threads. For example, if you emit on the same client instance on different threads at about the same time, the packets corresponding to these two emits may be sent in an incorrect order and mixed up.

If you are going to use the client from multiple threads, what you should do is protect the emits() with a lock.

[edvard-bjarnason]

Hmm, well this is a bit confusing statement, the client object is not thread-safe if it can't be used without errors concurrently from multiple threads. I'm running into a similar issue and this is probably what the problem is. If this is the problem then I would suggest to put the lock in the emit function of the client to make it thread safe.