Strange behaviour when sending a web socket request.

[de-ltd]

Hi,

So to briefly tell you about my set up, I have 4 websocket subscriptions running in 4 separate tokio threads which handle streams from different sources. Everything works completely fine, I have a 5th thread set up to connect to another web socket and send out requests using ws_client.request().

Upon calling ws_client.request(method, params) - every other websocket subscription in the other threads hangs and does not get any further messages. No disconnections or error messages happen, they just continues to wait for the next message which never comes.

I Have never seen this before working with other libs, I should be able to have multiple ws connections running in threads without conflicting. Does anyone know whats going on here? I have debugged it, it is definitely happening as soon as I send a message through .request() - the connection is always open and handled as soon as the program launches and does not cause any issues, it is only when we send a message does it hang all the other sockets.

Thank you.

[niklasad1]

Hey @de-ltd

I assume your are referring to tokio tasks right when you say tokio threads, correct?

My guess is that tokio will use 4 threads for the thread pool then each task blocks and prevents the tokio scheduler to reschedule other tasks then your fifth subscription will not get a slot because all threads are "blocked"...

So we are running our benchmarks with 1024 concurrent connections with 10 subscriptions per connection so seems unlikely that it's really a jsonrpsee issue to me.

Can you please share how to reproduce this? How are you spawning the tokio tasks that causes this or alternatively your repo/code?

Another solution is to use tokio console to debug it.

[de-ltd]

Yes sorry tasks,

"My guess is that tokio will use 4 threads for the thread pool then each task blocks and prevents the tokio scheduler to reschedule other tasks then your fifth subscription will not get a slot because all threads are blocked..."

I am not really sure what you mean by this - sorry I am quite new to tokio / async. But the 5th task is not actually a subscription, just a regular ws client to send and receive messages.

All 5 tokio tasks spawn as soon as the application is started and connect, the 4 that are ws subscriptions connect to their end points and then loop through received messages. The 5th one just connected and waits to be sent a payload via a channel to send out using request().

[niklasad1]

I am not really sure what you mean by this - sorry I am quite new to tokio / async. But the 5th task is not actually a subscription, just a regular ws client to send and receive messages.

All 5 tokio tasks spawn as soon as the application is started and connect, the 4 that are ws subscriptions connect to their end points and then loop through received messages. The 5th one just connected and waits to be sent a payload via a channel to send out using request().

I'm trying to ask whether you use any sync operations in your spawned tasks without "await points" because in such scenarios the tokio scheduler can't context-switch because it will execute until it yields (i.e, async-await in rust). See docs here: https://docs.rs/tokio/latest/tokio/task/index.html

Are your channels async? Are you using a loop without await points? If you need to block you can use https://docs.rs/tokio/latest/tokio/task/fn.spawn_blocking.html

You need to provide more information otherwise I have no way to investigate this....

[de-ltd]

ahhhh I had a thread::sleep in one of the tasks! possibly that was the culprit I will give it a test, thank you.

[niklasad1]

@de-ltd is this issue fixed on your end now?

[de-ltd]

Yes that was indeed the issue, blocking my other tasks. Thanks for the help.