Error creating thread: Resource temporarily unavailable

[joaoantoniocardoso]

Related report:

Another crash
Also something has changed with stream thumbnails.. they pop in and out a lot more now. I get the message
Thumbnail for source ""ball"" is temporarily unavailable. Try again later. Details: deadline has elapsed""""
Also happened on a customers ROV right now. They were at 60m depth and i had to go in and restart the MCM service.
Here are the recent logs as well as terminal output.

Possible thread leak, leading to SO thread limit, but which threads? From a brief investigation, it seems to be Mavlink and Signaller.

2023-11-21T09:29:36.117533Z DEBUG                                     MavlinkCamera-9 ThreadId(1375) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent

(mavlink-camera-manager:79): GStreamer-WARNING **: 09:29:36.347: failed to create thread: Error creating thread: Resource temporarily unavailable
2023-11-21T09:29:36.996633Z DEBUG                                         Signaller-1 ThreadId(06) accept_connection{peer=192.168.40.233:42002}:handle_connection{peer=192.168.40.233:42002}:handle_message{msg=Text(""{\""type\"":\""question\"",\""content\"":{\""type\"":\""startSession\"",\""content\"":{\""consumer_id\"":\""c891ea9f-18d1-42a9-9e4e-22c9c6a7c696\"",\""producer_id\"":\""b8bdeb99-07b0-42a6-989f-9e84105096e6\""}}}"") sender=UnboundedSender { chan: Tx { inner: Chan { tx: Tx { block_tail: 0x24f3d8f0, tail_position: 2 }, semaphore: Semaphore(0), rx_waker: AtomicWaker, tx_count: 1, rx_fields: ""..."" } } }}:add_session{bind=BindOffer { consumer_id: c891ea9f-18d1-42a9-9e4e-22c9c6a7c696, producer_id: b8bdeb99-07b0-42a6-989f-9e84105096e6 } sender=UnboundedSender { chan: Tx { inner: Chan { tx: Tx { block_tail: 0x24f3d8f0, tail_position: 2 }, semaphore: Semaphore(0), rx_waker: AtomicWaker, tx_count: 2, rx_fields: ""..."" } } }}: src/stream/manager.rs:430: WebRTC session created: dc94a03d-7175-4388-9985-7ac947511dbc
2023-11-21T09:29:37.003044Z DEBUG                                         Signaller-1 ThreadId(06) accept_connection{peer=192.168.40.233:42002}:handle_connection{peer=192.168.40.233:42002}: /cargo/registry/src/index.crates.io-6f17d22bba15001f/tungstenite-0.20.0/src/protocol/mod.rs:666: Received close frame: None    
2023-11-21T09:29:37.003179Z DEBUG                                         Signaller-1 ThreadId(06) accept_connection{peer=192.168.40.233:42002}:handle_connection{peer=192.168.40.233:42002}: /cargo/registry/src/index.crates.io-6f17d22bba15001f/tungstenite-0.20.0/src/protocol/mod.rs:683: Replying to close with Frame { header: FrameHeader { is_final: true, rsv1: false, rsv2: false, rsv3: false, opcode: Control(Close), mask: None }, payload: [] }    
2023-11-21T09:29:37.004473Z DEBUG                                         Signaller-1 ThreadId(06) /cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-tungstenite-0.20.0/src/lib.rs:355: websocket start_send error: WebSocket protocol error: Sending after closing is not allowed    
2023-11-21T09:29:37.004602Z ERROR                                         Signaller-1 ThreadId(06) src/stream/webrtc/signalling_server.rs:158: Failed repassing message from the MPSC to the WebSocket. Reason: Protocol(SendAfterClosing)
2023-11-21T09:29:37.007489Z  INFO                                         Signaller-1 ThreadId(06) src/stream/webrtc/signalling_server.rs:167: WebSocket connection closed: 192.168.40.233:42002
2023-11-21T09:29:37.007693Z DEBUG                                         Signaller-1 ThreadId(06) accept_connection{peer=192.168.40.233:42002}:handle_connection{peer=192.168.40.233:42002}: src/stream/webrtc/signalling_server.rs:199: Connection terminated: 192.168.40.233:42002

(mavlink-camera-manager:79): GLib-ERROR **: 09:29:37.029: creating thread 'webrtcbin-a843d7db-be79-437c-985c-895a9c9d74c6:ice': Error creating thread: Resource temporarily unavailable
Trace/breakpoint trap (core dumped)

[patrickelectric]

We need to find a way to shutdown the runtime, from the local test, it appears that the heartbeat and send_message threads stop, but the runtime itself drops but waits forever. with the runtime shutdown function we can force it to kill the OS threads.

[patrickelectric]

From a simple test fetching the thread names on htop when a camera connects and disconnected it appears that the origin of the problem comes from the queue thread.

code starts

camera reconnects

[joaoantoniocardoso]

Can't reproduce, probably fixed by #362