Connection failing on Safari when connecting via TCP

[holzgeist]

Describe the bug
We see regular connection losses and reconnections (every 30s) when running Livekit with Safari (both macOS and iOS) via ICE/TCP. It works fine with Chrome on macOS and with every Browser in macOS with ICE/UDP

Server

• Version: 1.5.2 (and whatever is deployed on meet.livekit.io as per 2024-02-14)
• Environment: local dev, kubernetes with public ip and meet.livekit.io

Client

• SDK: js
• Version: 2.0.1 (and meet.livekit.io)
• Browser: Safari 17.3
• OS: macOS 14.3 (Sonoma)

To Reproduce
Steps to reproduce the behavior (>90% of retries):

1. block outgoing UDP (see below)
2. Go to meet.livekit.io and start a meeting
3. Wait for a bit (usually <1min)
4. Briefly see a "Reconnecting" overlay and a WebSocket connection error in the logs:

[Error] WebSocket connection to 'wss://meet-uylq8icb.livekit.cloud/rtc?access_token=eyJhbGciOiJIUzI1NiJ9.eyJuYW1lIjoidXNlcm5hbWUiLCJ2aWRlbyI6eyJyb29tIjoiNXM4dS14aGhwIiwicm9vbUpvaW4iOnRydWUsImNhblB1Ymxpc2giOnRydWUsImNhblB1Ymxpc2hEYXRhIjp0cnVlLCJjYW5TdWJzY3JpYmUiOnRydWV9LCJpc3MiOiJBUElWZU1CcHF3eUxrTU4iLCJleHAiOjE3MDc5MjI5MDEsIm5iZiI6MCwic3ViIjoidXNlcm5hbWUifQ.UDs5UuJ3nSgPoaX03D0ZJyZIwkQF1h6bcfuN82Zj58M&auto_subscribe=1&sdk=js&version=1.15.12&protocol=11&adaptive_stream=1' failed: The operation couldn’t be completed. Socket is not connected
[Error] websocket error – {room: "5s8u-xhhp", roomSid: "RM_eytRdGE3srMf", identity: "username", …}
{room: "5s8u-xhhp", roomSid: "RM_eytRdGE3srMf", identity: "username", error: Event}Object
	handleWSError (3e572273-313afb0672996214.js:1:193624)
	(anonyme Funktion) (3e572273-313afb0672996214.js:1:186133)
	(anonyme Funktion) (3e572273-313afb0672996214.js:1:60773)
	Promise
	Tt (3e572273-313afb0672996214.js:1:60533)
[Warning] websocket closed – {room: "5s8u-xhhp", roomSid: "RM_eytRdGE3srMf", identity: "username", …} (3e572273-313afb0672996214.js, line 1)

Full log: meet.livekit.io.log

Expected behavior
Have a stable connection without frequent reconnects and interruption of video/audio stream

Screenshots
n/a

Additional context
How to block UDP on macOS:

• Add the following lines to /etc/pf.conf:

block out proto udp from any to any port 443 # we enabled UDP mux on this port
block out proto udp from any to any port 1024:60000 # block all high port ranges that ICE could use

• run sudo pfctl -f /etc/pf.conf to load rules
• run sudo pfctl -E to enable firewall

Alternatively run a local setup with following config:

rtc:
  udp_port: 0
  force_tcp: true

log_level: debug
logging:
  pion_level: debug
  json: false
  level: debug

Previous debugging
With help of wireshark and reading logs from above local config, I found the following:

From wireshark it looks like that the client is closing a WebRTC TCP connection, causing following warning on livekit server:

WARN	livekit.pion.tcp_mux	v2@v2.3.11/tcp_packet_conn.go:163	Failed to read streaming packet: EOF

This is the first line that's different from a working example (<10% of tries).
Wireshark shows a "TCP Window full" warning briefly after this connection terminates.

~10-15s later the first reconnect occurs.
Livekit Server sends a TCP RST packet on a different stream, causing the client to close the websocket an start a reconnection.
This happens regularly every ~30s after that

[davidzhao]

@holzgeist thanks for the excellent bug report. we'll investigate to see if there's some sort of incompatibility between the ICE/TCP implementation and Safari.