-
Notifications
You must be signed in to change notification settings - Fork 218
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support TCP for protocol messages #3242
Comments
I like the proposal and support this approach. I kinda wonder if it’s possible for another person to hijack the TCP connection. e.g. a client connects, receives a channel ID 5. Maybe the presence of a new channel is broadcasted, and a rogue client who sees this message quickly makes a TCP connection to hijack that ID. Maybe we don’t need to worry about this case, but it just kinda pops up. |
I believe this is very much possible. There aren't any guarantees currently either way as we don't have encryption. What's the disadvantage of trying to connect via TCP first? |
Thanks! I used your RPC server as a template for the connection handler.
That's certainly a good observation, and worth considering, even if unlikely. I think it could largely be mitigated by only acting upon a |
I couldn't think of a compatible way to associate a subsequent UDP audio stream with a TCP connection that was made first. Especially if they had travelled through NAT. When I thought of starting with UDP as normal, and then sending back the |
It sounds pretty good. I'd be inclined only to move explicitly those messages we know cause problems. However, the "infrastructure" should be there to support other messages. I guess only the audio packets themselves need remain on UDP. TCP/IP keep alive will be running on the TCP/IP connection, right? At the moment, a UDP drop out isn't seen as a "connection failure" by the server for quite a large window (relatively). Would the TCP/IP connection remain "connected" or drop out if keep alive failed? How would a "continuous" UDP audio connection work in this situation? |
Seems like that's a fundamental problem. So to not break backwards compatibility the server must still respond on UDP audio messages for session creation. Could we send an empty audio message to the server to query the version/capabilities, then do something comparable to what syn cookies do (https://en.m.wikipedia.org/wiki/SYN_cookies) for some kind of authentication and then set up a TCP connection. |
A limitation, certainly, but I wouldn't call it a problem.
Yes indeed.
I don't really think that gains us anything except quite a lot of unneeded complexity. I think the source IP is an adequate enough "secret" to validate the TCP connection that follows the start of the UDP session. |
Yes, I agree.
That's definitely true, also.
If I remember correctly (from a long time ago), TCP keepalive has a very long timeout, and it just intended to keep a session alive when there is no data to exchange. In the case of Jamulus, the server regularly sends the channel levels using |
OK, assuming that the TCP keepalive is longer than the Jamulus UDP audio "keep alive" time for a channel, we'd just need to ensure that part of channel clean up is to clear down the TCP socket, too? |
Yes, absolutely. |
Let's say a client connects and establishes a TCP connection. At some point the server tries to send a message over the TCP socket and gets an error back indicating the socket is no longer valid. If the UDP audio stream continues, how does recovery work in this situation? Would the client get a CONN_FAIL, too, and know to re-initiate the TCP connection? Will the server know what its state was at the time the connection failed for recovery? I guess association with the same CChannel would continue. |
Well at the very least, the server will close its end of the TCP socket and set At some point the client would also notice the socket had failed, and would either also revert to UDP-only, or could try re-connecting TCP. I think it would be unlikely in practice that the TCP socket would fail while the UDP is still working. A typical network outage would affect both at once. |
OK, so as I see it, even if the client has established TCP for certain messages, it should handle them arriving over UDP as currently -- but use an "unexpected UDP fallback" to indicate the server can't send the message over TCP and re-initiation of the socket is probably needed. |
As I have been thinking about the design of this, and conducting a few protocol experiments, I have found some significant issues that we need to solve, concerning the interaction between a client's Connect Dialog and the servers listed by the directory. I have some ideas, but would be grateful for any other suggestions. Background - the current Jamulus behaviour on UDP When the client's user opens the Connect Dialog, or changes the Directory selection (Genre) while the Connect Dialog is open, the following things happen:
As stated at the beginning of this Issue, it is that fact that the The Problem A client using the proposed TCP functionality would send the
Possible Solutions
IPv6 considerations Several of the existing protocol messages encode an IP address as 4 bytes. This is fine for an IPv4 address, but an IPv6 address takes 16 bytes, if represented in binary format. So we would also need to define new messages to replace the followingi, which all include an IPv4 address as a 4-byte binary value:
The suggested Conclusion This is turning out to be more complex than anticipated, so I would be interested in comments on the above, and suggestions for the way forward. |
I don't think solution 1 is feasible since NATs behave as they want. I'm a bit worried about TCP hole punching, since it most likely works different to UDP anyway (I remember having heared that it's non trivial) - this needs some more research. Food for thought:
|
If the Client initiates the Directory server list request over UDP and gets back an initial "CLM_TCP" response (with some token), it could then follow up with the TCP request for the server list (including the returned token) and the Directory would know the UDP and TCP port details. It would mean the Client knew immediately that the server supported TCP, without sending a TCP request and having to wait for that to time out, too, before initiating the UDP server list request, if it didn't. That would then allow the Directory to send TCP and UDP port details to the Servers over either TCP or UDP. |
I think for registration and Directory interaction, we'll need A Client wanting both IPv4 and IPv6 Servers would need to make two requests, as I can't think of a way to embed the two lots of semantics into one request... |
I like this idea, I'll give it some more thought. Thanks! |
Yes, I agree. Solution 1 was my first thought, and I was explaining why I didn't think it would work.
Thanks for the link below. I've read through it, and don't think it's suitable for what we want to achieve.
Maybe, but I'd be a bit worried about backward compatibility.
I don't think that is possible. Once a TCP connection is established, we can't move one end of the connection from one host to another. Unless I've misunderstood what you mean. Thanks! |
Just to add a bit more. For compatibility, the directory will still have to send the There is no per-client state stored in the directory to relate For a client-directory connection that drops fragmented UDP, the |
What is the current behaviour and why should it be changed?
All Jamulus protocol (non-audio) messages are currently delivered over the same UDP channel as the audio. For most protocol messages, this is fine, but those that send a list of servers from a directory, or a list of clients from a server, can generate a UDP datagram that is too large to fit into a single physical packet. Physical packets are constrained by the MTU of the Ethernet interface (normally 1500 bytes or less), and further by any limitations in links between hops on the internet. Neither the client nor the server has any control over these limitation. It's also possible a large welcome message could require fragmentation.
The UDP protocol itself allows datagrams up to be up to nearly 65535 bytes in size, minus any protocol overhead. IPv4 will allow nearly all of this size to be used, in theory. If the IPv4 datagram being sent by a node (host or router) is too large to fit into a single packet on the outgoing interface, the IP protocol will fragment the packet into pieces that do fit, with IP headers that contain the information needed to order and reassemble the fragments into a single datagram at the receiving end. Normally intermediate hops do not perform any reassembly, but will further fragment an IP packet if it will not fit the MTU of the outgoing interface.
The receiving end needs to store all the received fragments as they arrive and can only reassemble them into the original datagram once all fragments have been received. The loss of even one fragment renders the whole datagram lost, and the remaining received fragments consume resources until they time out and are discarded. There are also possibilities for a denial of service attack if an attacker deliberately sends lots of fragments with one or more missing.
If a directory has more than around 35 servers registered (depending on the length of the name, city, etc.), the list of servers sent to a client when requested is certain to be fragmented. Similarly, if a powerful server has a lot of clients connected, e.g. a big band or large choir, the list of clients sent to each connected client can get fragmented. In either of these cases, a client that is unable to receive fragmented IP packets will show an empty list or an empty mixer panel.
There are several reasons that fragmented IP datagrams can fail to make it from server to client:
The IPv6 limitation means that resolving this issue is a prerequisite to implementing IPv6 support in directories as per the ongoing discussion in https://github.com/orgs/jamulussoftware/discussions/1950.
Describe possible approaches
There is a longstanding discussion at https://github.com/orgs/jamulussoftware/discussions/1058 about the problems this issue is intended to solve, and mentioning various approaches that have been tried or proposed.
REQ_SPLIT_MESS_SUPPORT
,SPLIT_MESS_SUPPORTED
andSPECIAL_SPLIT_MESSAGE
. I'm not sure whether such split messages are ever used in practice, and it appears that they only apply to connected messages, not the connectionless messages which are most at risk from fragmentation. In addition, the size of split parts is fixed, and not intelligently determined from any kind of path MTU discovery.CLM_RED_SERVER_LIST
). This also fails to avoid the problem, as a directory list that may take around 7 fragments in its full form still takes around 3 fragments in its reduced form.The only possible solution is to send some protocol messages using TCP instead of UDP, when talking to a compatible client. UDP would still be available for backward compatibility when talking to older clients or older servers.
There are two kinds of protocol message that each need to be handled differently:
CLM_*
. These are unrelated to a channel (with one exception). They are mainly used by a client to fetch information for the Connect dialog:CLM_REQ_SERVER_LIST
).CLM_REQ_CONN_CLIENTS_LIST
).Connectionless Messages
For connectionless messages, the client can send a TCP connection request to the server, with a timeout. If the server supports TCP, this connection will be accepted and the client can then send the
CLM_REQ_*
message over the TCP connection. The server needs to interpret the message and send the response back over the same TCP connection. The client can then close the connection or leave it open for sending another message (tbd). If the TCP connection from the client is refused or times out (probably due to a firewall dropping the connect request), the client can fall back to the existing UDP usage to send the request. For this reason, the TCP connection timeout will need to be short, something like 2 seconds. This will be plenty of time for a compatible server to answer.I have a branch that implements the server side of connectionless messages over TCP, currently just for
CLM_REQ_SERVER_LIST
andCLM_REQ_CONN_CLIENTS_LIST
, but others could be added as needed. It can be seen at https://github.com/softins/jamulus/tree/tcp-protocol. It is necessary to pass the TCP socket pointer via the function calls, signals and slots, to the point at which the response message can be sent. If this socket pointer isnullptr
, the response will be send over UDP as presently, otherwise it will be sent to the referenced TCP socket.Note that due to the variable size of Jamulus protocol messages, and the stream-oriented nature of TCP sockets, it is necessary for the receiver at each end first to read the fixed-size header (9 bytes), determine from that header the payload size, and then read the payload, plus two more bytes for the CRC.
I have tested it using a Python client, based on @passing's jamulus-python project, but enhanced to support TCP. See https://github.com/softins/jamulus-python/tree/tcp-protocol.
The next step is to add to Jamulus the client side of using TCP for connectionless messages to fetch server and client lists.
Connected Channel Messages
For connected channel messages, the situation is a little more complicated. The following factors must be considered:
CONN_CLIENTS_LIST
).CONN_CLIENTS_LIST
to each client that is still connected. For a large busy server with many clients, this could be a long message and subject, presently, to UDP fragmentation. It should therefore be sent if possible over TCP.My proposal to solve the last point above is as follows:
CHostAddress
) for a matching channel (inCChannel vecChannels[]
), and on not finding a match, allocates a free channel in that array for the new client. It stores theCHostAddress
value in the allocated channel, and returns the ID (index) of the new channel.CLIENT_ID
connected channel message. This is all existing behaviour so far.CLIENT_ID
message, will initiate a TCP connection to the server. If the connection attempt fails, the client will assume the server is not TCP-enabled and will not retry. Operation will continue over UDP only as at present.CLIENT_ID
message to the server, specifying the client ID that it had just received from the server. This will enable the server to associate that particular TCP connection with the correctCChannel
, and the server will store thepTcpSocket
pointer in theCChannel
.CONN_CLIENTS_LIST
updates to the client over TCP if the socket pointer in the channel is not null, or otherwise over UDP. It could also send the welcome message over the same TCP socket, improving support for longer welcome messages.CLM_DISCONNECTION
the same as at present (over either UDP or TCP), but will also close any open TCP connection to the server.I have not yet implemented any of this connected channel functionality, beyond adding the
pTcpSocket
pointer to theCChannel
class.This is currently a work in progress, as described above. The purpose of this issue is to allow input from other contributors on the technical details mentioned above, and to keep the topic visible until the code is ready for a PR.
The expectation is that all the public directories will support TCP connections. This will also need suitable firewall rules at the servers. However, clients implementing all the above will still be backward-compatible with older directories and servers run by third parties. Similarly, older clients connecting to newer directories and servers will continue to operate as a present over UDP, with no use of TCP required.
Has this feature been discussed and generally agreed?
See the referenced discussion at https://github.com/orgs/jamulussoftware/discussions/1058 for history. I would value comments within this Issue regarding the solution I am proposing above. @pljones @ann0see @hoffie @dtinth and any others interested.
The text was updated successfully, but these errors were encountered: