Out-of-order writes

[kixelated]

Hey folks,

I'm using Quinn for Media over QUIC. One of the issues raised is that relaying streams can introduce head-of-line blocking over multiple hops.

There's a few ways to address this issue, but perhaps the easiest is that a relay supports reading AND writing to streams out-of-order. Quinn currently supports out-of-order reads via read_chunk, but the corresponding write_chunk does not support out-of-order writes despite the name.

Writing chunks out of order can be fault-prone due to flow control. Naively chaining read_chunk.await followed by write_chunk.await can cause a deadlock when the receive flow control is larger than the send flow control. A smart implementation that polls the write_chunk futures in parallel can avoid this, but it might be better to design an API that makes it more difficult to screw up. And of course, an application that leaves a gap in the stream will stop progress from being made.

[djc]

Interesting, so you are (planning to be) using Quinn at Twitch?

I think something like write_chunks() would make sense. Perhaps there should be some way of having the application provide a limit to the distance between the first gap and the last chunk before the API yields an error? Presumably this is something that could be configured, maybe in the TransportConfig?

[kixelated]

Interesting, so you are (planning to be) using Quinn at Twitch?

Ha, I wish. I left Twitch a few months ago and I'm taking some time off to work on Media over QUIC. The goal is to make a reference library/demo with moq-rs. I finally get to use Rust instead of being forced by management to write my own QUIC library in Go...

Also, I just published webtransport-quinn which you might be interested in. It's rough around the edges, but it hides the awful HTTP/3 layer and exposes the Quinn QUIC API... except no stream IDs and error codes are u32 because WebTransport.

[Ralith]

This use case makes sense to me. I'm happy to hear the unordered read API has found a non-theoretical application! We've actually had a bunch of users implementing some sort of tunneling or forwarding system, so this might be widely applicable.

The flow control problem is an interesting one. I think the best solution is to ensure that a relay doesn't issue more flow control credit to the sender than it gets from the receiver. That makes a deadlock impossible for any receiver behavior without relying on correct configuration tuning.

Supporting this in Quinn will require some new APIs to allow fine-grained application-layer control of stream flow control. We already offer Connection::set_receive_window; a similar setter could be exposed for RecvStreams. However, I think this type of relative, buffer size oriented API is not actually suitable, since it causes every read to issue new flow control credit, which may not be appropriate in this case. Instead, we probably need a setter that allows the application to specify the maximum stream offset directly, and a similar getter on SendStream to fetch the current maximum offset... and probably to await changes to it as well.

With those in place, we should be able to support a write_chunk_at(offset: usize, ...) API that is possible to use without a deadlock hazard; I'd be happy to review a PR for such.