Netty is significantly slower than OkHttp #6696
Comments
|
|
We did observe the issue of Netty being significantly slower (~6 to 7 times slower from the result above) than OkHttp in streaming RPCs by running our transport benchmark. We have not investigated the real cause of big performance gap between Netty and OkHttp. @ejona86 was suspecting that OkHttp is doing a good job combing buffers while Netty is doing a list of small fragments, but we do not know for sure. We may have some optimization for Netty to be done in the future. We tried some workaround to improve Netty's performance for some sort. You could try to do manual flow control (like this example) with each time requesting for 5~10 response messages instead of one (change |
|
To clarify a bit, if this was write path I could believe the buffer combining. But this changes just based on the client, so this is read path. So I believe this is because OkHttp uses locks to access buffered data whereas Netty has to communicate with the network thread which adds latency. A while back I made an optimization that would help with this, but I only enabled it for server-side, so we don't know if it helps. (There was also a much older PR that did the same thing, but was considered too complicated. We've known about this for a while, but nobody has really been complaining...)
Actually, just do request(5) once, and then from then on do request(1) when you complete processing a message. This adds a 4 message "buffer" between the two threads. We saw a bit more performance improvement with request(10), but it was also diminishing returns. |
|
i was also able to reproduce the result. with the request 5 hack, netty is as fast as okhttp. with request(10), netty is about 14% faster. request(5)request(10)os: 5.6.10-arch1-1 diff |
|
base line with #5820 |
|
I found a bug in #5820, OkHttp with executor is still faster but it looks much better than before (note: Netty has significantly higher byte throughput). |
|
Hi, I noticed exactly the same issue when I compared Rest (jetty) vs GRPC (netty) and GRPC was 3-5 times slower https://github.com/AlexeyPirogov/RestVsGrpc. And GRPC-HttpOk is faster than REST (make sense). |
|
benchmark result with updated #5820. the result seems positive without noticeable regression. before: https://pastebin.com/kNLRBFvC |
|
With #5820, async streaming went from 334,472 msg/s (Netty) and 802,237 (OkHttp) to 901,221 (Netty) and 1,123,479 (OkHttp). So you can see that I made Netty faster than OkHttp, except that I made a "mistake" and made OkHttp faster as well 😄. Netty is now 80% of the OkHttp speed, which is a bit disappointing still, but probably "good enough" for now. Blocking performance however is at 704,464 for Netty and 1,298,180 OkHttp, which means Netty is at 55% of the OkHttp speed, but 75% of its speed before my change. OkHttp just generally is faster with blocking than async. Fixing blocking would mean tearing down separation of concerns to avoid this thread hop, which seems invasive and requires even more thread safety. The lower async Netty performance could be caused by several things (I have no idea if these are issues, just possible theories for investigation), for example higher CPU usage in Netty itself, or the extra work for returning WINDOW_UPDATE flow control in Netty (enqueuing a Runnable to another thread for each message vs a lock in OkHttp). These message rates are much higher than most users should need, and a very small amount of batching (e.g., using a repeated field with 5 entries vs 5 separate grpc messages) probably provides much better results for the user, even with OkHttp. Sending zero-byte or few-byte messages is just typically fairly unnecessary. So I'm going to close this issue. If users still feel driven to use OkHttp instead, then I'd please request they file another issue and tell us what sort of thing they are doing (e.g., async vs blocking, why so many messages) and we can try and dive deeper. |
|
Oh, and I wanted to point out this is all optimizing receiving. Sending could see some gains via corking. But corking has kept being not quite important enough to figure out the precise API we'd expose, as there is a wide design space available and many opinions. |
|
If you are using the async stub API, you can workaround this issue in v1.30 and earlier, by: // If client is receiving many small messages (This will not work for the blocking iterator stub)
public class MorePipeliningClientInterceptor implements ClientInterceptor {
@Override
public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(MethodDescriptor<ReqT, RespT> method,
CallOptions callOptions, Channel next) {
return new SimpleForwardingClientCall<ReqT, RespT>(next.newCall(method, callOptions)) {
@Override
public void start(Listener<RespT> responseListener, Metadata headers) {
super.start(responseListener, headers);
super.request(5);
}
};
}
}
stub = stub.withInterceptors(new MorePipeliningClientInterceptor());
...// If server is receiving many small messages
public StreamObserver<Request> someMethod(StreamObserver<Response> observer) {
ServerCallStreamObserver<Response> serverCallObserver = (ServerCallStreamObserver<Response>) observer;
serverCallObserver.request(5);
...
} |
|
The changes that fixed this issue were reverted, fixed, and then reverted again. See #7168 |
|
Should this be re-opened? |
|
As I mentioned at #7168 (comment) , MigratingDeframer seems to have a threading issue. Unclear what it is at the moment, but it will need some serious investigation. |
@ejona86 thanks for providing specific numbers. Was flow control enabled in this case or disabled? I found your posts describing how to disable Flow Control for Async Stubs (using disableAutoRequestWithInitial(1)). |
Enabled. It is always enabled.
To be clear, the MorePipeliningClientInterceptor doesn't disable flow control but requests messages earlier to hide the latency the current grpc-netty code is introducing. Unfortunately the trick doesn't work with blocking because of this buffer size. Blocking does a neat trick with ThreadlessExecutor but it means that it would notice the additional messages and could deadlock. It would be possible to work around that with some additional hackery (replacing |
|
Thank you @ejona86 . Is it possible to completely disable Flow Control without going too deep? I compared performance of batched server streaming (streaming groups of objects - 'stream repeated X') vs normal streaming ('stream X'). That is why I wanted to check if it is possible to get same performance without batches but with disabled flow control. |
As I said, flow control is always enabled. And the problem I mentioned before with the blocking stub would also apply if you disabled flow control. I'd suggest using the async stub with MorePipeliningClientInterceptor. The iterator-based blocking stub is convenient, but it isn't without its problems (no easy way to kill the RPC, trivial to leak the RPC, exceptions during iteration). If you just use the async API and put the input from resulting callbacks on a BlockingQueue you can get similar behavior without too much boilerplate. I wish you didn't need to, but it isn't that bad of an option. |
With MorePipeliningClientInterceptor, message will be out of order? Executor submit 5 message tasks, tasks will be execute concurrently. |
StreamObserver is not thread-safe, so gRPC will not call |
|
any update on this? |
|
The trouble seen by migrating deframer is suspected to be related to issues with RejectedExecutionException in ThreadlessExecutor. We keep trying to enable RejectedExecutionException to find bugs, and we keep finding and fixing bugs. The most recent attempt was reverted in #10574 . |
any "reliable" repro? I can take a peek. |
|
@natl-set, are you impacted by the worse performance? If so, there are workarounds available for both client and server: #6696 (comment) |
What version of gRPC-Java are you using?
Tried this with all of:
What is your environment?
Both:
JDK: Zulu, java 8
What did you expect to see?
Netty & Okhttp clients should have relatively similar performance.
What did you see instead?
OkHttp is about 5x faster than Netty.
Steps to reproduce the bug
See methodology here: #6685
In those experiments, a server-side streaming RPC is set up where Netty is benchmarked at receiving 40k messages/sec while OkHttp receives 200k messages/sec (with low latency).
The text was updated successfully, but these errors were encountered: