Spring grpc vs vanilla grpc. Low performance

[dev-comrade]

Hi there.

I have a couple high load services in my company on vanilla java + grpc. At the moment i want to migrate them to springboot.

So when i wrote the one service on spring i discovered that it has very low performance and it's more than twice as slow.... I tested it with jmh and ghz.

Proto file for tests is

package comrade.test.proto;
option java_multiple_files = true;

service Comrade {
  rpc testComrade (ComradeRequest) returns (ComradeResponse) {}
}

message ComradeRequest {
  string name = 1;
}

message ComradeResponse {
  string comradeInfo = 2;
}

And results from ghz on springboot service

Summary:
  Count:        2937782
  Total:        60.00 s
  Slowest:      95.40 ms
  Fastest:      0.27 ms
  Average:      1.00 ms
  Requests/sec: 48962.08

Response time histogram:
  0.269 [1]     |
  9.782 [999737]        |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  19.295 [203]  |
  28.809 [8]    |
  38.322 [1]    |
  47.835 [0]    |
  57.348 [0]    |
  66.861 [0]    |
  76.374 [0]    |
  85.888 [0]    |
  95.401 [50]   |

Latency distribution:
  10 % in 0.81 ms 
  25 % in 0.87 ms 
  50 % in 0.96 ms 
  75 % in 1.06 ms 
  90 % in 1.21 ms 
  95 % in 1.45 ms 
  99 % in 1.92 ms 

Status code distribution:
  [OK]            2937774 responses   
  [Unavailable]   8 responses         

Error distribution:
  [8]   rpc error: code = Unavailable desc = transport is closing   

And vanilla service

Summary:
  Count:        6330372
  Total:        60.00 s
  Slowest:      79.71 ms
  Fastest:      0.07 ms
  Average:      0.45 ms
  Requests/sec: 105505.78

Response time histogram:
  0.067 [1]     |
  8.031 [999776]        |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  15.995 [156]  |
  23.960 [17]   |
  31.924 [0]    |
  39.888 [0]    |
  47.853 [0]    |
  55.817 [0]    |
  63.781 [0]    |
  71.745 [0]    |
  79.710 [50]   |

Latency distribution:
  10 % in 0.34 ms 
  25 % in 0.38 ms 
  50 % in 0.43 ms 
  75 % in 0.48 ms 
  90 % in 0.55 ms 
  95 % in 0.67 ms 
  99 % in 1.49 ms 

Status code distribution:
  [OK]            6330343 responses   
  [Unavailable]   29 responses        

Error distribution:
  [29]   rpc error: code = Unavailable desc = transport is closing   

So could you answer me it is normal for spring? Or maybe i did something wrong...

I can show you a project with code for tests if you want.

[jvmlet]

Yes, please share the load test project.
I hope you stripped down all the addons (security , validation ect) when comparing the results.

[dev-comrade]

I hope you stripped down all the addons (security , validation ect)

Unfortunately i didn't because i didn't know how to tune spring app performance :( I just create default spring boot application with grpc-spring-boot-starter and ran tests. If you have some recommendation it will be very useful for me

My repo with tests pringboot-grpc-benchmark

[jvmlet]

You are using another starter

<dependency>
            <groupId>net.devh</groupId>
            <artifactId>grpc-spring-boot-starter</artifactId>
            <version>2.13.1.RELEASE</version>
 </dependency>

I wonder what will the the performance test results if you use the right starter 😁

[dev-comrade]

Sorry ) Fixed it to the right starter

[jvmlet]

So now it's 16K/s vs 10K/s ?

[dev-comrade]

Not really..

If we told about test with using 1 cpu and 1 concurency results is: 16K/s vs 10K/s

But if we remove restrictions:

Run ghz in scope **vanilla** with port 62004 ---- 
Benchmark will be stopped after 1 minute ---- 

Summary:
  Name:         vanilla
  Count:        6349259
  Total:        60.00 s
  Slowest:      6.70 ms
  Fastest:      0.07 ms
  Average:      0.45 ms
  Requests/sec: 105820.55

Response time histogram:
  0.068 [1]     |
  0.731 [971853]        |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  1.394 [23187] |∎
  2.057 [4471]  |
  2.720 [374]   |
  3.383 [61]    |
  4.046 [3]     |
  4.709 [0]     |
  5.373 [0]     |
  6.036 [4]     |
  6.699 [46]    |

Latency distribution:
  10 % in 0.34 ms 
  25 % in 0.38 ms 
  50 % in 0.43 ms 
  75 % in 0.49 ms 
  90 % in 0.55 ms 
  95 % in 0.61 ms 
  99 % in 1.09 ms 

Status code distribution:
  [Canceled]      2 responses         
  [Unavailable]   40 responses        
  [OK]            6349217 responses   

Error distribution:
  [2]    rpc error: code = Canceled desc = grpc: the client connection is closing   
  [40]   rpc error: code = Unavailable desc = transport is closing      

VS

Run ghz in scope **spring** with port 61004 ---- 
Benchmark will be stopped after 1 minute ---- 

Summary:
  Name:         spring
  Count:        2970661
  Total:        60.00 s
  Slowest:      6.36 ms
  Fastest:      0.25 ms
  Average:      0.99 ms
  Requests/sec: 49510.77

Response time histogram:
  0.248 [1]     |
  0.859 [137784]        |∎∎∎∎∎∎∎
  1.469 [847142]        |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  2.080 [11557] |∎
  2.691 [2865]  |
  3.302 [596]   |
  3.913 [5]     |
  4.524 [0]     |
  5.134 [0]     |
  5.745 [0]     |
  6.356 [50]    |

Latency distribution:
  10 % in 0.84 ms 
  25 % in 0.89 ms 
  50 % in 0.95 ms 
  75 % in 1.03 ms 
  90 % in 1.11 ms 
  95 % in 1.17 ms 
  99 % in 1.62 ms 

Status code distribution:
  [OK]            2970642 responses   
  [Unavailable]   19 responses        

Error distribution:
  [19]   rpc error: code = Unavailable desc = transport is closing   

and I don't know why the spring is slower, more than twice....

[jvmlet]

Right, it shouldn't be . Other than bootstrap, the performance should be same. Are you sure its same version of grpc, same configuration?

[jvmlet]

1.32.1 vs 1.42.0... lets use the same versions

[dev-comrade]

done.

updated all dependencies which i can find, but it's nothing change

[skjolber]

I realize test project is in a very early stage, some suggestions:

• add GRPC methods for streaming
• reduce noise: send static data, no random data and such
• R2dbcAutoConfiguration should not be enabled?
• Server is created with all cores, but that could potentially interfere with the client?
• Embedd server directly in jmh project. Put client in a new module and clone jmh project if the two implementations cannot coexist.
• Reenable the warmpup
• Add a main method to simplify testing
• Use json + jmh visualizer as output
• Add docs suggestion setting boost off.
• Use visualvm cpu sampling to detect some of the hotspots

[jvmlet]

Few additions :

• Remove reactor-grpc-stub plugin , we want to be as close as possible to vanilla grpc
• Remove all cloud dependencies