🚀 Servers Benchmark

1 - Java 20 - Tomcat 10.1
2 - Java 20 - Netty 4
3 - Java 20 - Netty 4 Reactive
4 - Java 21 - Netty 4 - GraalVM 21 - Boot 3x
5 - Java 20 - Undertow
6 - Java 20 - Micronaut
7 - Java 20 - Quarkus
8 - Java 21 - Smart Http
9 - Java 21 - nio-iouring
10 - Kotlin 1.9 - Ktor
11 - NodeJS 20 -Express
12 - Deno 1.37 - Fresh
13 - Bun 1.0 - Elysia
14 - Bun 1.0 - Hono
15 - Python 3.11 - Twisted
16 - Python 3.11 - Tornado
17 - V 0.4 - pico
18 - C++ - Drogon
19 - Go 1.21 - net/http
20 - Rust 1.71 - Actix
21 - Rust 1.71 - Axum
22 - Rust 1.71 - may_minihttp
23 - Zig 0.11 - Zap
24 - Julia 1.8.5 - Genie
25 - Nim 2 - httpbeast
26 - Ocaml 5.1 - http_async
27 - PHP 8.1 - embedded server
28 - PHP 8.1 - Nginx
29 - Scala 3.0 - JDK 21 - Zio Http
30 - Scala 3.0 - JDK 21 - Play 2.9
31 - Scala 3.0 - JDK 21 - Akka-Http

🚀 Servers Benchmark - Contributions

1 - Rust 1.71 - Hyper (@andreixmartins)
2 - Lua 5.4 - Pegasus (@andreixmartins)
3 - Ruby 3.2 - TCP (@andreixmartins)
4 - Java 21 - default HTTP Server (@alex-carvalho)
5 - Bun http - (@mrarticuno)
6 - Bun http workers - (@mrarticuno)
7 - Nodejs 20 http - (@mrarticuno)
8 - Nodejs 20 http workers - (@mrarticuno)
9 - Python 3.11 Flask - (@rRickson)
10 - Python 3.11 fast-api-uvicorn - (@rRickson)
11 - Python 3.11 fast-api-hypercorn - (@rRickson)
12 - Java 21 - Quarkus Reactive - (@dadpig)
13 - Dart 3.1 - Shelf - (@brscherer)
14 - Java 17 - Quarkus Native - Mandrel Podman (@andremayer)
15 - Clojure 1.11 - Ring/Jetty - (@deividfsantos)
16 - Haskell Scotty - (@deividfsantos)
17 - Elixir - Phoenix/Cowboy - (@deividfsantos)
18 - C - (@Thorugoh)
19 - Racket - Spin (@kilpp)
20 - Crystal 1.9.2 - Spider - (@LucasKonrath)
21 - Java 8 - Blade - (@LucasKonrath)
22 - Gleam - Mist - (@LucasKonrath)
23 - Rust 1.7.1 - Gotham - (@andreixmartins)
24 - Rust 1.72.1 - Salvo (@andremayer)
25 - Rust 1.74.0-nightly - xitca (@andremayer)
26 - Haskell - Warp (@brunohaetinger)
27 - Kotlin -Http4k (@codegik)

Gatling reports 📈

Gatling reports - Contributions 📈

Throwing a little bit of Salt 🧂

Things that this should be doing to be better.

Wram up
Run on AWS
Have 2 different machines for server and gatling
Make more rounds with more users, 10k, 100k, 1M
Have other uses cases like, read json from DB, read static file, persist data, etc...

Results Summary (☕ Java)

1k users, during 1 minute non-stop

Boot-Netty : 60000 total/OK, p99: 2 ms
Boot-Tomcat : 60000 total/OK, p99: 6 ms
Boot-Undertow : 60000 total/OK, p99: 15 ms
Micronaut : 60000 total/OK, p99: 129 ms
Quarkus : 60000 total/OK, p99: 4 ms

Build All 💻

Make sure you have installed and configured on the $PATH

Java sdk 20
Kotlin 1.9.10
Zig version 0.11
Rust 1.71.0
Go 1.21.0
g++ 11.4.0
bun 1.0.0
Deno 1.37
Nodejs 20
Python 3.11
V 0.4
Julia 1.8.5
Nim 2.0
Lua 5.4
Ruby 3.2.2
Dart 3.2.1
Podman 3.4.2
Clojure 1.11 / Leiningen 2.9
Haskell 9.2 / Cabal 3.6
gcc 11.4
Racket 8.2
Ocaml 5.1 / opam 2.1 / dune 3.7
PHP 8.1
Scala 3.3.1 / sbt 1.9.6
Crystal 1.9.2
Gleam 0.31.0 / mist 0.13.2

./build-all.sh

build Java ☕

for Java - install sdkman - https://sdkman.io/

curl -s "https://get.sdkman.io" | bash

source "$HOME/.sdkman/bin/sdkman-init.sh"
sdk install java 20.0.2-amzn
sdk use java 20.0.2-amzn
java -version

❯ java -version
openjdk version "20.0.1" 2023-04-18
OpenJDK Runtime Environment Homebrew (build 20.0.1)
OpenJDK 64-Bit Server VM Homebrew (build 20.0.1, mixed mode, sharing)

./mvnw clean install

Important OS tuning

default file descriptors for linux is 1024, mac is 256 - need to be twecked.
also open files
switch to ip v4 here are steps on how to do it: https://gist.github.com/diegopacheco/ad1e63691380ad1a6b3be6b62910c3fb

for V and pico

gatling is having issues, but apache ab is working fine.

ab -n 60000 -c 1000 http://127.0.0.1:8080/

This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 6000 requests
Completed 12000 requests
Completed 18000 requests
Completed 24000 requests
Completed 30000 requests
Completed 36000 requests
Completed 42000 requests
Completed 48000 requests
Completed 54000 requests
Completed 60000 requests
Finished 60000 requests


Server Software:        
Server Hostname:        127.0.0.1
Server Port:            8080

Document Path:          /
Document Length:        36 bytes

Concurrency Level:      1000
Time taken for tests:   487.475 seconds
Complete requests:      60000
Failed requests:        0
Total transferred:      6000000 bytes
HTML transferred:       2160000 bytes
Requests per second:    123.08 [#/sec] (mean)
Time per request:       8124.588 [ms] (mean)
Time per request:       8.125 [ms] (mean, across all concurrent requests)
Transfer rate:          12.02 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   13   9.0     13      56
Processing:  7043 8026 271.9   8059   15159
Waiting:        0    7   6.6      6      69
Total:       7056 8039 272.1   8069   15179

Percentage of the requests served within a certain time (ms)
  50%   8069
  66%   8072
  75%   8075
  80%   8077
  90%   8090
  95%   8105
  98%   8126
  99%   8129
 100%  15179 (longest request)

Fix gatling for V and pic by sharing connection.

HttpProtocolBuilder httpProtocol = http
        .baseUrl("http://localhost:8080")
        .acceptHeader("text/html")
        .doNotTrackHeader("1")
        .userAgentHeader("Gat")
        .shareConnections();

Open a PR to add new language / server?

Create a gatling simulation
Create a gatling script
Create a run.sh script - add the tuning for linux os pls
Add the new project with new language/framework (1 pr per lang/framework)
Dont publish gatling results
I will re-run on my machine, you can run all in our machine in our fork to compare different results with different hardware
Add a different script to install dependencies

Note on Java 21

Java 21 there is some performance degradetion comparing spring boot 3x and netty.

Correto JDK 21

================================================================================
---- Global Information --------------------------------------------------------
> request count                                      60000 (OK=60000  KO=0     )
> min response time                                      0 (OK=0      KO=-     )
> max response time                                    276 (OK=276    KO=-     )
> mean response time                                     2 (OK=2      KO=-     )
> std deviation                                         14 (OK=14     KO=-     )
> response time 50th percentile                          1 (OK=1      KO=-     )
> response time 75th percentile                          1 (OK=1      KO=-     )
> response time 95th percentile                          1 (OK=1      KO=-     )
> response time 99th percentile                          8 (OK=8      KO=-     )
> mean requests/sec                                923.077 (OK=923.077 KO=-     )
---- Response Time Distribution ------------------------------------------------
> t < 800 ms                                         60000 (100%)
> 800 ms <= t < 1200 ms                                  0 (  0%)
> t >= 1200 ms                                           0 (  0%)
> failed                                                 0 (  0%)
================================================================================

Correto JDK 20

================================================================================
---- Global Information --------------------------------------------------------
> request count                                      60000 (OK=60000  KO=0     )
> min response time                                      0 (OK=0      KO=-     )
> max response time                                    267 (OK=267    KO=-     )
> mean response time                                     2 (OK=2      KO=-     )
> std deviation                                         13 (OK=13     KO=-     )
> response time 50th percentile                          1 (OK=1      KO=-     )
> response time 75th percentile                          1 (OK=1      KO=-     )
> response time 95th percentile                          1 (OK=1      KO=-     )
> response time 99th percentile                          7 (OK=7      KO=-     )
> mean requests/sec                                923.077 (OK=923.077 KO=-     )
---- Response Time Distribution ------------------------------------------------
> t < 800 ms                                         60000 (100%)
> 800 ms <= t < 1200 ms                                  0 (  0%)
> t >= 1200 ms                                           0 (  0%)
> failed                                                 0 (  0%)
================================================================================

Application boot time on Java 21, GraalVM 21 and Netty

0.049 seconds


  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v3.1.3)

2023-09-22T16:06:05.325-07:00  INFO 450420 --- [           main] c.g.d.sandboxspring.Application          : Starting AOT-processed Application using Java 21 with PID 450420 (/mnt/e35d88d4-42b9-49ea-bf29-c4c3b018d429/diego/git/diegopacheco/servers-benchmark/server-boot-netty/target/server-boot-netty started by diego in /mnt/e35d88d4-42b9-49ea-bf29-c4c3b018d429/diego/git/diegopacheco/servers-benchmark/server-boot-netty)
2023-09-22T16:06:05.325-07:00  INFO 450420 --- [           main] c.g.d.sandboxspring.Application          : No active profile set, falling back to 1 default profile: "default"
2023-09-22T16:06:05.361-07:00  INFO 450420 --- [           main] o.s.b.web.embedded.netty.NettyWebServer  : Netty started on port 8080
2023-09-22T16:06:05.362-07:00  INFO 450420 --- [           main] c.g.d.sandboxspring.Application          : Started Application in 0.049 seconds (process running for 0.055)
Spring Boot 3.1.x working!

Server Spec

Linux Kernel: 6.2.0-1009-lowlatency 
OS          : Ubuntu 22.04.3 LTS

inxi -Fxz

System:
  Kernel: 6.2.0-1009-lowlatency x86_64 bits: 64 compiler: N/A
    Desktop: GNOME 42.9 Distro: Ubuntu 22.04.3 LTS (Jammy Jellyfish)
Machine:
  Type: Laptop System: Avell High Performance product: Avell G1555 MUV / A62
    v: Standard serial: <superuser required>
  Mobo: Avell High Performance model: Avell G1555 MUV / A62 v: Standard
    serial: <superuser required> UEFI: American Megatrends v: N.1.04
    date: 08/13/2019
Battery:
  ID-1: BAT0 charge: 44.9 Wh (96.1%) condition: 46.7/46.7 Wh (100.0%)
    volts: 12.5 min: 11.4 model: standard status: Not charging
CPU:
  Info: 6-core model: Intel Core i7-9750H bits: 64 type: MT MCP
    arch: Coffee Lake rev: A cache: L1: 384 KiB L2: 1.5 MiB L3: 12 MiB
  Speed (MHz): avg: 3498 high: 4202 min/max: 800/4500 cores: 1: 4154
    2: 4200 3: 2600 4: 4196 5: 2600 6: 4092 7: 2600 8: 4127 9: 4202 10: 4014
    11: 2600 12: 2600 bogomips: 62399
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx
Graphics:
  Device-1: Intel CoffeeLake-H GT2 [UHD Graphics 630]
    vendor: Tongfang Hongkong driver: i915 v: kernel bus-ID: 00:02.0
  Device-2: NVIDIA TU116M [GeForce GTX 1660 Ti Mobile]
    vendor: Tongfang Hongkong driver: nouveau v: kernel bus-ID: 01:00.0
  Device-3: Logitech BRIO Ultra HD Webcam type: USB
    driver: snd-usb-audio,uvcvideo bus-ID: 1-12.4:10
  Device-4: Acer HD Webcam type: USB driver: uvcvideo bus-ID: 1-13:5
  Display: wayland server: X.Org v: 1.22.1.1 with: Xwayland v: 22.1.1
    compositor: gnome-shell driver: X: loaded: modesetting,nvidia
    unloaded: fbdev,nouveau,vesa gpu: i915,nouveau resolution:
    1: 3840x2160~60Hz 2: 1920x1080~60Hz
  OpenGL: renderer: Mesa Intel UHD Graphics 630 (CFL GT2)
    v: 4.6 Mesa 23.0.4-0ubuntu1~22.04.1 direct render: Yes
Network:
  Device-1: Intel Cannon Lake PCH CNVi WiFi driver: iwlwifi v: kernel
    bus-ID: 00:14.3
  IF: wlo1 state: up mac: <filter>
  Device-2: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet
    driver: r8169 v: kernel port: N/A bus-ID: 03:00.0
Drives:
  Local Storage: total: 1.36 TiB used: 742.63 GiB (53.1%)
  ID-1: /dev/nvme0n1 vendor: Samsung model: SSD 970 EVO Plus 500GB
    size: 465.76 GiB temp: 50.9 C
  ID-2: /dev/sda vendor: Crucial model: CT1000MX500SSD1 size: 931.51 GiB
Partition:
  ID-1: / size: 456.89 GiB used: 372.27 GiB (81.5%) fs: ext4
    dev: /dev/nvme0n1p2
  ID-2: /boot/efi size: 511 MiB used: 6.1 MiB (1.2%) fs: vfat
    dev: /dev/nvme0n1p1
Swap:
  ID-1: swap-1 type: file size: 2 GiB used: 0 KiB (0.0%) file: /swapfile
Sensors:
  System Temperatures: cpu: 84.0 C pch: 75.0 C mobo: 84.0 C gpu: nouveau
    temp: 69.0 C
  Fan Speeds (RPM): N/A
Info:
  Processes: 449 Uptime: 1d 45m Memory: 62.65 GiB used: 10.58 GiB (16.9%)
  Init: systemd runlevel: 5 Compilers: gcc: 11.4.0 clang: 16.0.4
  Packages: 4360 Shell: Zsh v: 5.8.1 inxi: 3.3.13

Gatling simulations summary

🏆 #1 - Winner
🥈 #2 - Second place
🥉 #3 - Thrid Place
🧮Criteria: Number of Requests, Max and p99 (usually max is what define it)

#	Simulation	Reqs	Min	Max	Mean	p50	p75	p95	p99
🏆	Nim2beast	60000	0	36	0	0	0	1	1
🥈	JavaIOUring	60000	0	40	0	0	1	1	1
🥉	VPicov	60000	0	43	0	0	0	1	1
4	RustXitca	60000	0	44	0	0	1	1	1
5	Ruby	60000	0	45	0	0	1	1	1
6	PHPEmbededServer	60000	0	46	0	0	1	1	1
7	BunHono	60000	0	46	0	0	1	1	1
8	Go	60000	0	48	0	0	0	1	1
9	RustMayMinihttp	60000	0	48	0	0	1	1	4
10	ZigZap	60000	0	52	0	0	0	1	1
11	BunHttpServer	60000	0	56	0	0	1	1	1
12	NodeJSHttpServer	60000	0	57	0	0	0	1	1
13	RustSalvo	60000	0	57	0	0	1	1	2
14	CScratch	60000	0	59	0	0	1	1	1
15	RustHyper	60000	0	61	0	0	1	1	1
16	HaskellScotty	60000	0	61	0	0	1	1	5
17	RustAxum	60000	0	63	0	0	0	1	2
18	RustActix	60000	0	65	0	0	1	1	1
19	BunElysia	60000	0	67	0	0	1	1	2
20	HaskellWarp	60000	0	68	0	0	0	1	4
21	RustGotham	60000	0	77	0	0	0	1	1
22	GleamMist	60000	0	78	0	0	1	1	1
23	CppDrogon	60000	0	79	0	0	1	1	2
24	Netty	60000	0	84	0	0	1	1	2
25	NodeJSExpress	60000	0	84	1	0	1	1	2
26	NodeJSHttpServerWorker	60000	0	88	1	0	1	1	3
27	Netty	60000	0	91	1	0	1	1	9
28	BunHttpServerWorker	60000	0	96	0	0	1	1	2
29	PHPNginx	60000	0	99	0	0	1	1	1
30	DenoFresh	60000	0	106	1	0	1	1	2
31	PythonTwisted	60000	0	106	2	1	1	3	35
32	JavaSmartHttp	60000	0	114	0	0	1	1	1
33	QuarkusNative	60000	0	114	1	0	1	1	18
34	Http4k	60000	0	200	1	0	1	1	1
35	QuarkusReact	60000	0	220	1	1	1	1	5
36	NettyMono2	60000	0	236	1	1	1	1	5
37	Quarkus	60000	0	266	1	1	1	1	4
38	Undertow	60000	0	325	1	0	1	1	15
39	JavaBlade	60000	0	356	1	0	0	1	3
40	Tomcat	60000	0	360	1	0	1	1	6
41	Micronaut	60000	0	468	4	0	1	2	129
42	ClojureRingJetty	60000	0	481	16	1	2	120	323
43	NettyMono	60000	0	545	7	0	1	1	335
44	Ktor	60000	0	696	6	0	1	3	245
45	Scala3ZioHttp	60000	0	888	5	0	1	1	222
46	Lua	60000	0	1033	0	0	1	1	1
47	CrystalSpiderGazelle	60000	0	1041	5	1	1	32	83
48	DartShelf	60000	0	1068	3	1	1	6	55
49	RacketSpin	60000	0	1103	51	3	35	308	627
50	OcamlHttpAsync	60000	0	1106	1	0	1	2	6
51	PythonTornado	60000	0	1306	30	1	19	117	347
52	Scala3AkkaHttp	60000	0	1870	7	0	1	1	6
53	ElixirPhoenixCowboy	60000	248	1985	735	695	851	1139	1563
54	Scala3Play2	60000	0	2764	91	0	1	11	2314
55	PythonFlask	60000	0	3135	14	1	1	104	143
56	JavaHttpServer	60000	0	3194	18	0	1	3	568
57	PythonFastApi	60000	53	10340	4486	3948	7229	9709	10032
58	JuliaGenie	57929	9	10971	721	12	12	7199	8634
59	PythonFastApiHypercorn	56878	26	45622	25555	29579	37271	40736	41760

Name		Name	Last commit message	Last commit date
Latest commit History 211 Commits
.clj-kondo/.cache/v1		.clj-kondo/.cache/v1
.github		.github
.lsp/.cache		.lsp/.cache
.mvn/wrapper		.mvn/wrapper
bun-elysia		bun-elysia
bun-hono		bun-hono
bun-server-workers		bun-server-workers
bun-server		bun-server
c-scratch		c-scratch
clojure-ring-jetty		clojure-ring-jetty
cpp-drogon		cpp-drogon
crystal-spider-gazelle		crystal-spider-gazelle
dart-shelf		dart-shelf
deno-fresh		deno-fresh
elixir_phoenix_cowboy		elixir_phoenix_cowboy
gatling-benchmarks		gatling-benchmarks
gleam-mist		gleam-mist
go-nethttp		go-nethttp
haskell-scotty		haskell-scotty
haskell-warp		haskell-warp
java-blade		java-blade
java-httpserver		java-httpserver
java-nio_uring		java-nio_uring
java-smart-http		java-smart-http
julia-genie		julia-genie
k6-benchmarks		k6-benchmarks
kotlin-http4k		kotlin-http4k
kotlin-ktor		kotlin-ktor
lua-pegasus		lua-pegasus
native-quarkus		native-quarkus
nim-httpbeast		nim-httpbeast
nodejs-20-express		nodejs-20-express
nodejs-20-http-server-no-dependencies-workers		nodejs-20-http-server-no-dependencies-workers
nodejs-20-http-server-no-dependencies		nodejs-20-http-server-no-dependencies
ocaml-http_async/project		ocaml-http_async/project
php-embeded-server		php-embeded-server
php-nginx		php-nginx
play-2.9-scala-3x-jdk21		play-2.9-scala-3x-jdk21
python-fast-api-hypercorn		python-fast-api-hypercorn
python-fast-api-uvicorn		python-fast-api-uvicorn
python-flask		python-flask
python-tornado		python-tornado
python-twisted		python-twisted
racket-spin		racket-spin
reactive-quarkus		reactive-quarkus
ruby-http		ruby-http
rust-actix		rust-actix
rust-axum		rust-axum
rust-gotham		rust-gotham
rust-hyper		rust-hyper
rust-may_minihttp		rust-may_minihttp
rust-salvo		rust-salvo
rust-xitca		rust-xitca
scala3-akka-http		scala3-akka-http
scala3-zio-http		scala3-zio-http
server-boot-netty		server-boot-netty
server-boot-tomcat		server-boot-tomcat
server-boot-undertow		server-boot-undertow
server-micronaut		server-micronaut
server-quarkus		server-quarkus
vpicov		vpicov
zig-zap		zig-zap
.gitignore		.gitignore
.sdkmanrc		.sdkmanrc
LICENSE		LICENSE
README.md		README.md
build-all.sh		build-all.sh
mvnw		mvnw
mvnw.cmd		mvnw.cmd
pom.xml		pom.xml

License

diegopacheco/servers-benchmark

Folders and files

Latest commit

History