Add a benchmark executable #79

Joannis · 2023-07-07T08:18:24Z

=================
ProtocolBenchmark
=================

RESP Array Parsing
╒════════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                     │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞════════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Malloc (total)             │       1 │       1 │       1 │       1 │       1 │       1 │       1 │   10000 │
├────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Memory (resident peak) (M) │      19 │      19 │      19 │      19 │      19 │      19 │      19 │   10000 │
├────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Throughput (# / s) (K)     │    1264 │    1143 │    1143 │    1090 │    1000 │     292 │      27 │   10000 │
├────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Time (total CPU) (μs)      │      20 │      21 │      21 │      22 │      22 │      38 │     108 │   10000 │
├────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Time (wall clock) (ns)     │     791 │     875 │     875 │     917 │    1000 │    3417 │   36375 │   10000 │
╘════════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

RESP Conversation
╒════════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                     │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞════════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Malloc (total)             │      96 │      96 │      96 │      96 │      96 │      96 │      96 │   10000 │
├────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Memory (resident peak) (M) │      20 │      20 │      20 │      20 │      20 │      20 │      20 │   10000 │
├────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Throughput (# / s) (K)     │      51 │      48 │      48 │      47 │      44 │       9 │       0 │   10000 │
├────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Time (total CPU) (μs)      │      45 │      48 │      48 │      49 │      52 │     109 │     300 │   10000 │
├────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Time (wall clock) (μs)     │      19 │      20 │      20 │      21 │      22 │     108 │    8323 │   10000 │
╘════════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

RESP3 Array Parsing
╒════════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                     │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞════════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Malloc (total)             │       0 │       0 │       0 │       0 │       0 │       0 │       0 │   10000 │
├────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Memory (resident peak) (M) │      19 │      19 │      19 │      19 │      19 │      19 │      19 │   10000 │
├────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Throughput (# / s) (M)     │      24 │      12 │      11 │       8 │       8 │       0 │       0 │   10000 │
├────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Time (total CPU) (μs)      │      19 │      20 │      20 │      20 │      21 │      38 │      64 │   10000 │
├────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Time (wall clock) (ns)     │      41 │      83 │      84 │     125 │     125 │    1375 │   77083 │   10000 │
╘════════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

RESP3 Conversation
╒════════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Metric                     │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞════════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ Malloc (total)             │      58 │      58 │      58 │      58 │      58 │      58 │      58 │   10000 │
├────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Memory (resident peak) (M) │      19 │      19 │      19 │      19 │      19 │      19 │      19 │   10000 │
├────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Throughput (# / s) (K)     │      86 │      81 │      81 │      80 │      78 │      21 │       0 │   10000 │
├────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Time (total CPU) (μs)      │      35 │      37 │      37 │      38 │      38 │      73 │     160 │   10000 │
├────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Time (wall clock) (μs)     │      11 │      12 │      12 │      12 │      12 │      46 │    1997 │   10000 │
╘════════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

fabianfett

2 small first notes

Package.swift

Sources/RediStackPerformanceTester/main.swift

fabianfett

Two questions...

Sources/RESP3/RESP3Token.swift

Sources/RediStackPerformanceTester/main.swift

…ing errors when unwrapping the value

hassila · 2023-12-20T14:43:21Z

Benchmarks/Package.swift

+    ],
+    dependencies: [
+        .package(path: "../"),
+        .package(url: "https://github.com/ordo-one/package-benchmark.git", exact: "1.11.1"),


I'd suggest to use from:, or at least bump to a recent version (.e.g, 1.18.0) - 1.11.1 is quite old by now and NIO has from rather than exact.

I realised this too, but had an issue with the latest build. I don't remember what it was, I believe my SPM caches were broken

Ok, @Joannis let me know if you have any issues with the package itself, 1.20 just released with quite a few fixes visavi 1.11.

hassila · 2023-12-20T14:46:55Z

Benchmarks/Benchmarks/ProtocolBenchmark/ProtocolBenchmark.swift

+    let resp3ArrayValueBuffer = ByteBuffer(string: "*2\r\n$3\r\nGET\r\n$7\r\nwelcome\r\n")
+    let resp3ArrayCount = 2
+
+    Benchmark("RESP3 Array Parsing") { benchmark in


Suggested change

Benchmark("RESP3 Array Parsing") { benchmark in

Benchmark("RESP3 Array Parsing", configuration: .init(scalingFactor: .kilo)) { benchmark in

for _ in benchmark.scaledIterations {

try runRESP3ArrayParsing(

valueBuffer: resp3ArrayValueBuffer,

valueCount: resp3ArrayCount

)

}

Would suggest using such scaling factor (.kilo) for the RESP Array Parsing and RESP3 Array Parsing benchmarks, as they have very short runtimes. The scaling factor is used as an inner loop and the results will be scaled up accordingly for throughput if you update to a recent benchmark version.

fabianfett reviewed Jul 7, 2023

View reviewed changes

Package.swift Outdated Show resolved Hide resolved

Sources/RediStackPerformanceTester/main.swift Outdated Show resolved Hide resolved

Joannis marked this pull request as ready for review October 21, 2023 22:49

Joannis requested a review from fabianfett October 21, 2023 22:49

fabianfett reviewed Oct 27, 2023

View reviewed changes

Sources/RESP3/RESP3Token.swift Show resolved Hide resolved

Sources/RediStackPerformanceTester/main.swift Outdated Show resolved Hide resolved

Joannis added 6 commits December 16, 2023 11:55

Add a benchmark executable

a1abe65

Add license headers

f7bed03

Remove RediStackPerformanceTester from the package's products

600d239

Add a variant of RESP3Token that does not do validation, while provid…

f384171

…ing errors when unwrapping the value

Fix the unit tests

6ca5f57

Add benchmarks based on ordo-one's package-benchmark

4f6ae6f

Joannis force-pushed the feature/jo/resp-benchmarker branch from 996b843 to 4f6ae6f Compare December 16, 2023 10:55

Joannis requested a review from fabianfett December 16, 2023 10:56

hassila reviewed Dec 20, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a benchmark executable #79

Add a benchmark executable #79

Joannis commented Jul 7, 2023 •

edited

fabianfett left a comment

fabianfett left a comment

hassila Dec 20, 2023

Joannis Dec 20, 2023

hassila Dec 21, 2023

hassila Dec 20, 2023

hassila Dec 20, 2023

-    Benchmark("RESP3 Array Parsing") { benchmark in
+    Benchmark("RESP3 Array Parsing", configuration: .init(scalingFactor: .kilo)) { benchmark in
+       for _ in benchmark.scaledIterations {
+        try runRESP3ArrayParsing(
+             valueBuffer: resp3ArrayValueBuffer,
+             valueCount: resp3ArrayCount
+         )
+         }

Add a benchmark executable #79

Are you sure you want to change the base?

Add a benchmark executable #79

Conversation

Joannis commented Jul 7, 2023 • edited

fabianfett left a comment

Choose a reason for hiding this comment

fabianfett left a comment

Choose a reason for hiding this comment

hassila Dec 20, 2023

Choose a reason for hiding this comment

Joannis Dec 20, 2023

Choose a reason for hiding this comment

hassila Dec 21, 2023

Choose a reason for hiding this comment

hassila Dec 20, 2023

Choose a reason for hiding this comment

hassila Dec 20, 2023

Choose a reason for hiding this comment

Joannis commented Jul 7, 2023 •

edited