Streaming PromQL engine: binary arithmetic operations with one-to-one matching #8096

charleskorn · 2024-05-10T03:43:06Z

What this PR does

This PR adds support for binary arithmetic operations with one-to-one matching to the streaming PromQL engine.

This means it now supports expressions like the following:

metric_a + metric_b
metric_a - on (env) metric_b
metric_a * ignoring (env) metric_b

It also supports binary operations with the other supported expression types, for example rate(metric_a[1m]) + rate(metric_b[1m]) and sum(metric_a + metric_b).

The +, -, *, /, %, ^ and atan2 operations are supported.

Comparison operations like == and >, and logical operations like or and unless are not supported. One-to-many and many-to-one matching with group_left or group_right are also not supported. These features will be added in future PRs.

Performance

In general, the streaming engine is substantially faster than Prometheus' engine, particularly for queries that select many series or evaluate many steps - around 60-70% faster in many cases. Similarly, the streaming engine has far lower peak memory utilisation in all but one scenario, with 60-70% lower memory utilisation for queries that select many series and many steps.

The only scenario where peak memory utilisation is not lower with the streaming engine is for queries where one side of the binary operator has more series than the other (a_100{l=~"[13579]."} - b_100 and a_2000{l=~"1..."} - b_2000 below). I haven't yet been able to get to the bottom of this, and will follow up with a separate PR to fix the issue rather than wait and make this PR even bigger.

Detailed benchmark comparison with Prometheus' engine

First section below shows latency, second section shows peak memory utilisation.

a_X and b_X in the queries below are metrics with X series.

                                                                       │      standard      │              streaming              │
                                                                       │       sec/op       │    sec/op     vs base               │
Query/a_1_-_b_1,_instant_query                                                 225.9µ ±  1%   250.8µ ±  6%  +11.00% (p=0.002 n=6)
Query/a_1_-_b_1,_range_query_with_100_steps                                    260.3µ ±  0%   260.6µ ± 14%        ~ (p=1.000 n=6)
Query/a_1_-_b_1,_range_query_with_1000_steps                                   618.3µ ±  2%   365.8µ ±  2%  -40.83% (p=0.002 n=6)
Query/a_100_-_b_100,_instant_query                                             1.182m ±  1%   1.159m ± 16%        ~ (p=0.065 n=6)
Query/a_100_-_b_100,_range_query_with_100_steps                                4.437m ±  1%   2.039m ±  2%  -54.04% (p=0.002 n=6)
Query/a_100_-_b_100,_range_query_with_1000_steps                               33.51m ±  0%   10.20m ±  1%  -69.56% (p=0.002 n=6)
Query/a_2000_-_b_2000,_instant_query                                           15.92m ±  1%   15.40m ±  1%   -3.28% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_100_steps                              82.01m ±  1%   32.78m ±  1%  -60.02% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_1000_steps                             735.9m ±  1%   185.6m ±  0%  -74.78% (p=0.002 n=6)
Query/a_1_-_b_1,_range_query_with_10000_steps                                  3.298m ±  1%   1.279m ±  1%  -61.22% (p=0.002 n=6)
Query/a_100_-_b_100,_range_query_with_10000_steps                             322.21m ±  0%   86.60m ±  1%  -73.12% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_10000_steps                             7.571 ±  0%    1.728 ±  6%  -77.17% (p=0.002 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_instant_query                              1.114m ±  1%   1.092m ±  3%   -1.98% (p=0.041 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_100_steps                 3.182m ±  1%   1.773m ±  1%  -44.27% (p=0.002 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_1000_steps               21.739m ±  0%   8.026m ±  3%  -63.08% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_instant_query                               12.093m ±  1%   9.258m ±  1%  -23.44% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps                   52.50m ±  1%   18.50m ±  2%  -64.76% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps                  470.5m ±  1%   103.1m ±  1%  -78.09% (p=0.002 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_instant_query                             232.9µ ±  1%   240.4µ ±  2%   +3.22% (p=0.002 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_100_steps                276.6µ ±  2%   254.3µ ±  3%   -8.08% (p=0.002 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_1000_steps               710.4µ ±  2%   360.5µ ±  3%  -49.25% (p=0.002 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_instant_query                         1.150m ±  1%   1.115m ±  1%   -3.03% (p=0.002 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_100_steps            5.453m ±  1%   2.302m ±  1%  -57.77% (p=0.002 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_1000_steps           43.07m ±  1%   11.64m ±  1%  -72.98% (p=0.002 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_instant_query                       15.28m ±  1%   14.94m ±  2%   -2.26% (p=0.015 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_100_steps         102.03m ±  1%   37.98m ±  1%  -62.78% (p=0.002 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_1000_steps         928.4m ±  1%   216.2m ±  3%  -76.71% (p=0.002 n=6)
Query/sum(a_1_+_b_1),_instant_query                                            225.9µ ±  1%   233.0µ ± 18%   +3.17% (p=0.002 n=6)
Query/sum(a_1_+_b_1),_range_query_with_100_steps                               266.8µ ±  2%   263.6µ ±  9%        ~ (p=0.310 n=6)
Query/sum(a_1_+_b_1),_range_query_with_1000_steps                              650.4µ ±  2%   351.5µ ±  3%  -45.95% (p=0.002 n=6)
Query/sum(a_100_+_b_100),_instant_query                                        1.189m ±  1%   1.165m ±  2%   -2.04% (p=0.002 n=6)
Query/sum(a_100_+_b_100),_range_query_with_100_steps                           4.504m ±  1%   2.158m ± 17%  -52.07% (p=0.002 n=6)
Query/sum(a_100_+_b_100),_range_query_with_1000_steps                          34.07m ±  1%   10.51m ±  1%  -69.16% (p=0.002 n=6)
Query/sum(a_2000_+_b_2000),_instant_query                                      15.94m ±  1%   15.51m ±  5%        ~ (p=0.310 n=6)
Query/sum(a_2000_+_b_2000),_range_query_with_100_steps                         83.55m ±  2%   33.44m ±  2%  -59.97% (p=0.002 n=6)
Query/sum(a_2000_+_b_2000),_range_query_with_1000_steps                        764.1m ±  0%   191.0m ±  2%  -75.00% (p=0.002 n=6)

                                                                       │      standard      │               streaming                │
                                                                       │         B          │       B         vs base                │
Query/a_1_-_b_1,_instant_query                                                72.75Mi ±  1%    72.34Mi ±  2%         ~ (p=0.132 n=6)
Query/a_1_-_b_1,_range_query_with_100_steps                                   71.61Mi ±  1%    71.88Mi ±  1%         ~ (p=0.485 n=6)
Query/a_1_-_b_1,_range_query_with_1000_steps                                  68.48Mi ±  1%    70.20Mi ±  1%    +2.51% (p=0.002 n=6)
Query/a_100_-_b_100,_instant_query                                            66.76Mi ±  1%    66.43Mi ±  2%         ~ (p=0.420 n=6)
Query/a_100_-_b_100,_range_query_with_100_steps                               67.09Mi ±  1%    66.52Mi ±  1%    -0.84% (p=0.006 n=6)
Query/a_100_-_b_100,_range_query_with_1000_steps                              73.14Mi ±  3%    69.22Mi ±  1%    -5.36% (p=0.002 n=6)
Query/a_2000_-_b_2000,_instant_query                                          69.03Mi ±  1%    69.47Mi ±  1%         ~ (p=0.619 n=6)
Query/a_2000_-_b_2000,_range_query_with_100_steps                             89.50Mi ±  3%    78.26Mi ±  1%   -12.56% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_1000_steps                            213.2Mi ±  0%    139.5Mi ±  1%   -34.56% (p=0.002 n=6)
Query/a_1_-_b_1,_range_query_with_10000_steps                                 68.86Mi ±  1%    70.04Mi ±  2%    +1.71% (p=0.026 n=6)
Query/a_100_-_b_100,_range_query_with_10000_steps                             158.6Mi ±  5%    113.2Mi ±  5%   -28.65% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_10000_steps                          1600.8Mi ± 11%    460.1Mi ±  3%   -71.26% (p=0.002 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_instant_query                             67.02Mi ±  1%    66.43Mi ±  2%         ~ (p=0.420 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_100_steps                66.80Mi ±  1%    66.62Mi ±  1%         ~ (p=1.000 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_1000_steps               70.56Mi ±  1%    68.10Mi ±  1%    -3.49% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_instant_query                               68.22Mi ±  3%   139.30Mi ±  3%  +104.20% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps                  81.85Mi ±  1%   111.61Mi ± 37%   +36.36% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps                 189.6Mi ±  6%    175.0Mi ± 21%    -7.72% (p=0.004 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_instant_query                            72.02Mi ±  1%    72.05Mi ±  1%         ~ (p=0.894 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_100_steps               71.26Mi ±  1%    71.98Mi ±  1%         ~ (p=0.132 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_1000_steps              67.88Mi ±  1%    69.65Mi ±  1%    +2.60% (p=0.004 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_instant_query                        66.96Mi ±  1%    65.92Mi ±  1%    -1.55% (p=0.009 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_100_steps           67.11Mi ±  1%    66.75Mi ±  2%         ~ (p=0.143 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_1000_steps          72.39Mi ±  3%    68.47Mi ±  2%    -5.42% (p=0.002 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_instant_query                      69.16Mi ±  2%    68.61Mi ±  1%         ~ (p=0.058 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_100_steps         86.86Mi ±  4%    77.52Mi ±  2%   -10.75% (p=0.002 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_1000_steps        211.6Mi ±  0%    139.5Mi ±  2%   -34.11% (p=0.002 n=6)
Query/sum(a_1_+_b_1),_instant_query                                           72.75Mi ±  1%    72.49Mi ±  1%         ~ (p=0.370 n=6)
Query/sum(a_1_+_b_1),_range_query_with_100_steps                              71.85Mi ±  1%    72.67Mi ±  1%    +1.14% (p=0.002 n=6)
Query/sum(a_1_+_b_1),_range_query_with_1000_steps                             68.41Mi ±  1%    74.67Mi ±  1%    +9.15% (p=0.002 n=6)
Query/sum(a_100_+_b_100),_instant_query                                       66.73Mi ±  1%    66.48Mi ±  1%         ~ (p=0.310 n=6)
Query/sum(a_100_+_b_100),_range_query_with_100_steps                          66.50Mi ±  1%    66.18Mi ±  1%         ~ (p=0.132 n=6)
Query/sum(a_100_+_b_100),_range_query_with_1000_steps                         73.99Mi ±  1%    65.91Mi ±  2%   -10.92% (p=0.002 n=6)
Query/sum(a_2000_+_b_2000),_instant_query                                     69.02Mi ±  1%    68.75Mi ±  1%         ~ (p=0.071 n=6)
Query/sum(a_2000_+_b_2000),_range_query_with_100_steps                        89.06Mi ±  2%    69.47Mi ±  3%   -22.00% (p=0.002 n=6)
Query/sum(a_2000_+_b_2000),_range_query_with_1000_steps                      213.24Mi ±  0%    78.56Mi ±  2%   -63.16% (p=0.002 n=6)

Fix for issue where peak memory utilisation is much higher than Prometheus' engine for queries that select many series

This PR also fixes the issue where latency and peak memory utilisation is much higher than Prometheus' engine for queries that select many series. Peak memory utilisation for these queries is now near identical in these cases, while the streaming engine is still significantly faster in all benchmarks. See the commit message on 825b4dc for more explanation of the fix.

Detailed benchmark comparison with Prometheus' engine

First section below shows latency, second section shows peak memory utilisation.

a_X and b_X in the queries below are metrics with X series.

                                                                       │      standard      │              streaming              │
                                                                       │       sec/op       │    sec/op     vs base               │
Query/a_1,_instant_query                                                       127.9µ ±  5%   120.6µ ±  4%   -5.72% (p=0.004 n=6)
Query/a_1,_range_query_with_100_steps                                          141.8µ ± 15%   131.8µ ±  8%        ~ (p=0.132 n=6)
Query/a_1,_range_query_with_1000_steps                                         181.0µ ±  2%   171.4µ ±  1%   -5.33% (p=0.002 n=6)
Query/a_100,_instant_query                                                     577.7µ ±  7%   558.4µ ±  3%   -3.35% (p=0.002 n=6)
Query/a_100,_range_query_with_100_steps                                        1.060m ±  5%   1.039m ±  5%        ~ (p=0.240 n=6)
Query/a_100,_range_query_with_1000_steps                                       5.276m ±  1%   5.191m ±  2%   -1.61% (p=0.015 n=6)
Query/a_2000,_instant_query                                                    7.207m ±  0%   7.015m ±  1%   -2.65% (p=0.002 n=6)
Query/a_2000,_range_query_with_100_steps                                       15.88m ±  1%   15.39m ±  1%   -3.08% (p=0.002 n=6)
Query/a_2000,_range_query_with_1000_steps                                      94.69m ±  1%   90.68m ±  0%   -4.24% (p=0.002 n=6)
Query/rate(a_1[1m]),_instant_query                                             123.1µ ±  7%   112.7µ ±  3%   -8.43% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_100_steps                                134.4µ ±  4%   120.3µ ±  4%  -10.48% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_1000_steps                               236.3µ ±  1%   178.0µ ±  1%  -24.69% (p=0.002 n=6)
Query/rate(a_100[1m]),_instant_query                                           570.8µ ±  2%   557.4µ ±  4%   -2.34% (p=0.004 n=6)
Query/rate(a_100[1m]),_range_query_with_100_steps                              1.606m ±  2%   1.153m ±  0%  -28.20% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_1000_steps                            10.467m ±  2%   5.811m ±  1%  -44.48% (p=0.002 n=6)
Query/rate(a_2000[1m]),_instant_query                                          7.007m ±  0%   6.811m ±  1%   -2.80% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_100_steps                             26.64m ±  1%   17.95m ±  2%  -32.62% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_1000_steps                            195.2m ±  2%   104.3m ±  0%  -46.57% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_10000_steps                             1174.8µ ±  2%   744.3µ ±  1%  -36.64% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_10000_steps                            97.00m ±  1%   52.99m ±  6%  -45.38% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_10000_steps                            2.004 ±  2%    1.047 ±  5%  -47.75% (p=0.002 n=6)
Query/rate(a_1[1d]),_instant_query                                             745.6µ ±  2%   553.4µ ±  7%  -25.77% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_100_steps                               1302.3µ ±  1%   988.9µ ±  1%  -24.07% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_1000_steps                               6.227m ±  1%   4.860m ±  1%  -21.94% (p=0.002 n=6)
Query/rate(a_100[1d]),_instant_query                                           43.68m ±  0%   33.71m ±  0%  -22.83% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_100_steps                              98.27m ±  1%   76.75m ±  1%  -21.90% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_1000_steps                             587.2m ±  1%   461.3m ±  0%  -21.44% (p=0.002 n=6)
Query/rate(a_2000[1d]),_instant_query                                          813.1m ±  1%   618.5m ±  0%  -23.93% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_100_steps                              1.914 ±  1%    1.482 ±  8%  -22.58% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_1000_steps                            11.671 ±  1%    9.149 ±  0%  -21.61% (p=0.002 n=6)

                                                                       │      standard      │               streaming                │
                                                                       │         B          │       B         vs base                │
Query/a_1,_instant_query                                                      72.84Mi ±  1%    72.25Mi ±  1%    -0.82% (p=0.015 n=6)
Query/a_1,_range_query_with_100_steps                                         72.70Mi ±  1%    71.51Mi ±  1%    -1.64% (p=0.004 n=6)
Query/a_1,_range_query_with_1000_steps                                        72.05Mi ±  2%    70.04Mi ±  1%    -2.80% (p=0.002 n=6)
Query/a_100,_instant_query                                                    66.33Mi ±  2%    65.80Mi ±  1%         ~ (p=0.368 n=6)
Query/a_100,_range_query_with_100_steps                                       67.19Mi ±  1%    65.77Mi ±  1%    -2.12% (p=0.004 n=6)
Query/a_100,_range_query_with_1000_steps                                      68.54Mi ±  1%    68.41Mi ±  1%         ~ (p=0.394 n=6)
Query/a_2000,_instant_query                                                   68.42Mi ±  3%    68.45Mi ±  0%         ~ (p=1.000 n=6)
Query/a_2000,_range_query_with_100_steps                                      74.73Mi ±  1%    76.11Mi ±  1%    +1.84% (p=0.002 n=6)
Query/a_2000,_range_query_with_1000_steps                                     134.5Mi ±  1%    136.3Mi ±  9%         ~ (p=0.509 n=6)
Query/rate(a_1[1m]),_instant_query                                            71.85Mi ±  1%    71.84Mi ±  1%         ~ (p=0.818 n=6)
Query/rate(a_1[1m]),_range_query_with_100_steps                               72.88Mi ±  2%    71.96Mi ±  1%         ~ (p=0.065 n=6)
Query/rate(a_1[1m]),_range_query_with_1000_steps                              70.88Mi ±  2%    69.79Mi ±  1%         ~ (p=0.067 n=6)
Query/rate(a_100[1m]),_instant_query                                          66.68Mi ±  1%    65.83Mi ±  0%    -1.28% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_100_steps                             66.08Mi ±  1%    65.91Mi ±  2%         ~ (p=0.699 n=6)
Query/rate(a_100[1m]),_range_query_with_1000_steps                            67.52Mi ±  1%    68.53Mi ±  1%    +1.49% (p=0.002 n=6)
Query/rate(a_2000[1m]),_instant_query                                         69.01Mi ±  1%    68.42Mi ±  2%    -0.85% (p=0.015 n=6)
Query/rate(a_2000[1m]),_range_query_with_100_steps                            73.89Mi ±  2%    75.41Mi ±  3%    +2.06% (p=0.006 n=6)
Query/rate(a_2000[1m]),_range_query_with_1000_steps                           131.7Mi ±  1%    132.8Mi ±  2%         ~ (p=0.292 n=6)
Query/rate(a_1[1m]),_range_query_with_10000_steps                             67.65Mi ±  1%    68.72Mi ±  1%    +1.58% (p=0.004 n=6)
Query/rate(a_100[1m]),_range_query_with_10000_steps                           103.6Mi ±  1%    112.3Mi ±  4%    +8.41% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_10000_steps                          543.8Mi ±  5%    421.5Mi ±  3%   -22.49% (p=0.002 n=6)
Query/rate(a_1[1d]),_instant_query                                            67.98Mi ±  1%    73.12Mi ±  2%    +7.56% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_100_steps                               67.20Mi ±  0%    71.86Mi ±  3%    +6.94% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_1000_steps                              65.39Mi ±  1%    64.44Mi ±  2%         ~ (p=0.258 n=6)
Query/rate(a_100[1d]),_instant_query                                          70.02Mi ±  2%    70.53Mi ±  4%         ~ (p=0.329 n=6)
Query/rate(a_100[1d]),_range_query_with_100_steps                             68.98Mi ±  4%    68.83Mi ±  3%         ~ (p=0.589 n=6)
Query/rate(a_100[1d]),_range_query_with_1000_steps                            63.19Mi ±  2%    65.43Mi ±  5%         ~ (p=0.240 n=6)
Query/rate(a_2000[1d]),_instant_query                                         78.38Mi ±  1%    76.97Mi ±  1%    -1.80% (p=0.004 n=6)
Query/rate(a_2000[1d]),_range_query_with_100_steps                            78.30Mi ±  4%    78.70Mi ±  2%         ~ (p=0.331 n=6)
Query/rate(a_2000[1d]),_range_query_with_1000_steps                           111.8Mi ±  4%    110.5Mi ±  4%         ~ (p=0.937 n=6)

Which issue(s) this PR fixes or relates to

(none)

Checklist

Tests updated.
[n/a] Documentation added.
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
[n/a] about-versioning.md updated with experimental features.

…ing and arithmetic operations

…no duplicate timestamps.

…e side of a binary operation

…any more series than the other

…utputSeries`.

…buffer in memory

…mark

This dramatically reduces peak memory consumption, as we'll no longer potentially over-allocate by a factor of 10. For example, previously, if we needed a slice of 1001 points, we'd end up using a slice with capacity 10000. Now it will only be 1024. This fixes the issue where queries that return many series result in peak memory utilisation far greater than Prometheus' engine. Full benchmark results: goos: darwin goarch: arm64 pkg: github.com/grafana/mimir/pkg/streamingpromql/benchmarks │ standard │ streaming │ │ sec/op │ sec/op vs base │ Query/a_1,_instant_query 127.9µ ± 5% 120.6µ ± 4% -5.72% (p=0.004 n=6) Query/a_1,_range_query_with_100_steps 141.8µ ± 15% 131.8µ ± 8% ~ (p=0.132 n=6) Query/a_1,_range_query_with_1000_steps 181.0µ ± 2% 171.4µ ± 1% -5.33% (p=0.002 n=6) Query/a_100,_instant_query 577.7µ ± 7% 558.4µ ± 3% -3.35% (p=0.002 n=6) Query/a_100,_range_query_with_100_steps 1.060m ± 5% 1.039m ± 5% ~ (p=0.240 n=6) Query/a_100,_range_query_with_1000_steps 5.276m ± 1% 5.191m ± 2% -1.61% (p=0.015 n=6) Query/a_2000,_instant_query 7.207m ± 0% 7.015m ± 1% -2.65% (p=0.002 n=6) Query/a_2000,_range_query_with_100_steps 15.88m ± 1% 15.39m ± 1% -3.08% (p=0.002 n=6) Query/a_2000,_range_query_with_1000_steps 94.69m ± 1% 90.68m ± 0% -4.24% (p=0.002 n=6) Query/rate(a_1[1m]),_instant_query 123.1µ ± 7% 112.7µ ± 3% -8.43% (p=0.002 n=6) Query/rate(a_1[1m]),_range_query_with_100_steps 134.4µ ± 4% 120.3µ ± 4% -10.48% (p=0.002 n=6) Query/rate(a_1[1m]),_range_query_with_1000_steps 236.3µ ± 1% 178.0µ ± 1% -24.69% (p=0.002 n=6) Query/rate(a_100[1m]),_instant_query 570.8µ ± 2% 557.4µ ± 4% -2.34% (p=0.004 n=6) Query/rate(a_100[1m]),_range_query_with_100_steps 1.606m ± 2% 1.153m ± 0% -28.20% (p=0.002 n=6) Query/rate(a_100[1m]),_range_query_with_1000_steps 10.467m ± 2% 5.811m ± 1% -44.48% (p=0.002 n=6) Query/rate(a_2000[1m]),_instant_query 7.007m ± 0% 6.811m ± 1% -2.80% (p=0.002 n=6) Query/rate(a_2000[1m]),_range_query_with_100_steps 26.64m ± 1% 17.95m ± 2% -32.62% (p=0.002 n=6) Query/rate(a_2000[1m]),_range_query_with_1000_steps 195.2m ± 2% 104.3m ± 0% -46.57% (p=0.002 n=6) Query/rate(a_1[1m]),_range_query_with_10000_steps 1174.8µ ± 2% 744.3µ ± 1% -36.64% (p=0.002 n=6) Query/rate(a_100[1m]),_range_query_with_10000_steps 97.00m ± 1% 52.99m ± 6% -45.38% (p=0.002 n=6) Query/rate(a_2000[1m]),_range_query_with_10000_steps 2.004 ± 2% 1.047 ± 5% -47.75% (p=0.002 n=6) Query/rate(a_1[1d]),_instant_query 745.6µ ± 2% 553.4µ ± 7% -25.77% (p=0.002 n=6) Query/rate(a_1[1d]),_range_query_with_100_steps 1302.3µ ± 1% 988.9µ ± 1% -24.07% (p=0.002 n=6) Query/rate(a_1[1d]),_range_query_with_1000_steps 6.227m ± 1% 4.860m ± 1% -21.94% (p=0.002 n=6) Query/rate(a_100[1d]),_instant_query 43.68m ± 0% 33.71m ± 0% -22.83% (p=0.002 n=6) Query/rate(a_100[1d]),_range_query_with_100_steps 98.27m ± 1% 76.75m ± 1% -21.90% (p=0.002 n=6) Query/rate(a_100[1d]),_range_query_with_1000_steps 587.2m ± 1% 461.3m ± 0% -21.44% (p=0.002 n=6) Query/rate(a_2000[1d]),_instant_query 813.1m ± 1% 618.5m ± 0% -23.93% (p=0.002 n=6) Query/rate(a_2000[1d]),_range_query_with_100_steps 1.914 ± 1% 1.482 ± 8% -22.58% (p=0.002 n=6) Query/rate(a_2000[1d]),_range_query_with_1000_steps 11.671 ± 1% 9.149 ± 0% -21.61% (p=0.002 n=6) Query/a_1_-_b_1,_instant_query 225.9µ ± 1% 211.4µ ± 2% -6.42% (p=0.002 n=6) Query/a_1_-_b_1,_range_query_with_100_steps 260.3µ ± 0% 225.9µ ± 2% -13.23% (p=0.002 n=6) Query/a_1_-_b_1,_range_query_with_1000_steps 618.3µ ± 2% 319.6µ ± 1% -48.31% (p=0.002 n=6) Query/a_100_-_b_100,_instant_query 1.182m ± 1% 1.125m ± 0% -4.82% (p=0.002 n=6) Query/a_100_-_b_100,_range_query_with_100_steps 4.437m ± 1% 2.042m ± 1% -53.98% (p=0.002 n=6) Query/a_100_-_b_100,_range_query_with_1000_steps 33.51m ± 0% 10.19m ± 1% -69.59% (p=0.002 n=6) Query/a_2000_-_b_2000,_instant_query 15.92m ± 1% 15.38m ± 1% -3.42% (p=0.002 n=6) Query/a_2000_-_b_2000,_range_query_with_100_steps 82.01m ± 1% 32.76m ± 1% -60.06% (p=0.002 n=6) Query/a_2000_-_b_2000,_range_query_with_1000_steps 735.9m ± 1% 187.5m ± 1% -74.53% (p=0.002 n=6) Query/a_1_-_b_1,_range_query_with_10000_steps 3.298m ± 1% 1.273m ± 1% -61.40% (p=0.002 n=6) Query/a_100_-_b_100,_range_query_with_10000_steps 322.21m ± 0% 87.20m ± 0% -72.94% (p=0.002 n=6) Query/a_2000_-_b_2000,_range_query_with_10000_steps 7.571 ± 0% 1.733 ± 3% -77.11% (p=0.002 n=6) Query/a_100{l=~"[13579]."}_-_b_100,_instant_query 1.114m ± 1% 1.086m ± 1% -2.52% (p=0.002 n=6) Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_100_steps 3.182m ± 1% 1.803m ± 1% -43.32% (p=0.002 n=6) Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_1000_steps 21.739m ± 0% 8.109m ± 1% -62.70% (p=0.002 n=6) Query/a_2000{l=~"1..."}_-_b_2000,_instant_query 12.093m ± 1% 9.135m ± 2% -24.46% (p=0.002 n=6) Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps 52.50m ± 1% 18.63m ± 2% -64.51% (p=0.002 n=6) Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps 470.5m ± 1% 103.8m ± 0% -77.94% (p=0.002 n=6) Query/sum(a_1),_instant_query 118.5µ ± 2% 110.8µ ± 4% -6.42% (p=0.002 n=6) Query/sum(a_1),_range_query_with_100_steps 129.1µ ± 1% 117.5µ ± 2% -9.00% (p=0.002 n=6) Query/sum(a_1),_range_query_with_1000_steps 207.4µ ± 1% 172.5µ ± 1% -16.83% (p=0.002 n=6) Query/sum(a_100),_instant_query 584.5µ ± 1% 561.9µ ± 3% -3.87% (p=0.002 n=6) Query/sum(a_100),_range_query_with_100_steps 1.092m ± 1% 1.020m ± 1% -6.59% (p=0.002 n=6) Query/sum(a_100),_range_query_with_1000_steps 5.683m ± 1% 5.199m ± 0% -8.53% (p=0.002 n=6) Query/sum(a_2000),_instant_query 7.251m ± 1% 7.010m ± 0% -3.32% (p=0.002 n=6) Query/sum(a_2000),_range_query_with_100_steps 16.84m ± 1% 15.74m ± 1% -6.53% (p=0.002 n=6) Query/sum(a_2000),_range_query_with_1000_steps 111.55m ± 2% 93.75m ± 0% -15.96% (p=0.002 n=6) Query/sum_by_(l)(h_1),_instant_query 148.6µ ± 2% 139.7µ ± 2% -5.98% (p=0.002 n=6) Query/sum_by_(l)(h_1),_range_query_with_100_steps 183.7µ ± 1% 170.9µ ± 2% -6.98% (p=0.002 n=6) Query/sum_by_(l)(h_1),_range_query_with_1000_steps 533.7µ ± 1% 480.8µ ± 2% -9.92% (p=0.002 n=6) Query/sum_by_(l)(h_100),_instant_query 2.576m ± 1% 2.458m ± 1% -4.55% (p=0.002 n=6) Query/sum_by_(l)(h_100),_range_query_with_100_steps 5.505m ± 1% 5.193m ± 1% -5.65% (p=0.002 n=6) Query/sum_by_(l)(h_100),_range_query_with_1000_steps 33.01m ± 1% 29.73m ± 1% -9.94% (p=0.002 n=6) Query/sum_by_(l)(h_2000),_instant_query 46.80m ± 1% 45.32m ± 1% -3.16% (p=0.002 n=6) Query/sum_by_(l)(h_2000),_range_query_with_100_steps 107.41m ± 1% 98.70m ± 1% -8.11% (p=0.002 n=6) Query/sum_by_(l)(h_2000),_range_query_with_1000_steps 705.7m ± 0% 563.3m ± 1% -20.17% (p=0.002 n=6) Query/sum_by_(le)(h_1),_instant_query 150.1µ ± 1% 141.2µ ± 3% -5.95% (p=0.002 n=6) Query/sum_by_(le)(h_1),_range_query_with_100_steps 187.2µ ± 1% 174.5µ ± 2% -6.80% (p=0.002 n=6) Query/sum_by_(le)(h_1),_range_query_with_1000_steps 562.1µ ± 0% 500.4µ ± 2% -10.98% (p=0.002 n=6) Query/sum_by_(le)(h_100),_instant_query 2.525m ± 2% 2.412m ± 1% -4.46% (p=0.002 n=6) Query/sum_by_(le)(h_100),_range_query_with_100_steps 5.403m ± 1% 5.072m ± 1% -6.13% (p=0.002 n=6) Query/sum_by_(le)(h_100),_range_query_with_1000_steps 32.40m ± 3% 29.59m ± 1% -8.67% (p=0.002 n=6) Query/sum_by_(le)(h_2000),_instant_query 45.69m ± 1% 43.78m ± 1% -4.18% (p=0.002 n=6) Query/sum_by_(le)(h_2000),_range_query_with_100_steps 104.06m ± 0% 96.44m ± 1% -7.32% (p=0.002 n=6) Query/sum_by_(le)(h_2000),_range_query_with_1000_steps 683.8m ± 0% 560.5m ± 1% -18.03% (p=0.002 n=6) Query/rate(a_1[1m])_+_rate(b_1[1m]),_instant_query 232.9µ ± 1% 216.2µ ± 2% -7.17% (p=0.002 n=6) Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_100_steps 276.6µ ± 2% 231.2µ ± 2% -16.42% (p=0.002 n=6) Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_1000_steps 710.4µ ± 2% 340.7µ ± 3% -52.04% (p=0.002 n=6) Query/rate(a_100[1m])_+_rate(b_100[1m]),_instant_query 1.150m ± 1% 1.102m ± 0% -4.15% (p=0.002 n=6) Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_100_steps 5.453m ± 1% 2.268m ± 1% -58.41% (p=0.002 n=6) Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_1000_steps 43.07m ± 1% 11.55m ± 1% -73.18% (p=0.002 n=6) Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_instant_query 15.28m ± 1% 14.82m ± 1% -3.04% (p=0.002 n=6) Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_100_steps 102.03m ± 1% 37.52m ± 1% -63.22% (p=0.002 n=6) Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_1000_steps 928.4m ± 1% 214.7m ± 0% -76.87% (p=0.002 n=6) Query/sum(a_1_+_b_1),_instant_query 225.9µ ± 1% 214.6µ ± 2% -5.00% (p=0.002 n=6) Query/sum(a_1_+_b_1),_range_query_with_100_steps 266.8µ ± 2% 228.0µ ± 2% -14.55% (p=0.002 n=6) Query/sum(a_1_+_b_1),_range_query_with_1000_steps 650.4µ ± 2% 329.4µ ± 2% -49.36% (p=0.002 n=6) Query/sum(a_100_+_b_100),_instant_query 1.189m ± 1% 1.133m ± 1% -4.72% (p=0.002 n=6) Query/sum(a_100_+_b_100),_range_query_with_100_steps 4.504m ± 1% 2.043m ± 1% -54.64% (p=0.002 n=6) Query/sum(a_100_+_b_100),_range_query_with_1000_steps 34.07m ± 1% 10.34m ± 0% -69.64% (p=0.002 n=6) Query/sum(a_2000_+_b_2000),_instant_query 15.94m ± 1% 15.41m ± 1% -3.34% (p=0.002 n=6) Query/sum(a_2000_+_b_2000),_range_query_with_100_steps 83.55m ± 2% 33.20m ± 1% -60.26% (p=0.002 n=6) Query/sum(a_2000_+_b_2000),_range_query_with_1000_steps 764.1m ± 0% 192.3m ± 0% -74.84% (p=0.002 n=6) Query/sum_by_(le)(rate(h_1[1m])),_instant_query 153.3µ ± 1% 142.2µ ± 1% -7.26% (p=0.002 n=6) Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_100_steps 241.2µ ± 4% 184.2µ ± 3% -23.64% (p=0.002 n=6) Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_1000_steps 932.3µ ± 6% 540.8µ ± 1% -41.99% (p=0.002 n=6) Query/sum_by_(le)(rate(h_100[1m])),_instant_query 2.455m ± 1% 2.341m ± 2% -4.62% (p=0.002 n=6) Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_100_steps 9.185m ± 8% 5.771m ± 0% -37.16% (p=0.002 n=6) Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_1000_steps 73.48m ± 0% 33.33m ± 1% -54.64% (p=0.002 n=6) Query/sum_by_(le)(rate(h_2000[1m])),_instant_query 43.97m ± 2% 42.68m ± 0% -2.95% (p=0.002 n=6) Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_100_steps 165.6m ± 15% 111.2m ± 1% -32.84% (p=0.002 n=6) Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_1000_steps 1327.0m ± 0% 641.8m ± 1% -51.63% (p=0.002 n=6) geomean 7.430m 5.174m -30.37% │ standard │ streaming │ │ B/op │ B/op vs base │ Query/a_1,_instant_query 20.20Ki ± 0% 17.24Ki ± 0% -14.65% (p=0.002 n=6) Query/a_1,_range_query_with_100_steps 21.16Ki ± 0% 17.84Ki ± 0% -15.69% (p=0.002 n=6) Query/a_1,_range_query_with_1000_steps 27.14Ki ± 1% 23.91Ki ± 0% -11.88% (p=0.002 n=6) Query/a_100,_instant_query 140.1Ki ± 0% 122.0Ki ± 0% -12.95% (p=0.002 n=6) Query/a_100,_range_query_with_100_steps 196.0Ki ± 0% 182.1Ki ± 0% -7.09% (p=0.002 n=6) Query/a_100,_range_query_with_1000_steps 770.4Ki ± 0% 758.2Ki ± 0% -1.58% (p=0.002 n=6) Query/a_2000,_instant_query 2.407Mi ± 0% 2.086Mi ± 0% -13.36% (p=0.002 n=6) Query/a_2000,_range_query_with_100_steps 3.428Mi ± 0% 3.265Mi ± 0% -4.77% (p=0.002 n=6) Query/a_2000,_range_query_with_1000_steps 14.50Mi ± 0% 14.37Mi ± 0% -0.88% (p=0.002 n=6) Query/rate(a_1[1m]),_instant_query 22.44Ki ± 0% 17.67Ki ± 0% -21.27% (p=0.002 n=6) Query/rate(a_1[1m]),_range_query_with_100_steps 23.29Ki ± 0% 18.24Ki ± 0% -21.69% (p=0.002 n=6) Query/rate(a_1[1m]),_range_query_with_1000_steps 29.13Ki ± 1% 23.98Ki ± 0% -17.69% (p=0.002 n=6) Query/rate(a_100[1m]),_instant_query 164.3Ki ± 0% 124.9Ki ± 0% -23.97% (p=0.002 n=6) Query/rate(a_100[1m]),_range_query_with_100_steps 220.1Ki ± 0% 184.9Ki ± 0% -16.01% (p=0.002 n=6) Query/rate(a_100[1m]),_range_query_with_1000_steps 758.7Ki ± 0% 724.8Ki ± 0% -4.47% (p=0.002 n=6) Query/rate(a_2000[1m]),_instant_query 2.857Mi ± 0% 2.137Mi ± 0% -25.18% (p=0.002 n=6) Query/rate(a_2000[1m]),_range_query_with_100_steps 3.879Mi ± 0% 3.326Mi ± 1% -14.25% (p=0.002 n=6) Query/rate(a_2000[1m]),_range_query_with_1000_steps 14.32Mi ± 0% 13.84Mi ± 0% -3.40% (p=0.002 n=6) Query/rate(a_1[1m]),_range_query_with_10000_steps 79.90Ki ± 6% 66.90Ki ± 2% -16.27% (p=0.002 n=6) Query/rate(a_100[1m]),_range_query_with_10000_steps 4.993Mi ± 4% 4.986Mi ± 0% ~ (p=1.000 n=6) Query/rate(a_2000[1m]),_range_query_with_10000_steps 1068.8Mi ± 0% 599.7Mi ± 0% -43.89% (p=0.002 n=6) Query/rate(a_1[1d]),_instant_query 632.50Ki ± 3% 67.00Ki ± 5% -89.41% (p=0.002 n=6) Query/rate(a_1[1d]),_range_query_with_100_steps 651.78Ki ± 1% 66.88Ki ± 5% -89.74% (p=0.002 n=6) Query/rate(a_1[1d]),_range_query_with_1000_steps 649.71Ki ± 3% 76.55Ki ± 16% -88.22% (p=0.002 n=6) Query/rate(a_100[1d]),_instant_query 5.181Mi ± 1% 4.681Mi ± 1% -9.66% (p=0.002 n=6) Query/rate(a_100[1d]),_range_query_with_100_steps 5.197Mi ± 3% 4.685Mi ± 2% -9.84% (p=0.002 n=6) Query/rate(a_100[1d]),_range_query_with_1000_steps 5.498Mi ± 7% 5.169Mi ± 2% ~ (p=0.180 n=6) Query/rate(a_2000[1d]),_instant_query 86.90Mi ± 1% 85.94Mi ± 1% -1.11% (p=0.009 n=6) Query/rate(a_2000[1d]),_range_query_with_100_steps 92.64Mi ± 1% 92.15Mi ± 1% ~ (p=0.240 n=6) Query/rate(a_2000[1d]),_range_query_with_1000_steps 126.4Mi ± 1% 125.8Mi ± 0% -0.47% (p=0.015 n=6) Query/a_1_-_b_1,_instant_query 41.54Ki ± 0% 36.42Ki ± 0% -12.33% (p=0.002 n=6) Query/a_1_-_b_1,_range_query_with_100_steps 47.81Ki ± 0% 37.61Ki ± 0% -21.32% (p=0.002 n=6) Query/a_1_-_b_1,_range_query_with_1000_steps 102.49Ki ± 0% 49.73Ki ± 0% -51.48% (p=0.002 n=6) Query/a_100_-_b_100,_instant_query 372.1Ki ± 0% 262.7Ki ± 0% -29.40% (p=0.002 n=6) Query/a_100_-_b_100,_range_query_with_100_steps 586.3Ki ± 0% 383.7Ki ± 0% -34.55% (p=0.002 n=6) Query/a_100_-_b_100,_range_query_with_1000_steps 2.541Mi ± 0% 1.503Mi ± 0% -40.85% (p=0.002 n=6) Query/a_2000_-_b_2000,_instant_query 6.968Mi ± 0% 4.516Mi ± 0% -35.19% (p=0.002 n=6) Query/a_2000_-_b_2000,_range_query_with_100_steps 9.187Mi ± 0% 6.820Mi ± 0% -25.76% (p=0.002 n=6) Query/a_2000_-_b_2000,_range_query_with_1000_steps 32.16Mi ± 0% 29.13Mi ± 1% -9.42% (p=0.002 n=6) Query/a_1_-_b_1,_range_query_with_10000_steps 688.7Ki ± 3% 139.7Ki ± 2% -79.72% (p=0.002 n=6) Query/a_100_-_b_100,_range_query_with_10000_steps 19.575Mi ± 1% 9.913Mi ± 0% -49.36% (p=0.002 n=6) Query/a_2000_-_b_2000,_range_query_with_10000_steps 3107.2Mi ± 0% 698.1Mi ± 0% -77.53% (p=0.002 n=6) Query/a_100{l=~"[13579]."}_-_b_100,_instant_query 291.1Ki ± 0% 211.4Ki ± 0% -27.38% (p=0.002 n=6) Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_100_steps 467.5Ki ± 0% 401.0Ki ± 0% -14.23% (p=0.002 n=6) Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_1000_steps 2.092Mi ± 0% 1.929Mi ± 0% -7.79% (p=0.002 n=6) Query/a_2000{l=~"1..."}_-_b_2000,_instant_query 5.258Mi ± 0% 3.011Mi ± 0% -42.73% (p=0.002 n=6) Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps 6.911Mi ± 0% 4.723Mi ± 1% -31.66% (p=0.002 n=6) Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps 23.54Mi ± 1% 20.19Mi ± 1% -14.24% (p=0.002 n=6) Query/sum(a_1),_instant_query 21.31Ki ± 0% 17.83Ki ± 0% -16.34% (p=0.002 n=6) Query/sum(a_1),_range_query_with_100_steps 22.26Ki ± 0% 19.51Ki ± 0% -12.35% (p=0.002 n=6) Query/sum(a_1),_range_query_with_1000_steps 28.32Ki ± 1% 33.65Ki ± 0% +18.83% (p=0.002 n=6) Query/sum(a_100),_instant_query 144.4Ki ± 0% 123.5Ki ± 0% -14.50% (p=0.002 n=6) Query/sum(a_100),_range_query_with_100_steps 204.2Ki ± 0% 183.7Ki ± 0% -10.03% (p=0.002 n=6) Query/sum(a_100),_range_query_with_1000_steps 779.0Ki ± 0% 769.8Ki ± 0% -1.18% (p=0.002 n=6) Query/sum(a_2000),_instant_query 2.471Mi ± 0% 2.072Mi ± 0% -16.13% (p=0.002 n=6) Query/sum(a_2000),_range_query_with_100_steps 3.561Mi ± 0% 3.179Mi ± 0% -10.72% (p=0.002 n=6) Query/sum(a_2000),_range_query_with_1000_steps 14.68Mi ± 0% 14.39Mi ± 0% -1.98% (p=0.002 n=6) Query/sum_by_(l)(h_1),_instant_query 27.61Ki ± 0% 23.24Ki ± 0% -15.85% (p=0.002 n=6) Query/sum_by_(l)(h_1),_range_query_with_100_steps 31.28Ki ± 0% 27.60Ki ± 0% -11.77% (p=0.002 n=6) Query/sum_by_(l)(h_1),_range_query_with_1000_steps 67.41Ki ± 0% 71.68Ki ± 0% +6.34% (p=0.002 n=6) Query/sum_by_(l)(h_100),_instant_query 906.3Ki ± 0% 748.2Ki ± 0% -17.44% (p=0.002 n=6) Query/sum_by_(l)(h_100),_range_query_with_100_steps 1.212Mi ± 0% 1.174Mi ± 0% -3.15% (p=0.002 n=6) Query/sum_by_(l)(h_100),_range_query_with_1000_steps 4.570Mi ± 1% 5.346Mi ± 1% +16.99% (p=0.002 n=6) Query/sum_by_(l)(h_2000),_instant_query 18.97Mi ± 0% 15.99Mi ± 1% -15.71% (p=0.002 n=6) Query/sum_by_(l)(h_2000),_range_query_with_100_steps 25.21Mi ± 0% 25.76Mi ± 1% +2.19% (p=0.002 n=6) Query/sum_by_(l)(h_2000),_range_query_with_1000_steps 91.63Mi ± 1% 124.01Mi ± 3% +35.34% (p=0.002 n=6) Query/sum_by_(le)(h_1),_instant_query 29.12Ki ± 0% 23.37Ki ± 0% -19.77% (p=0.002 n=6) Query/sum_by_(le)(h_1),_range_query_with_100_steps 32.55Ki ± 0% 33.19Ki ± 0% +1.98% (p=0.002 n=6) Query/sum_by_(le)(h_1),_range_query_with_1000_steps 68.69Ki ± 0% 117.21Ki ± 0% +70.64% (p=0.002 n=6) Query/sum_by_(le)(h_100),_instant_query 868.9Ki ± 0% 729.8Ki ± 0% -16.01% (p=0.002 n=6) Query/sum_by_(le)(h_100),_range_query_with_100_steps 1.181Mi ± 0% 1.052Mi ± 0% -10.87% (p=0.002 n=6) Query/sum_by_(le)(h_100),_range_query_with_1000_steps 4.542Mi ± 0% 4.491Mi ± 1% -1.11% (p=0.002 n=6) Query/sum_by_(le)(h_2000),_instant_query 18.12Mi ± 0% 15.45Mi ± 0% -14.73% (p=0.002 n=6) Query/sum_by_(le)(h_2000),_range_query_with_100_steps 24.50Mi ± 0% 22.18Mi ± 0% -9.46% (p=0.002 n=6) Query/sum_by_(le)(h_2000),_range_query_with_1000_steps 91.00Mi ± 0% 89.33Mi ± 1% -1.84% (p=0.002 n=6) Query/rate(a_1[1m])_+_rate(b_1[1m]),_instant_query 46.02Ki ± 0% 37.26Ki ± 0% -19.05% (p=0.002 n=6) Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_100_steps 52.31Ki ± 0% 38.45Ki ± 0% -26.49% (p=0.002 n=6) Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_1000_steps 106.31Ki ± 0% 49.80Ki ± 0% -53.15% (p=0.002 n=6) Query/rate(a_100[1m])_+_rate(b_100[1m]),_instant_query 415.2Ki ± 0% 267.0Ki ± 0% -35.69% (p=0.002 n=6) Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_100_steps 629.6Ki ± 0% 387.8Ki ± 0% -38.42% (p=0.002 n=6) Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_1000_steps 2.512Mi ± 0% 1.435Mi ± 0% -42.90% (p=0.002 n=6) Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_instant_query 7.780Mi ± 0% 4.586Mi ± 0% -41.05% (p=0.002 n=6) Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_100_steps 10.024Mi ± 0% 6.886Mi ± 1% -31.31% (p=0.002 n=6) Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_1000_steps 31.72Mi ± 1% 27.82Mi ± 1% -12.27% (p=0.002 n=6) Query/sum(a_1_+_b_1),_instant_query 42.80Ki ± 0% 37.06Ki ± 0% -13.42% (p=0.002 n=6) Query/sum(a_1_+_b_1),_range_query_with_100_steps 49.12Ki ± 0% 39.33Ki ± 0% -19.93% (p=0.002 n=6) Query/sum(a_1_+_b_1),_range_query_with_1000_steps 103.73Ki ± 0% 59.51Ki ± 0% -42.63% (p=0.002 n=6) Query/sum(a_100_+_b_100),_instant_query 376.7Ki ± 0% 264.2Ki ± 0% -29.86% (p=0.002 n=6) Query/sum(a_100_+_b_100),_range_query_with_100_steps 595.1Ki ± 0% 384.4Ki ± 0% -35.40% (p=0.002 n=6) Query/sum(a_100_+_b_100),_range_query_with_1000_steps 2.549Mi ± 0% 1.514Mi ± 0% -40.61% (p=0.002 n=6) Query/sum(a_2000_+_b_2000),_instant_query 7.032Mi ± 0% 4.487Mi ± 0% -36.20% (p=0.002 n=6) Query/sum(a_2000_+_b_2000),_range_query_with_100_steps 9.335Mi ± 6% 6.728Mi ± 0% -27.94% (p=0.002 n=6) Query/sum(a_2000_+_b_2000),_range_query_with_1000_steps 32.44Mi ± 1% 29.36Mi ± 2% -9.48% (p=0.002 n=6) Query/sum_by_(le)(rate(h_1[1m])),_instant_query 32.88Ki ± 0% 23.80Ki ± 0% -27.61% (p=0.002 n=6) Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_100_steps 36.00Ki ± 0% 33.58Ki ± 0% -6.71% (p=0.002 n=6) Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_1000_steps 69.37Ki ± 0% 114.78Ki ± 0% +65.46% (p=0.002 n=6) Query/sum_by_(le)(rate(h_100[1m])),_instant_query 1007.7Ki ± 0% 750.0Ki ± 0% -25.57% (p=0.002 n=6) Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_100_steps 1.315Mi ± 0% 1.073Mi ± 0% -18.42% (p=0.002 n=6) Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_1000_steps 4.488Mi ± 0% 4.308Mi ± 0% -4.01% (p=0.002 n=6) Query/sum_by_(le)(rate(h_2000[1m])),_instant_query 20.78Mi ± 0% 15.81Mi ± 0% -23.90% (p=0.002 n=6) Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_100_steps 27.08Mi ± 1% 22.56Mi ± 0% -16.72% (p=0.002 n=6) Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_1000_steps 278.70Mi ± 0% 85.93Mi ± 0% -69.17% (p=0.002 n=6) geomean 1.147Mi 884.4Ki -24.72% │ standard │ streaming │ │ allocs/op │ allocs/op vs base │ Query/a_1,_instant_query 360.0 ± 0% 303.0 ± 0% -15.83% (p=0.002 n=6) Query/a_1,_range_query_with_100_steps 371.0 ± 0% 308.0 ± 0% -16.98% (p=0.002 n=6) Query/a_1,_range_query_with_1000_steps 401.0 ± 0% 338.0 ± 0% -15.71% (p=0.002 n=6) Query/a_100,_instant_query 1.975k ± 0% 1.910k ± 0% -3.29% (p=0.002 n=6) Query/a_100,_range_query_with_100_steps 2.486k ± 0% 2.417k ± 0% -2.78% (p=0.002 n=6) Query/a_100,_range_query_with_1000_steps 5.499k ± 0% 5.432k ± 0% -1.22% (p=0.002 n=6) Query/a_2000,_instant_query 32.79k ± 0% 32.70k ± 0% -0.27% (p=0.002 n=6) Query/a_2000,_range_query_with_100_steps 42.84k ± 0% 42.76k ± 0% -0.18% (p=0.002 n=6) Query/a_2000,_range_query_with_1000_steps 102.9k ± 0% 102.9k ± 0% -0.06% (p=0.002 n=6) Query/rate(a_1[1m]),_instant_query 403.0 ± 0% 313.0 ± 0% -22.33% (p=0.002 n=6) Query/rate(a_1[1m]),_range_query_with_100_steps 414.0 ± 0% 318.0 ± 0% -23.19% (p=0.002 n=6) Query/rate(a_1[1m]),_range_query_with_1000_steps 441.0 ± 0% 345.0 ± 0% -21.77% (p=0.002 n=6) Query/rate(a_100[1m]),_instant_query 2.422k ± 0% 2.019k ± 0% -16.64% (p=0.002 n=6) Query/rate(a_100[1m]),_range_query_with_100_steps 2.934k ± 0% 2.527k ± 0% -13.87% (p=0.002 n=6) Query/rate(a_100[1m]),_range_query_with_1000_steps 5.649k ± 0% 5.242k ± 0% -7.21% (p=0.002 n=6) Query/rate(a_2000[1m]),_instant_query 40.85k ± 0% 34.71k ± 0% -15.02% (p=0.002 n=6) Query/rate(a_2000[1m]),_range_query_with_100_steps 50.90k ± 0% 44.78k ± 0% -12.02% (p=0.002 n=6) Query/rate(a_2000[1m]),_range_query_with_1000_steps 105.00k ± 0% 98.90k ± 0% -5.81% (p=0.002 n=6) Query/rate(a_1[1m]),_range_query_with_10000_steps 670.0 ± 0% 574.0 ± 0% -14.33% (p=0.002 n=6) Query/rate(a_100[1m]),_range_query_with_10000_steps 28.63k ± 0% 28.23k ± 0% -1.39% (p=0.002 n=6) Query/rate(a_2000[1m]),_range_query_with_10000_steps 576.2k ± 0% 564.0k ± 0% -2.13% (p=0.002 n=6) Query/rate(a_1[1d]),_instant_query 646.5 ± 0% 542.0 ± 0% -16.16% (p=0.002 n=6) Query/rate(a_1[1d]),_range_query_with_100_steps 655.5 ± 0% 545.0 ± 0% -16.86% (p=0.002 n=6) Query/rate(a_1[1d]),_range_query_with_1000_steps 676.0 ± 0% 567.0 ± 0% -16.12% (p=0.002 n=6) Query/rate(a_100[1d]),_instant_query 25.37k ± 0% 24.94k ± 0% -1.71% (p=0.002 n=6) Query/rate(a_100[1d]),_range_query_with_100_steps 25.67k ± 0% 25.26k ± 0% -1.59% (p=0.002 n=6) Query/rate(a_100[1d]),_range_query_with_1000_steps 27.78k ± 0% 27.40k ± 0% -1.36% (p=0.002 n=6) Query/rate(a_2000[1d]),_instant_query 501.6k ± 0% 491.8k ± 0% -1.95% (p=0.002 n=6) Query/rate(a_2000[1d]),_range_query_with_100_steps 509.9k ± 0% 503.8k ± 0% -1.20% (p=0.002 n=6) Query/rate(a_2000[1d]),_range_query_with_1000_steps 552.1k ± 0% 546.1k ± 0% -1.08% (p=0.002 n=6) Query/a_1_-_b_1,_instant_query 720.0 ± 0% 626.0 ± 0% -13.06% (p=0.002 n=6) Query/a_1_-_b_1,_range_query_with_100_steps 934.0 ± 0% 636.0 ± 0% -31.91% (p=0.002 n=6) Query/a_1_-_b_1,_range_query_with_1000_steps 2795.0 ± 0% 697.0 ± 0% -75.06% (p=0.002 n=6) Query/a_100_-_b_100,_instant_query 4.579k ± 0% 4.246k ± 0% -7.27% (p=0.002 n=6) Query/a_100_-_b_100,_range_query_with_100_steps 5.800k ± 0% 5.260k ± 0% -9.31% (p=0.002 n=6) Query/a_100_-_b_100,_range_query_with_1000_steps 15.51k ± 0% 11.29k ± 0% -27.20% (p=0.002 n=6) Query/a_2000_-_b_2000,_instant_query 77.69k ± 0% 73.50k ± 0% -5.40% (p=0.002 n=6) Query/a_2000_-_b_2000,_range_query_with_100_steps 93.95k ± 0% 93.56k ± 0% -0.42% (p=0.002 n=6) Query/a_2000_-_b_2000,_range_query_with_1000_steps 216.1k ± 0% 213.8k ± 0% -1.04% (p=0.002 n=6) Query/a_1_-_b_1,_range_query_with_10000_steps 21.256k ± 0% 1.154k ± 0% -94.57% (p=0.002 n=6) Query/a_100_-_b_100,_range_query_with_10000_steps 97.56k ± 0% 56.64k ± 0% -41.95% (p=0.002 n=6) Query/a_2000_-_b_2000,_range_query_with_10000_steps 1.172M ± 0% 1.126M ± 0% -3.95% (p=0.002 n=6) Query/a_100{l=~"[13579]."}_-_b_100,_instant_query 3.577k ± 0% 3.302k ± 0% -7.69% (p=0.002 n=6) Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_100_steps 4.565k ± 0% 4.061k ± 0% -11.04% (p=0.002 n=6) Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_1000_steps 12.508k ± 0% 8.592k ± 0% -31.31% (p=0.002 n=6) Query/a_2000{l=~"1..."}_-_b_2000,_instant_query 57.48k ± 0% 45.14k ± 0% -21.46% (p=0.002 n=6) Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps 69.74k ± 0% 56.46k ± 0% -19.04% (p=0.002 n=6) Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps 161.7k ± 0% 122.1k ± 0% -24.50% (p=0.002 n=6) Query/sum(a_1),_instant_query 384.0 ± 0% 312.0 ± 0% -18.75% (p=0.002 n=6) Query/sum(a_1),_range_query_with_100_steps 395.0 ± 0% 321.0 ± 0% -18.73% (p=0.002 n=6) Query/sum(a_1),_range_query_with_1000_steps 425.0 ± 0% 351.0 ± 0% -17.41% (p=0.002 n=6) Query/sum(a_100),_instant_query 1.999k ± 0% 1.920k ± 0% -3.95% (p=0.002 n=6) Query/sum(a_100),_range_query_with_100_steps 2.511k ± 0% 2.429k ± 0% -3.25% (p=0.002 n=6) Query/sum(a_100),_range_query_with_1000_steps 5.524k ± 0% 5.444k ± 0% -1.46% (p=0.002 n=6) Query/sum(a_2000),_instant_query 32.81k ± 0% 32.71k ± 0% -0.30% (p=0.002 n=6) Query/sum(a_2000),_range_query_with_100_steps 42.86k ± 0% 42.76k ± 0% -0.23% (p=0.002 n=6) Query/sum(a_2000),_range_query_with_1000_steps 103.0k ± 0% 102.9k ± 0% -0.06% (p=0.002 n=6) Query/sum_by_(l)(h_1),_instant_query 482.0 ± 0% 404.0 ± 0% -16.18% (p=0.002 n=6) Query/sum_by_(l)(h_1),_range_query_with_100_steps 518.0 ± 0% 438.0 ± 0% -15.44% (p=0.002 n=6) Query/sum_by_(l)(h_1),_range_query_with_1000_steps 698.0 ± 0% 619.0 ± 0% -11.32% (p=0.002 n=6) Query/sum_by_(l)(h_100),_instant_query 10.86k ± 0% 10.76k ± 0% -0.94% (p=0.002 n=6) Query/sum_by_(l)(h_100),_range_query_with_100_steps 13.87k ± 0% 14.19k ± 0% +2.29% (p=0.002 n=6) Query/sum_by_(l)(h_100),_range_query_with_1000_steps 31.93k ± 0% 32.27k ± 0% +1.07% (p=0.002 n=6) Query/sum_by_(l)(h_2000),_instant_query 223.0k ± 1% 210.1k ± 0% -5.78% (p=0.002 n=6) Query/sum_by_(l)(h_2000),_range_query_with_100_steps 269.2k ± 0% 279.3k ± 0% +3.74% (p=0.002 n=6) Query/sum_by_(l)(h_2000),_range_query_with_1000_steps 629.6k ± 0% 643.9k ± 0% +2.27% (p=0.002 n=6) Query/sum_by_(le)(h_1),_instant_query 490.0 ± 0% 409.0 ± 0% -16.53% (p=0.002 n=6) Query/sum_by_(le)(h_1),_range_query_with_100_steps 526.0 ± 0% 463.5 ± 0% -11.88% (p=0.002 n=6) Query/sum_by_(le)(h_1),_range_query_with_1000_steps 706.0 ± 0% 644.0 ± 0% -8.78% (p=0.002 n=6) Query/sum_by_(le)(h_100),_instant_query 10.74k ± 0% 10.65k ± 0% -0.87% (p=0.002 n=6) Query/sum_by_(le)(h_100),_range_query_with_100_steps 13.76k ± 0% 13.68k ± 0% -0.54% (p=0.002 n=6) Query/sum_by_(le)(h_100),_range_query_with_1000_steps 31.81k ± 0% 31.75k ± 0% -0.19% (p=0.002 n=6) Query/sum_by_(le)(h_2000),_instant_query 218.8k ± 1% 206.9k ± 0% -5.48% (p=0.002 n=6) Query/sum_by_(le)(h_2000),_range_query_with_100_steps 267.0k ± 0% 267.2k ± 0% ~ (p=0.240 n=6) Query/sum_by_(le)(h_2000),_range_query_with_1000_steps 627.4k ± 0% 627.7k ± 0% +0.04% (p=0.002 n=6) Query/rate(a_1[1m])_+_rate(b_1[1m]),_instant_query 798.0 ± 0% 645.0 ± 0% -19.17% (p=0.002 n=6) Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_100_steps 1012.0 ± 0% 656.0 ± 0% -35.18% (p=0.002 n=6) Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_1000_steps 2867.0 ± 0% 710.0 ± 0% -75.24% (p=0.002 n=6) Query/rate(a_100[1m])_+_rate(b_100[1m]),_instant_query 5.468k ± 0% 4.465k ± 0% -18.35% (p=0.002 n=6) Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_100_steps 6.690k ± 0% 5.480k ± 0% -18.09% (p=0.002 n=6) Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_1000_steps 15.80k ± 0% 10.92k ± 0% -30.90% (p=0.002 n=6) Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_instant_query 93.80k ± 0% 77.52k ± 0% -17.35% (p=0.002 n=6) Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_100_steps 110.06k ± 0% 97.59k ± 0% -11.33% (p=0.002 n=6) Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_1000_steps 220.1k ± 0% 205.8k ± 0% -6.52% (p=0.002 n=6) Query/sum(a_1_+_b_1),_instant_query 744.0 ± 0% 635.0 ± 0% -14.65% (p=0.002 n=6) Query/sum(a_1_+_b_1),_range_query_with_100_steps 958.0 ± 0% 650.0 ± 0% -32.15% (p=0.002 n=6) Query/sum(a_1_+_b_1),_range_query_with_1000_steps 2819.0 ± 0% 710.0 ± 0% -74.81% (p=0.002 n=6) Query/sum(a_100_+_b_100),_instant_query 4.603k ± 0% 4.256k ± 0% -7.54% (p=0.002 n=6) Query/sum(a_100_+_b_100),_range_query_with_100_steps 5.824k ± 0% 5.271k ± 0% -9.49% (p=0.002 n=6) Query/sum(a_100_+_b_100),_range_query_with_1000_steps 15.53k ± 0% 11.30k ± 0% -27.22% (p=0.002 n=6) Query/sum(a_2000_+_b_2000),_instant_query 77.72k ± 0% 73.51k ± 0% -5.42% (p=0.002 n=6) Query/sum(a_2000_+_b_2000),_range_query_with_100_steps 93.99k ± 1% 93.57k ± 0% -0.45% (p=0.002 n=6) Query/sum(a_2000_+_b_2000),_range_query_with_1000_steps 216.1k ± 0% 214.0k ± 0% -0.99% (p=0.002 n=6) Query/sum_by_(le)(rate(h_1[1m])),_instant_query 554.0 ± 0% 418.0 ± 0% -24.55% (p=0.002 n=6) Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_100_steps 582.0 ± 0% 473.0 ± 0% -18.73% (p=0.002 n=6) Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_1000_steps 745.0 ± 0% 635.0 ± 0% -14.77% (p=0.002 n=6) Query/sum_by_(le)(rate(h_100[1m])),_instant_query 13.20k ± 0% 11.26k ± 0% -14.68% (p=0.002 n=6) Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_100_steps 16.20k ± 0% 14.30k ± 0% -11.77% (p=0.002 n=6) Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_1000_steps 32.46k ± 0% 30.56k ± 0% -5.86% (p=0.002 n=6) Query/sum_by_(le)(rate(h_2000[1m])),_instant_query 276.4k ± 1% 218.9k ± 0% -20.81% (p=0.002 n=6) Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_100_steps 315.3k ± 0% 279.2k ± 0% -11.45% (p=0.002 n=6) Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_1000_steps 665.7k ± 0% 603.7k ± 0% -9.31% (p=0.002 n=6) geomean 11.10k 9.219k -16.97% │ standard │ streaming │ │ B │ B vs base │ Query/a_1,_instant_query 72.84Mi ± 1% 72.25Mi ± 1% -0.82% (p=0.015 n=6) Query/a_1,_range_query_with_100_steps 72.70Mi ± 1% 71.51Mi ± 1% -1.64% (p=0.004 n=6) Query/a_1,_range_query_with_1000_steps 72.05Mi ± 2% 70.04Mi ± 1% -2.80% (p=0.002 n=6) Query/a_100,_instant_query 66.33Mi ± 2% 65.80Mi ± 1% ~ (p=0.368 n=6) Query/a_100,_range_query_with_100_steps 67.19Mi ± 1% 65.77Mi ± 1% -2.12% (p=0.004 n=6) Query/a_100,_range_query_with_1000_steps 68.54Mi ± 1% 68.41Mi ± 1% ~ (p=0.394 n=6) Query/a_2000,_instant_query 68.42Mi ± 3% 68.45Mi ± 0% ~ (p=1.000 n=6) Query/a_2000,_range_query_with_100_steps 74.73Mi ± 1% 76.11Mi ± 1% +1.84% (p=0.002 n=6) Query/a_2000,_range_query_with_1000_steps 134.5Mi ± 1% 136.3Mi ± 9% ~ (p=0.509 n=6) Query/rate(a_1[1m]),_instant_query 71.85Mi ± 1% 71.84Mi ± 1% ~ (p=0.818 n=6) Query/rate(a_1[1m]),_range_query_with_100_steps 72.88Mi ± 2% 71.96Mi ± 1% ~ (p=0.065 n=6) Query/rate(a_1[1m]),_range_query_with_1000_steps 70.88Mi ± 2% 69.79Mi ± 1% ~ (p=0.067 n=6) Query/rate(a_100[1m]),_instant_query 66.68Mi ± 1% 65.83Mi ± 0% -1.28% (p=0.002 n=6) Query/rate(a_100[1m]),_range_query_with_100_steps 66.08Mi ± 1% 65.91Mi ± 2% ~ (p=0.699 n=6) Query/rate(a_100[1m]),_range_query_with_1000_steps 67.52Mi ± 1% 68.53Mi ± 1% +1.49% (p=0.002 n=6) Query/rate(a_2000[1m]),_instant_query 69.01Mi ± 1% 68.42Mi ± 2% -0.85% (p=0.015 n=6) Query/rate(a_2000[1m]),_range_query_with_100_steps 73.89Mi ± 2% 75.41Mi ± 3% +2.06% (p=0.006 n=6) Query/rate(a_2000[1m]),_range_query_with_1000_steps 131.7Mi ± 1% 132.8Mi ± 2% ~ (p=0.292 n=6) Query/rate(a_1[1m]),_range_query_with_10000_steps 67.65Mi ± 1% 68.72Mi ± 1% +1.58% (p=0.004 n=6) Query/rate(a_100[1m]),_range_query_with_10000_steps 103.6Mi ± 1% 112.3Mi ± 4% +8.41% (p=0.002 n=6) Query/rate(a_2000[1m]),_range_query_with_10000_steps 543.8Mi ± 5% 421.5Mi ± 3% -22.49% (p=0.002 n=6) Query/rate(a_1[1d]),_instant_query 67.98Mi ± 1% 73.12Mi ± 2% +7.56% (p=0.002 n=6) Query/rate(a_1[1d]),_range_query_with_100_steps 67.20Mi ± 0% 71.86Mi ± 3% +6.94% (p=0.002 n=6) Query/rate(a_1[1d]),_range_query_with_1000_steps 65.39Mi ± 1% 64.44Mi ± 2% ~ (p=0.258 n=6) Query/rate(a_100[1d]),_instant_query 70.02Mi ± 2% 70.53Mi ± 4% ~ (p=0.329 n=6) Query/rate(a_100[1d]),_range_query_with_100_steps 68.98Mi ± 4% 68.83Mi ± 3% ~ (p=0.589 n=6) Query/rate(a_100[1d]),_range_query_with_1000_steps 63.19Mi ± 2% 65.43Mi ± 5% ~ (p=0.240 n=6) Query/rate(a_2000[1d]),_instant_query 78.38Mi ± 1% 76.97Mi ± 1% -1.80% (p=0.004 n=6) Query/rate(a_2000[1d]),_range_query_with_100_steps 78.30Mi ± 4% 78.70Mi ± 2% ~ (p=0.331 n=6) Query/rate(a_2000[1d]),_range_query_with_1000_steps 111.8Mi ± 4% 110.5Mi ± 4% ~ (p=0.937 n=6) Query/a_1_-_b_1,_instant_query 72.75Mi ± 1% 72.88Mi ± 1% ~ (p=0.290 n=6) Query/a_1_-_b_1,_range_query_with_100_steps 71.61Mi ± 1% 71.97Mi ± 1% ~ (p=0.420 n=6) Query/a_1_-_b_1,_range_query_with_1000_steps 68.48Mi ± 1% 70.09Mi ± 1% +2.35% (p=0.002 n=6) Query/a_100_-_b_100,_instant_query 66.76Mi ± 1% 65.92Mi ± 1% -1.25% (p=0.011 n=6) Query/a_100_-_b_100,_range_query_with_100_steps 67.09Mi ± 1% 66.21Mi ± 1% -1.30% (p=0.002 n=6) Query/a_100_-_b_100,_range_query_with_1000_steps 73.14Mi ± 3% 68.69Mi ± 1% -6.09% (p=0.002 n=6) Query/a_2000_-_b_2000,_instant_query 69.03Mi ± 1% 68.98Mi ± 1% ~ (p=0.699 n=6) Query/a_2000_-_b_2000,_range_query_with_100_steps 89.50Mi ± 3% 78.59Mi ± 2% -12.19% (p=0.002 n=6) Query/a_2000_-_b_2000,_range_query_with_1000_steps 213.2Mi ± 0% 139.4Mi ± 1% -34.62% (p=0.002 n=6) Query/a_1_-_b_1,_range_query_with_10000_steps 68.86Mi ± 1% 70.09Mi ± 1% +1.78% (p=0.004 n=6) Query/a_100_-_b_100,_range_query_with_10000_steps 158.6Mi ± 5% 113.7Mi ± 3% -28.34% (p=0.002 n=6) Query/a_2000_-_b_2000,_range_query_with_10000_steps 1600.8Mi ± 11% 458.4Mi ± 1% -71.36% (p=0.002 n=6) Query/a_100{l=~"[13579]."}_-_b_100,_instant_query 67.02Mi ± 1% 66.34Mi ± 1% ~ (p=0.065 n=6) Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_100_steps 66.80Mi ± 1% 67.16Mi ± 1% ~ (p=0.065 n=6) Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_1000_steps 70.56Mi ± 1% 71.12Mi ± 1% ~ (p=0.193 n=6) Query/a_2000{l=~"1..."}_-_b_2000,_instant_query 68.22Mi ± 3% 141.39Mi ± 3% +107.26% (p=0.002 n=6) Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps 81.85Mi ± 1% 113.49Mi ± 37% +38.66% (p=0.002 n=6) Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps 189.6Mi ± 6% 177.4Mi ± 6% -6.46% (p=0.041 n=6) Query/sum(a_1),_instant_query 72.89Mi ± 1% 72.74Mi ± 1% ~ (p=0.331 n=6) Query/sum(a_1),_range_query_with_100_steps 72.87Mi ± 1% 73.45Mi ± 1% ~ (p=0.126 n=6) Query/sum(a_1),_range_query_with_1000_steps 71.77Mi ± 2% 79.17Mi ± 2% +10.31% (p=0.002 n=6) Query/sum(a_100),_instant_query 66.07Mi ± 1% 65.70Mi ± 1% ~ (p=0.132 n=6) Query/sum(a_100),_range_query_with_100_steps 66.69Mi ± 1% 65.60Mi ± 1% -1.63% (p=0.002 n=6) Query/sum(a_100),_range_query_with_1000_steps 68.49Mi ± 1% 65.65Mi ± 1% -4.15% (p=0.002 n=6) Query/sum(a_2000),_instant_query 68.56Mi ± 1% 67.64Mi ± 1% -1.34% (p=0.009 n=6) Query/sum(a_2000),_range_query_with_100_steps 75.28Mi ± 2% 66.77Mi ± 1% -11.31% (p=0.002 n=6) Query/sum(a_2000),_range_query_with_1000_steps 133.92Mi ± 1% 70.73Mi ± 2% -47.19% (p=0.002 n=6) Query/sum_by_(l)(h_1),_instant_query 70.35Mi ± 2% 69.88Mi ± 1% ~ (p=0.061 n=6) Query/sum_by_(l)(h_1),_range_query_with_100_steps 70.41Mi ± 2% 69.94Mi ± 2% ~ (p=0.485 n=6) Query/sum_by_(l)(h_1),_range_query_with_1000_steps 67.91Mi ± 2% 69.67Mi ± 1% +2.59% (p=0.002 n=6) Query/sum_by_(l)(h_100),_instant_query 66.62Mi ± 3% 66.23Mi ± 1% -0.57% (p=0.041 n=6) Query/sum_by_(l)(h_100),_range_query_with_100_steps 69.16Mi ± 1% 68.66Mi ± 1% ~ (p=0.132 n=6) Query/sum_by_(l)(h_100),_range_query_with_1000_steps 88.68Mi ± 2% 76.27Mi ± 1% -13.99% (p=0.002 n=6) Query/sum_by_(l)(h_2000),_instant_query 72.65Mi ± 2% 70.82Mi ± 2% -2.52% (p=0.009 n=6) Query/sum_by_(l)(h_2000),_range_query_with_100_steps 119.36Mi ± 1% 82.17Mi ± 2% -31.16% (p=0.002 n=6) Query/sum_by_(l)(h_2000),_range_query_with_1000_steps 464.1Mi ± 0% 150.4Mi ± 3% -67.60% (p=0.002 n=6) Query/sum_by_(le)(h_1),_instant_query 70.67Mi ± 1% 70.34Mi ± 1% ~ (p=0.071 n=6) Query/sum_by_(le)(h_1),_range_query_with_100_steps 70.23Mi ± 1% 74.91Mi ± 2% +6.66% (p=0.002 n=6) Query/sum_by_(le)(h_1),_range_query_with_1000_steps 68.41Mi ± 1% 86.61Mi ± 1% +26.61% (p=0.002 n=6) Query/sum_by_(le)(h_100),_instant_query 67.10Mi ± 1% 66.30Mi ± 1% -1.20% (p=0.030 n=6) Query/sum_by_(le)(h_100),_range_query_with_100_steps 69.29Mi ± 1% 66.02Mi ± 1% -4.72% (p=0.002 n=6) Query/sum_by_(le)(h_100),_range_query_with_1000_steps 87.10Mi ± 2% 69.05Mi ± 1% -20.73% (p=0.002 n=6) Query/sum_by_(le)(h_2000),_instant_query 72.02Mi ± 2% 71.38Mi ± 3% ~ (p=0.093 n=6) Query/sum_by_(le)(h_2000),_range_query_with_100_steps 113.55Mi ± 1% 69.91Mi ± 1% -38.43% (p=0.002 n=6) Query/sum_by_(le)(h_2000),_range_query_with_1000_steps 431.33Mi ± 0% 73.33Mi ± 1% -83.00% (p=0.002 n=6) Query/rate(a_1[1m])_+_rate(b_1[1m]),_instant_query 72.02Mi ± 1% 72.16Mi ± 1% ~ (p=0.589 n=6) Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_100_steps 71.26Mi ± 1% 71.87Mi ± 1% +0.86% (p=0.015 n=6) Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_1000_steps 67.88Mi ± 1% 70.08Mi ± 1% +3.23% (p=0.002 n=6) Query/rate(a_100[1m])_+_rate(b_100[1m]),_instant_query 66.96Mi ± 1% 65.84Mi ± 1% -1.68% (p=0.002 n=6) Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_100_steps 67.11Mi ± 1% 66.54Mi ± 1% ~ (p=0.093 n=6) Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_1000_steps 72.39Mi ± 3% 67.70Mi ± 2% -6.49% (p=0.002 n=6) Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_instant_query 69.16Mi ± 2% 69.09Mi ± 2% ~ (p=0.563 n=6) Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_100_steps 86.86Mi ± 4% 78.24Mi ± 2% -9.92% (p=0.002 n=6) Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_1000_steps 211.6Mi ± 0% 140.8Mi ± 3% -33.47% (p=0.002 n=6) Query/sum(a_1_+_b_1),_instant_query 72.75Mi ± 1% 72.20Mi ± 1% ~ (p=0.065 n=6) Query/sum(a_1_+_b_1),_range_query_with_100_steps 71.85Mi ± 1% 72.59Mi ± 1% +1.03% (p=0.002 n=6) Query/sum(a_1_+_b_1),_range_query_with_1000_steps 68.41Mi ± 1% 75.12Mi ± 1% +9.80% (p=0.002 n=6) Query/sum(a_100_+_b_100),_instant_query 66.73Mi ± 1% 66.24Mi ± 1% ~ (p=0.065 n=6) Query/sum(a_100_+_b_100),_range_query_with_100_steps 66.50Mi ± 1% 65.92Mi ± 1% -0.87% (p=0.041 n=6) Query/sum(a_100_+_b_100),_range_query_with_1000_steps 73.99Mi ± 1% 65.73Mi ± 2% -11.16% (p=0.002 n=6) Query/sum(a_2000_+_b_2000),_instant_query 69.02Mi ± 1% 68.65Mi ± 3% ~ (p=0.589 n=6) Query/sum(a_2000_+_b_2000),_range_query_with_100_steps 89.06Mi ± 2% 68.27Mi ± 2% -23.35% (p=0.002 n=6) Query/sum(a_2000_+_b_2000),_range_query_with_1000_steps 213.24Mi ± 0% 77.19Mi ± 1% -63.80% (p=0.002 n=6) Query/sum_by_(le)(rate(h_1[1m])),_instant_query 70.17Mi ± 1% 69.88Mi ± 1% ~ (p=0.699 n=6) Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_100_steps 69.22Mi ± 2% 74.95Mi ± 1% +8.27% (p=0.002 n=6) Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_1000_steps 67.19Mi ± 1% 87.00Mi ± 2% +29.49% (p=0.002 n=6) Query/sum_by_(le)(rate(h_100[1m])),_instant_query 66.81Mi ± 1% 66.08Mi ± 1% -1.10% (p=0.002 n=6) Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_100_steps 68.78Mi ± 2% 66.34Mi ± 1% -3.56% (p=0.002 n=6) Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_1000_steps 84.90Mi ± 3% 68.48Mi ± 2% -19.33% (p=0.002 n=6) Query/sum_by_(le)(rate(h_2000[1m])),_instant_query 70.59Mi ± 3% 70.88Mi ± 2% ~ (p=0.589 n=6) Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_100_steps 113.20Mi ± 2% 69.80Mi ± 1% -38.33% (p=0.002 n=6) Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_1000_steps 270.64Mi ± 3% 71.90Mi ± 2% -73.43% (p=0.002 n=6) geomean 85.81Mi 78.13Mi -8.95%

# Conflicts: # pkg/streamingpromql/engine_test.go # pkg/streamingpromql/query.go

charleskorn · 2024-05-10T06:00:37Z

pkg/streamingpromql/testdata/ours/binary_operators.test

+  right_side{env="test"} 100 200 300
+
+eval_fail range from 0 to 42m step 6m left_side * on (env) right_side
+  # TODO: expected_message multiple matches for labels: many-to-one matching must be explicit (group_left/group_right)


Note to reviewers: this TODO is pending prometheus/prometheus#14038.

If that PR doesn't merge before this one, I'll update this in a follow-up PR.

charleskorn · 2024-05-10T06:00:49Z

pkg/streamingpromql/testdata/ours/binary_operators.test

+  right_side{env="test", pod="d"} _ 10 11
+
+eval_fail range from 0 to 42m step 6m left_side * on (env) right_side
+  # TODO: expected_message found duplicate series for the match group {env="test"} on the right hand-side of the operation: [{__name__="right_side", env="test", pod="b"}, {__name__="right_side", env="test", pod="a"}];many-to-many matching not allowed: matching labels must be unique on one side


Note to reviewers: this TODO is pending prometheus/prometheus#14038.

If that PR doesn't merge before this one, I'll update this in a follow-up PR.

charleskorn · 2024-05-10T06:57:15Z

pkg/streamingpromql/operator/binary_operation.go

+	// TODO: Prometheus' engine uses strings for the key here, which would avoid issues with hash collisions, but seems much slower.
+	// Either we should use strings, or we'll need to deal with hash collisions.
+	hashFunc := b.hashFunc()
+	outputSeriesMap := map[uint64]*binaryOperationOutputSeries{}
+
+	// TODO: is it better to use whichever side has fewer series for this first loop? Should result in a smaller map and therefore less work later on
+	// Would need to be careful about 'or' and 'unless' cases


Note to reviewers: I'll address both of these comments in a future PR.

I've addressed the first of these (hash collisions) in 20344f5.

jhesketh

lgtm, just nits/q's :-)

pkg/streamingpromql/operator/binary_operation.go

pkg/streamingpromql/operator/binary_operation_test.go

This solves the issue of hash conflicts, and it doesn't introduce a significant performance or memory utilisation penalty.

pkg/streamingpromql/operator/binary_operation.go

tools/benchmark-query-engine/main.go

…ter`

jhesketh

lgtm, just one last nit/question :-)

jhesketh · 2024-05-14T05:29:52Z

pkg/streamingpromql/testdata/ours/binary_operators.test

+  left_side{env="test", pod="a", group="foo"} 1 2 3
+  left_side{env="test", pod="b", group="bar"} 4 5 6
+  left_side{env="prod", pod="a", group="baz"} 7 8 9


For these tests to validate labelsFunc, does it make sense to list the labels in different orders. eg:

Suggested change

left_side{env="test", pod="a", group="foo"} 1 2 3

left_side{env="test", pod="b", group="bar"} 4 5 6

left_side{env="prod", pod="a", group="baz"} 7 8 9

left_side{env="test", pod="a", group="foo"} 1 2 3

left_side{pod="b", env="test", group="bar"} 4 5 6

left_side{ group="baz", env="prod", pod="a",} 7 8 9

I don't think this will influence the order they're received by the binary operation operator - these series are written a the TSDB and then read back later by the query. I'd expect the labels returned by the TSDB to be sorted regardless of the order they appeared when written.

charleskorn · 2024-05-14T06:17:44Z

Given the approval, I'm going to merge this, but any post-merge feedback is, as always, more than welcome.

charleskorn added 30 commits May 8, 2024 13:53

Add some test cases

dfbd2c6

Enable upstream test cases for binary operators with one-to-one match…

74f17a9

…ing and arithmetic operations

Fix invalid test case

f909d55

Make it clear that points are expected to be in timestamp order with …

3cba572

…no duplicate timestamps.

Enable relevant benchmarks

9ed7961

Enable test cases for atan2

3ef9607

Initial (WIP) version of one-to-one matching

67414c5

Update comment

81fd3e9

Fix issue where on and ignoring are ignored

a24f632

Use correct terminology

a0c966e

Fix issue where errors are lost if result is a matrix

8183253

Add support for case where multiple different series contribute to on…

e2430bc

…e side of a binary operation

Add comment explaining when slices are returned to the pool

c8438af

Add further comment

1152fb7

Remove redundant checks

b9c6770

Log output from benchmark binary if it fails

29698cd

Don't limit the number of samples loaded in a query

dedf3e4

Return point slices to the pool once we're done with them

84b3a8e

Split SeriesMetadata into smaller methods

7465322

Clarify comment

55fd653

Add extra benchmark

c2d4946

Add comment explaining tradeoff

ead5557

Rename binaryOperationSeriesPair to binaryOperationOutputSeries

5162a1d

Add benchmark for the case where one side of a binary operation has m…

236f161

…any more series than the other

Further simplify SeriesMetadata by moving more logic into `computeO…

7834b9e

…utputSeries`.

Sort output series to minimise the number of input series we need to …

5555bb7

…buffer in memory

Add support for capturing CPU and memory profiles while running bench…

dc1457e

…mark

Reuse slices when computing result.

9d4bc14

Merge branch 'main' into charleskorn/binary-operators

957eb35

# Conflicts: # pkg/streamingpromql/engine_test.go # pkg/streamingpromql/query.go

charleskorn added 5 commits May 10, 2024 13:49

Add changelog entry

e07463c

Fix flaky test.

bc557db

Remove outdated comments

a23be64

Don't bother buffering series that won't be used.

7241678

Clarify expected behaviour of Close

3c971f4

charleskorn mentioned this pull request May 10, 2024

Don't allocate an unnecessary slice when accumulating chunks #8098

Merged

charleskorn added 2 commits May 10, 2024 16:08

Move FallbackEngine and NotSupportedError to their own package.

053f3d3

Consolidate check for supported operator into one place

3f1aa3a

charleskorn commented May 10, 2024

View reviewed changes

charleskorn marked this pull request as ready for review May 10, 2024 06:39

charleskorn requested a review from a team as a code owner May 10, 2024 06:39

charleskorn commented May 10, 2024

View reviewed changes

jhesketh approved these changes May 13, 2024

View reviewed changes

charleskorn mentioned this pull request May 14, 2024

Streaming PromQL engine: native histograms #8121

Open

4 tasks

charleskorn and others added 2 commits May 14, 2024 13:45

Use string of group labels as map key when constructing output series

20344f5

This solves the issue of hash conflicts, and it doesn't introduce a significant performance or memory utilisation penalty.

Merge branch 'main' into charleskorn/binary-operators

7751e87

jhesketh reviewed May 14, 2024

View reviewed changes

pkg/streamingpromql/operator/binary_operation.go Show resolved Hide resolved

tools/benchmark-query-engine/main.go Outdated Show resolved Hide resolved

charleskorn added 6 commits May 14, 2024 14:12

Fix indentation

dac8815

Add tests for sorting behaviour

a737a98

Reduce duplication in favourRightSideSorter and `favourRightSideSor…

108ce4e

…ter`

Fix linting

98c8f60

Add tests for binaryOperationSeriesBuffer.

5818cd1

Add test cases for on and ignoring with multiple labels

61b59a1

charleskorn requested a review from jhesketh May 14, 2024 05:07

jhesketh approved these changes May 14, 2024

View reviewed changes

charleskorn merged commit faf2b84 into main May 14, 2024
29 checks passed

charleskorn deleted the charleskorn/binary-operators branch May 14, 2024 06:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Streaming PromQL engine: binary arithmetic operations with one-to-one matching #8096

Streaming PromQL engine: binary arithmetic operations with one-to-one matching #8096

charleskorn commented May 10, 2024 •

edited

charleskorn May 10, 2024 •

edited

charleskorn May 10, 2024 •

edited

charleskorn May 10, 2024

charleskorn May 14, 2024

jhesketh left a comment

jhesketh left a comment

jhesketh May 14, 2024

charleskorn May 14, 2024

charleskorn commented May 14, 2024

Streaming PromQL engine: binary arithmetic operations with one-to-one matching #8096

Streaming PromQL engine: binary arithmetic operations with one-to-one matching #8096

Conversation

charleskorn commented May 10, 2024 • edited

What this PR does

Performance

Fix for issue where peak memory utilisation is much higher than Prometheus' engine for queries that select many series

Which issue(s) this PR fixes or relates to

Checklist

charleskorn May 10, 2024 • edited

Choose a reason for hiding this comment

charleskorn May 10, 2024 • edited

Choose a reason for hiding this comment

charleskorn May 10, 2024

Choose a reason for hiding this comment

charleskorn May 14, 2024

Choose a reason for hiding this comment

jhesketh left a comment

Choose a reason for hiding this comment

jhesketh left a comment

Choose a reason for hiding this comment

jhesketh May 14, 2024

Choose a reason for hiding this comment

charleskorn May 14, 2024

Choose a reason for hiding this comment

charleskorn commented May 14, 2024

charleskorn commented May 10, 2024 •

edited

charleskorn May 10, 2024 •

edited

charleskorn May 10, 2024 •

edited