BUG: Benchmark by name shows only 7 results on main screen, but there are many more shown on historical plot

[DeaMariaLeon]

For example algorithms.Hashing.time_frame shows 7 results here:

But in the historical evolution section or page, there are a lot more results (more than 7):

Am I missing something?
In case this helps (this is temporary):
http://15.235.45.255:5000/

[austin3dickey]

Like #1541, I think this is also a known limitation of populating these numbers from a cache. Worth looking into how we'd fix it, balancing correctness with performance.

[jgehrcke]

Hey Dea María!

I would like to add a few thoughts in addition to what Austin said.

The "benchmarks" page is fed from an in-memory cache of the N most recently submitted results. N is currently not very easy to set as an admin (that is work that needs to be done).

It's hard-coded. It is much smaller in testing config (50000) than it is in non-testing (800000).

Relevant code:

conbench/conbench/bmrt.py

Line 64 in 14b2a6d

BMRT_CACHE_SIZE = 0.8 * 10**6

That is, one way for you to increase that number is to not run in TESTING mode.

Maybe try doing that. In the long term you might run into out-of-memory errors then when having many results and too little memory.

The actual number (the cache size) should be made nicely configurable because after all memory usage of the Conbench container/process very much depends on this, linearly. We've picked 800000 pretty arbitrarily for the infrastructure we use, where I believe each container has about 8 GB of memory for itself.

At the bottom of the page you will find details. For example, in your case it reads:

1150 unique benchmark names seen across the 50000 newest results between 2023-11-25 00:24:11 UTC and 2023-11-25 00:54:34 UTC (~0 days).

In case of Arrow Conbench it reads at the time of writing:

635 unique benchmark names seen across the 758136 newest results between 2023-12-27 16:04:46 UTC and 2024-01-09 16:22:05 UTC (~13 days).

[DeaMariaLeon]

Thanks a lot for the explanation.