New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
--benchmark-samples
does not seem to work as expected
#2813
Comments
You can't do this with Catch2, because the benchmark will always run the function at least twice. Once for the sample you ask for, but before that happens it will run at least once for the warmup.
|
And wouldn't it be relevant to specifically test if |
That could be done, but that is a very specific, and frankly weird, use case. Catch2 primarily targets microbenchmarking and attempts to provide high quality statistical output. Doing only one sample, especially without warm up, can't provide that, and the only advantage of using Furthermore, if this is a function that can only be called once in program's lifetime (e.g. |
Without entering the details of my use case, it consists on few steps, running in a given order, each of them being internally protected against replay attacks. And I want to benchmark each of the steps, and the benchmark results to be included in the report (thus go through the reporter). But most important, independently of my use case, I would normally expect while (current_warmup_time <= max_warmup_time) {
// do something
} so that |
Hi all,
I have some functions I'm trying to benchmark, and some of them can only be called once (internal mechanisms prevent a user to call them twice). The benchmarked function cannot be modified for obvious reasons, and there is no way I can prevent them from checking they are called only once.
After few search, I found the
--benchmark-samples
option, which allows to set the number of times the code withinBENCHMARK
macro is called. Unfortunately it seems it does not act as intended. I even used all the arsenal of available options to control benchmarks as follows--benchmark-samples 1 --benchmark-resamples 1 --benchmark-warmup-time 0 --benchmark-no-analysis
Nothing changed.
What am I doing wrong? What should I do instead ?
Here is a sample code that produces the issue.
The output is the following :
Where you can see that the string
********** THIS IS SPARTA *************
is printed more than once.Please help/advise.
Thanks.
The text was updated successfully, but these errors were encountered: